vault backup: 2023-05-23 09:28:54

Affected files:
.obsidian/workspace.json
STEM/AI/Neural Networks/MLP.md
STEM/AI/Neural Networks/MLP/Activation Functions.md
STEM/AI/Neural Networks/MLP/Back-Propagation.md
STEM/AI/Neural Networks/MLP/Decision Boundary.md
STEM/img/hidden-neuron-decision.png
STEM/img/mlp-non-linear-decision.png
STEM/img/mlp-xor-2.png
STEM/img/mlp-xor.png
STEM/img/sigmoid.png
STEM/img/tlu.png

AI/Neural Networks/MLP.md

@@ -12,4 +12,11 @@ A finite feed-forward MLP with 1 hidden layer can in theory approximate any math
 -   In practice not trainable with [[Back-Propagation|BP]]
 
 ![[activation-function.png]]
-![[mlp-arch-diagram.png]]
+![[mlp-arch-diagram.png]]
+## Weight Matrix
+-   Use matrix multiplication for layer output
+-   TLU is hard limiter
+![[tlu.png]]
+- $o_1$ to $o_4$ must all be one to overcome -3.5 bias and force output to 1
+![[mlp-non-linear-decision.png]]
+- Can generate a non-linear decision boundary

AI/Neural Networks/MLP/Activation Functions.md

@@ -0,0 +1,32 @@
+## Sigmoid
+-   Logistic function
+-   Normalises
+-   Introduces non-linearity
+-   Easy to take derivative
+$$\frac d {dx} \sigma(x)=
+\frac d {dx} \left[ 
+\frac 1 {1+e^{-x}}
+\right]
+=\sigma(x)\cdot(1-\sigma(x))$$
+
+![[sigmoid.png]]
+### Derivative
+
+$$y_j(n)=\varphi_j(v_j(n))=
+\frac 1 {1+e^{-v_j(n)}}$$
+$$\frac{\partial y_j(n)}{\partial v_j(n)}=
+\varphi_j'(v_j(n))=
+\frac{e^{-v_j(n)}}{(1+e^{-v_j(n)})^2}=
+y_j(n)(1-y_j(n))$$
+- Nice derivative
+- Max value of $\varphi_j'(v_j(n))$ occurs when $y_j(n)=0.5$
+- Min value of 0 when $y_j=0$ or $1$
+- Initial weights chosen so not saturated at 0 or 1
+
+If $y=\frac u v$
+Where $u$ and $v$ are differential functions
+
+$$\frac{dy}{dx}=\frac d {dx}\left(\frac u v\right)$$
+
+$$\frac{dy}{dx}=
+\frac {v \frac d {dx}(u) - u\frac d {dx}(v)} {v^2}$$

AI/Neural Networks/MLP/Back-Propagation.md

@@ -103,4 +103,5 @@ $$\Delta w_{ji}(n)=\eta\cdot\delta_j(n)\cdot y_i(n)$$
 	-   Proportional to the change in weights last iteration
 		-   Can shoot past local minima if descending quickly
 
-![[mlp-global-minimum.png]]
+![[mlp-global-minimum.png]]
+

AI/Neural Networks/MLP/Decision Boundary.md

@@ -0,0 +1,4 @@
+![[hidden-neuron-decision.png]]
+![[mlp-xor.png]]
+
+![[mlp-xor-2.png]]