vault backup: 2023-05-23 09:28:54

Affected files:
.obsidian/workspace.json
STEM/AI/Neural Networks/MLP.md
STEM/AI/Neural Networks/MLP/Activation Functions.md
STEM/AI/Neural Networks/MLP/Back-Propagation.md
STEM/AI/Neural Networks/MLP/Decision Boundary.md
STEM/img/hidden-neuron-decision.png
STEM/img/mlp-non-linear-decision.png
STEM/img/mlp-xor-2.png
STEM/img/mlp-xor.png
STEM/img/sigmoid.png
STEM/img/tlu.png
This commit is contained in:
andy 2023-05-23 09:28:54 +01:00
parent f0e8559252
commit f65496a79f
10 changed files with 46 additions and 2 deletions

View File

@ -12,4 +12,11 @@ A finite feed-forward MLP with 1 hidden layer can in theory approximate any math
- In practice not trainable with [[Back-Propagation|BP]] - In practice not trainable with [[Back-Propagation|BP]]
![[activation-function.png]] ![[activation-function.png]]
![[mlp-arch-diagram.png]] ![[mlp-arch-diagram.png]]
## Weight Matrix
- Use matrix multiplication for layer output
- TLU is hard limiter
![[tlu.png]]
- $o_1$ to $o_4$ must all be one to overcome -3.5 bias and force output to 1
![[mlp-non-linear-decision.png]]
- Can generate a non-linear decision boundary

View File

@ -0,0 +1,32 @@
## Sigmoid
- Logistic function
- Normalises
- Introduces non-linearity
- Easy to take derivative
$$\frac d {dx} \sigma(x)=
\frac d {dx} \left[
\frac 1 {1+e^{-x}}
\right]
=\sigma(x)\cdot(1-\sigma(x))$$
![[sigmoid.png]]
### Derivative
$$y_j(n)=\varphi_j(v_j(n))=
\frac 1 {1+e^{-v_j(n)}}$$
$$\frac{\partial y_j(n)}{\partial v_j(n)}=
\varphi_j'(v_j(n))=
\frac{e^{-v_j(n)}}{(1+e^{-v_j(n)})^2}=
y_j(n)(1-y_j(n))$$
- Nice derivative
- Max value of $\varphi_j'(v_j(n))$ occurs when $y_j(n)=0.5$
- Min value of 0 when $y_j=0$ or $1$
- Initial weights chosen so not saturated at 0 or 1
If $y=\frac u v$
Where $u$ and $v$ are differential functions
$$\frac{dy}{dx}=\frac d {dx}\left(\frac u v\right)$$
$$\frac{dy}{dx}=
\frac {v \frac d {dx}(u) - u\frac d {dx}(v)} {v^2}$$

View File

@ -103,4 +103,5 @@ $$\Delta w_{ji}(n)=\eta\cdot\delta_j(n)\cdot y_i(n)$$
- Proportional to the change in weights last iteration - Proportional to the change in weights last iteration
- Can shoot past local minima if descending quickly - Can shoot past local minima if descending quickly
![[mlp-global-minimum.png]] ![[mlp-global-minimum.png]]

View File

@ -0,0 +1,4 @@
![[hidden-neuron-decision.png]]
![[mlp-xor.png]]
![[mlp-xor-2.png]]

Binary file not shown.

After

Width:  |  Height:  |  Size: 106 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

BIN
img/mlp-xor-2.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 120 KiB

BIN
img/mlp-xor.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

BIN
img/sigmoid.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

BIN
img/tlu.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB