sarsoo/stem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

vault backup: 2023-05-31 22:11:33

Browse Source 
Affected files:
.obsidian/workspace.json
STEM/AI/Neural Networks/Activation Functions.md
STEM/AI/Neural Networks/Transformers/Attention.md
STEM/CS/ABI.md
This commit is contained in:
andy 2023-05-31 22:11:34 +01:00
parent 1ad823d42c
commit bfdc107e5d
3 changed files with 8 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
AI/Neural Networks/Activation Functions.md
View File

@ -38,7 +38,7 @@ y_j(n)(1-y_j(n))$$
- Nice derivative - Nice derivative
- Max value of $\varphi_j'(v_j(n))$ occurs when $y_j(n)=0.5$ - Max value of $\varphi_j'(v_j(n))$ occurs when $y_j(n)=0.5$
- Min value of 0 when $y_j=0$ or $1$ - Min value of 0 when $y_j=0$ or $1$
- Initial [[Weight Init|weights]] chosen so not saturated at 0 or 1 - Initial [weights](Weight%20Init.md) chosen so not saturated at 0 or 1
If $y=\frac u v$ If $y=\frac u v$
Where $u$ and $v$ are differential functions Where $u$ and $v$ are differential functions
@ -53,7 +53,7 @@ Rectilinear
- For deep networks - For deep networks
- $y=max(0,x)$ - $y=max(0,x)$
- CNNs - CNNs
- Breaks associativity of successive [[convolution]]s - Breaks associativity of successive [convolutions](../../Signal%20Proc/Convolution.md)
- Critical for learning complex functions - Critical for learning complex functions
- Sometimes small scalar for negative - Sometimes small scalar for negative
- Leaky ReLu - Leaky ReLu

4
AI/Neural Networks/Transformers/Attention.md
View File

@ -6,8 +6,8 @@
- Sigma pi units - Sigma pi units
- Hyper-networks - Hyper-networks
- Draw from relevant state at any preceding point along sequence - Draw from relevant state at any preceding point along sequence
- Addresses [[RNN]]s vanishing gradient issues - Addresses [RNNs](../RNN/RNN.md) vanishing gradient issues
- [[LSTM]] tends to poorly preserve far back [[Neural Networks#Knowledge|knowledge]] - [LSTM](../RNN/LSTM.md) tends to poorly preserve far back [knowledge](../Neural%20Networks.md#Knowledge)
- Attention layer access all previous states and weighs according to learned measure of relevance - Attention layer access all previous states and weighs according to learned measure of relevance
- Allows referring arbitrarily far back to relevant tokens - Allows referring arbitrarily far back to relevant tokens
- Can be addd to [[RNN]]s - Can be addd to [[RNN]]s

8
CS/ABI.md
View File

@ -2,7 +2,7 @@
- Machine code therefore hardware-dependent - Machine code therefore hardware-dependent
- API defines this structure in source code - API defines this structure in source code
- Adherence usually responsibility of - Adherence usually responsibility of
- [[Compilers]] - [Compilers](Compilers.md)
- OS - OS
- Library author - Library author
@ -13,7 +13,7 @@
- Stack organisation - Stack organisation
- Memory access types - Memory access types
- Size, layouts and alignments of basic data types - Size, layouts and alignments of basic data types
- [[Calling Conventions]] - [Calling Conventions](Calling%20Conventions.md)
- How function arguments are passed - How function arguments are passed
- Stack or register - Stack or register
- Which registers for which function param - Which registers for which function param
@ -31,5 +31,5 @@
# Embedded ABI # Embedded ABI
- File format, data types, register usage, stack frame organisation, function parameter passing conventions - File format, data types, register usage, stack frame organisation, function parameter passing conventions
- For embedded OS - For embedded OS
- [[Compilers]] create object code compatible with code from other [[compilers]] - [Compilers](Compilers.md) create object code compatible with code from other [compilers](Compilers.md)
- Link libraries from different [[compilers]] - Link libraries from different [compilers](Compilers.md)