andy
4cc2e79866
Affected files: .obsidian/global-search.json .obsidian/workspace.json Health/Alexithymia.md Health/BWS.md STEM/AI/Neural Networks/Activation Functions.md STEM/AI/Neural Networks/Architectures.md STEM/AI/Neural Networks/CNN/CNN.md STEM/AI/Neural Networks/MLP/Back-Propagation.md STEM/AI/Neural Networks/Transformers/Attention.md STEM/CS/Calling Conventions.md STEM/CS/Languages/Assembly.md
52 lines
2.1 KiB
Markdown
52 lines
2.1 KiB
Markdown
- Meant to mimic cognitive attention
|
|
- Picks out relevant bits of information
|
|
- Use gradient descent
|
|
- Used in 90s
|
|
- Multiplicative modules
|
|
- Sigma pi units
|
|
- Hyper-networks
|
|
- Draw from relevant state at any preceding point along sequence
|
|
- Addresses [RNNs](../RNN/RNN.md) vanishing gradient issues
|
|
- [LSTM](../RNN/LSTM.md) tends to poorly preserve far back [knowledge](../Neural%20Networks.md#Knowledge)
|
|
- Attention layer access all previous states and weighs according to learned measure of relevance
|
|
- Allows referring arbitrarily far back to relevant tokens
|
|
- Can be addd to [RNNs](../RNN/RNN.md)
|
|
- In 2016, a new type of highly parallelisable _decomposable attention_ was successfully combined with a [feedforward](../Architectures.md) network
|
|
- Attention useful in of itself, not just with [RNNs](../RNN/RNN.md)
|
|
- [Transformers](Transformers.md) use attention without recurrent connections
|
|
- Process all tokens simultaneously
|
|
- Calculate attention weights in successive layers
|
|
|
|
# Scaled Dot-Product
|
|
- Calculate attention weights between all tokens at once
|
|
- Learn 3 [weight](../Weight%20Init.md) matrices
|
|
- Query
|
|
- $W_Q$
|
|
- Key
|
|
- $W_K$
|
|
- Value
|
|
- $W_V$
|
|
- Word vectors
|
|
- For each token, $i$, input word embedding, $x_i$
|
|
- Multiply with each of above to produce vector
|
|
- Query Vector
|
|
- $q_i=x_iW_Q$
|
|
- Key Vector
|
|
- $k_i=x_iW_K$
|
|
- Value Vector
|
|
- $v_i=x_iW_V$
|
|
- Attention vector
|
|
- Query and key vectors between token $i$ and $j$
|
|
- $a_{ij}=q_i\cdot k_j$
|
|
- Divided by root of dimensionality of key vectors
|
|
- $\sqrt{d_k}$
|
|
- Pass through softmax to normalise
|
|
- $W_Q$ and $W_K$ are different matrices
|
|
- Attention can be non-symmetric
|
|
- Token $i$ attends to $j$ ($q_i\cdot k_j$ is large)
|
|
- Doesn't imply that $j$ attends to $i$ ($q_j\cdot k_i$ can be small)
|
|
- Output for token $i$ is weighted sum of value vectors of all tokens weighted by $a_{ij}$
|
|
- Attention from token $i$ to each other token
|
|
- $Q, K, V$ are matrices where $i$th row are vectors $q_i, k_i, v_i$ respectively
|
|
$$\text{Attention}(Q,K,V)=\text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V$$
|
|
- softmax taken over horizontal axis |