stem/AI/Neural Networks/Transformers/Attention.md
andy 4cc2e79866 vault backup: 2023-05-31 22:21:56
Affected files:
.obsidian/global-search.json
.obsidian/workspace.json
Health/Alexithymia.md
Health/BWS.md
STEM/AI/Neural Networks/Activation Functions.md
STEM/AI/Neural Networks/Architectures.md
STEM/AI/Neural Networks/CNN/CNN.md
STEM/AI/Neural Networks/MLP/Back-Propagation.md
STEM/AI/Neural Networks/Transformers/Attention.md
STEM/CS/Calling Conventions.md
STEM/CS/Languages/Assembly.md
2023-05-31 22:21:56 +01:00

2.1 KiB

  • Meant to mimic cognitive attention
    • Picks out relevant bits of information
    • Use gradient descent
  • Used in 90s
    • Multiplicative modules
    • Sigma pi units
    • Hyper-networks
  • Draw from relevant state at any preceding point along sequence
    • Addresses RNNs vanishing gradient issues
    • LSTM tends to poorly preserve far back knowledge
  • Attention layer access all previous states and weighs according to learned measure of relevance
    • Allows referring arbitrarily far back to relevant tokens
  • Can be addd to RNNs
  • In 2016, a new type of highly parallelisable decomposable attention was successfully combined with a feedforward network
    • Attention useful in of itself, not just with RNNs
  • Transformers use attention without recurrent connections
    • Process all tokens simultaneously
    • Calculate attention weights in successive layers

Scaled Dot-Product

  • Calculate attention weights between all tokens at once
  • Learn 3 weight matrices
    • Query
      • W_Q
    • Key
      • W_K
    • Value
      • W_V
  • Word vectors
    • For each token, i, input word embedding, x_i
      • Multiply with each of above to produce vector
    • Query Vector
      • q_i=x_iW_Q
    • Key Vector
      • k_i=x_iW_K
    • Value Vector
      • v_i=x_iW_V
  • Attention vector
    • Query and key vectors between token i and j
    • a_{ij}=q_i\cdot k_j
    • Divided by root of dimensionality of key vectors
      • \sqrt{d_k}
    • Pass through softmax to normalise
  • W_Q and W_K are different matrices
    • Attention can be non-symmetric
    • Token i attends to j (q_i\cdot k_j is large)
      • Doesn't imply that j attends to i (q_j\cdot k_i can be small)
  • Output for token i is weighted sum of value vectors of all tokens weighted by a_{ij}
    • Attention from token i to each other token
  • Q, K, V are matrices where $i$th row are vectors q_i, k_i, v_i respectively
\text{Attention}(Q,K,V)=\text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V
  • softmax taken over horizontal axis