andy
4cc2e79866
Affected files: .obsidian/global-search.json .obsidian/workspace.json Health/Alexithymia.md Health/BWS.md STEM/AI/Neural Networks/Activation Functions.md STEM/AI/Neural Networks/Architectures.md STEM/AI/Neural Networks/CNN/CNN.md STEM/AI/Neural Networks/MLP/Back-Propagation.md STEM/AI/Neural Networks/Transformers/Attention.md STEM/CS/Calling Conventions.md STEM/CS/Languages/Assembly.md |
||
|---|---|---|
| .. | ||
| Attention.md | ||
| LLM.md | ||
| README.md | ||
| Transformers.md | ||
- Attention
- Weighting significance of parts of the input
- Including recursive output
- Weighting significance of parts of the input
- Similar to RNNs
- Process sequential data
- Translation & text summarisation
- Differences
- Process input all at once
- Largely replaced LSTM and gated recurrent units (GRU) which had attention mechanics
- No recurrent structure
!
Examples
- BERT
- Bidirectional Encoder Representations from Transformers
- Original GPT
transformers-explained-visually-part-1-overview-of-functionality
Architecture
Input
- Byte-pair encoding tokeniser
- Mapped via word embedding into vector
- Positional information added
Encoder/Decoder
- Similar to seq2seq models
- Create internal representation
- Encoder layers
- Create encodings that contain information about which parts of input are relevant to each other
- Subsequent encoder layers receive previous encoding layers output
- Decoder layers
- Takes encodings and does opposite
- Uses incorporated textual information to produce output
- Has attention to draw information from output of previous decoders before drawing from encoders
- Both use attention
- Both use MLP layers for additional processing of outputs
- Contain residual connections & layer norm steps