# Transformer ## The Shape of the Idea The Transformer is a **neural network architecture for processing sequences** (Vaswani et al., 2017). It replaced RNNs and CNNs as the dominant architecture for language, vision, and beyond. **The one-sentence version**: A Transformer turns a sequence of tokens into a sequence of contextualized representations by repeatedly applying two operations -- **self-attention** (tokens talk to each other) and **feed-forward networks** (each token thinks alone) -- with residual connections holding everything together. ``` Architecture = Stack of identical blocks Each block: Input → [Self-Attention] → Add & Norm → [FFN] → Add & Norm → Output ↑ tokens talk to ↑ skip ↑ tokens think ↑ skip each other connection individually connection ``` ### The Dependency Graph (Not a Timeline) ```mermaid graph TB subgraph "One Transformer Block" IN["Input Embeddings<br>(batch x seq_len x d_model)"] SA["Multi-Head Self-Attention<br>'Who should I pay attention to?'"] AN1["Add & LayerNorm"] FFN["Feed-Forward Network<br>'What does this information mean?'"] AN2["Add & LayerNorm"] OUT["Output Representations"] end IN --> SA IN -->|"residual"| AN1 SA --> AN1 AN1 --> FFN AN1 -->|"residual"| AN2 FFN --> AN2 AN2 --> OUT ``` ### Original Architecture Dimensions (2017 Paper) | Parameter | Symbol | Value | What it means | |-----------|--------|-------|---------------| | Model dimension | `d_model` | 512 | Width of every representation vector | | Attention heads | `h` | 8 | Parallel attention patterns | | Key/Value dimension | `d_k = d_v` | 64 | = `d_model / h` per head | | FFN hidden size | `d_ff` | 2048 | 4x expansion inside FFN | | Encoder layers | `N` | 6 | Depth of encoder stack | | Decoder layers | `N` | 6 | Depth of decoder stack | Results: **28.4 BLEU** on WMT 2014 EN-DE, **41.8 BLEU** on EN-FR. Trained on 8 GPUs for 3.5 days. --- ## Self-Attention: The Core Mechanism ![[transformer-self-attention-qkv.png]] ### What Self-Attention Actually Does Self-attention lets every token in a sequence **look at every other token** and decide: "how relevant is that token to understanding me?" **The formula**: $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$ ### Q, K, V -- The Database Lookup Analogy (Made Precise) Think of it as a **soft database query**. Every token simultaneously acts as: | Role | Symbol | What it is | Analogy | |------|--------|------------|---------| | **Query** | `Q` | "What am I looking for?" | The search query you type | | **Key** | `K` | "What do I advertise as?" | The index/tag on each database row | | **Value** | `V` | "What information do I carry?" | The actual data in the row | **How the lookup works, step by step**: 1. Each token creates its Q, K, V by multiplying its embedding by three learned weight matrices (`W_Q`, `W_K`, `W_V`) 2. `QK^T` computes the **relevance score** between every query-key pair (dot product = similarity) 3. Divide by `sqrt(d_k)` to prevent dot products from getting too large (which would push softmax into saturation) 4. `softmax` converts scores into a probability distribution (weights sum to 1) 5. Multiply by `V` to get a **weighted blend** of all values **Concrete example**: Processing "The **animal** didn't cross the street because **it** was too tired." When encoding "it", the Query of "it" strongly matches the Key of "animal" (high dot product). So "it"'s output representation becomes a weighted blend of all tokens' Values, dominated by "animal"'s Value. ### Visual: Self-Attention as a Routing Network ``` Every token broadcasts: "Here's my Key (what I offer) and my Value (my data)" Every token also asks: "Here's my Query (what I need)" Q_it · K_the = low score → small weight Q_it · K_animal = HIGH score → large weight ← "it" attends to "animal" Q_it · K_didn't = low score → small weight Q_it · K_cross = low score → small weight Q_it · K_street = medium score→ some weight ... Output_it = 0.02·V_the + 0.71·V_animal + 0.01·V_didn't + ... (weighted sum) ``` **Everyday system analogy**: Self-attention is like a **network switch with content-based routing**. Each packet (token) carries an address label (Key) and a payload (Value). When a new packet (Query) arrives, the switch computes how well each address matches, then delivers a **weighted mix** of all payloads. Unlike a real switch, this is soft routing -- every destination contributes, just with different weights. --- ## The Attention Matrix: O(n^2) Cost ![[transformer-quadratic-scaling.png]] ### What It Looks Like For a sequence of `n` tokens, the attention scores form an `n x n` matrix: ``` Key_1 Key_2 Key_3 ... Key_n Query_1 [ 0.1 0.7 0.05 ... 0.02 ] ← row sums to 1 (after softmax) Query_2 [ 0.3 0.1 0.4 ... 0.05 ] Query_3 [ 0.05 0.15 0.2 ... 0.3 ] ... Query_n [ 0.02 0.01 0.03 ... 0.6 ] Each cell = "how much does Query_i attend to Key_j?" ``` ### The Quadratic Scaling Problem | Sequence length | Attention matrix size | Memory (FP16) | Relative cost | |-----------------|----------------------|---------------|---------------| | 512 | 262K entries | ~0.5 MB | 1x | | 2,048 | 4.2M entries | ~8 MB | 16x | | 8,192 | 67M entries | ~128 MB | 256x | | 32,768 | 1.07B entries | ~2 GB | 4,096x | | 131,072 | 17.2B entries | ~32 GB | 65,536x | **Visual for quadratic scaling**: Imagine a city where every person must shake hands with every other person. Double the population and you quadruple the handshakes. That is O(n^2). This is the **fundamental bottleneck** of Transformers. Every architectural innovation since 2017 is, in some way, fighting this quadratic wall. --- ## Multi-Head Attention: A Committee of Readers ![[transformer-multi-head-attention.png]] ### Why Multiple Heads? A single attention pattern forces one notion of "relevance." But language has many simultaneous relationships: - Syntactic: "it" → "animal" (coreference) - Positional: "it" → nearby words (local context) - Semantic: "tired" → "animal" (property attribution) **Multi-head attention** runs `h` independent attention operations in parallel, each with its own `W_Q`, `W_K`, `W_V` matrices: $\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O$ where $\text{head}_i = \text{Attention}(XW_i^Q, XW_i^K, XW_i^V)$ ### What Each Head Learns Each head operates in a smaller subspace (`d_k = d_model / h = 512/8 = 64`). Empirically, different heads specialize: - Some heads track **syntactic dependencies** (subject-verb agreement) - Some track **positional patterns** (attend to previous/next token) - Some track **semantic similarity** (synonyms, co-references) - Some track **rare but important long-range dependencies** **Spatial visual**: Picture 8 translators reading the same document simultaneously. One focuses on grammar, another on tone, another on factual claims, another on citations. Their reports get concatenated and compressed into a single summary (the `W_O` projection). ```mermaid graph LR X["Input X"] --> H1["Head 1<br>d=64"] X --> H2["Head 2<br>d=64"] X --> H3["Head 3<br>d=64"] X --> H4["...<br>d=64"] X --> H8["Head 8<br>d=64"] H1 --> CAT["Concat<br>512-dim"] H2 --> CAT H3 --> CAT H4 --> CAT H8 --> CAT CAT --> WO["W_O projection<br>512 -> 512"] WO --> OUT["Output"] ``` **Total compute is the same** as single-head attention with full `d_model` -- you just partition the dimensions. --- ## Positional Encoding: Why Transformers Need It ![[transformer-positional-encoding.png]] ### The Problem: Permutation Invariance Self-attention treats input as a **set, not a sequence**. Swap any two tokens and the attention scores just swap correspondingly. "Dog bites man" and "Man bites dog" produce identical attention patterns without positional information. RNNs get position for free (they process tokens sequentially). Transformers process all tokens in parallel, so position must be **injected explicitly**. ### Three Approaches #### 1. Sinusoidal (Original, 2017) $PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$ $PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$ Each position gets a unique 512-dim vector. Added directly to the token embedding. Different dimensions oscillate at different frequencies (like a Fourier basis), so the model can learn to detect relative distances. **Visual**: A 512-dimensional barcode where each bar oscillates at a different frequency. Position 0 and position 100 have clearly different barcodes, and the difference between positions 5 and 10 looks similar to the difference between 105 and 110. **Limitation**: Theoretically extrapolates to unseen lengths, but in practice performance degrades. #### 2. Learned Positional Embeddings (BERT, GPT-2) Just learn a separate embedding vector for each position (0 to max_length). Simple, works well, but **hard-capped** at training length. #### 3. RoPE -- Rotary Position Embedding (Modern Standard) Used in LLaMA, Mistral, GPT-NeoX, and most modern LLMs. **Core idea**: Instead of adding position to the embedding, **rotate** the Q and K vectors by an angle proportional to their position. The dot product `Q_i · K_j` then naturally depends on the **relative distance** `i - j`. Properties: - Encodes absolute position via rotation - Captures relative distance through the dot product - Decays attention with distance (built-in recency bias) - Flexible sequence length (no hard cap) **Visual**: Imagine Q and K vectors on a clock face. Token at position 5 rotates its hand to 5 o'clock. Token at position 8 rotates to 8 o'clock. Their dot product depends on the **angle between them** (3 hours apart), not their absolute positions. #### 4. ALiBi -- Attention with Linear Biases Instead of modifying embeddings, **subtract a linear penalty** from attention scores proportional to distance: ``` attention_score(i, j) = Q_i · K_j - m · |i - j| ``` where `m` is a per-head slope. Closer tokens get higher scores. Trained on 1024 tokens, extrapolates cleanly to 2048+. Trains 11% faster, uses 11% less memory than positional embeddings. --- ## The Feed-Forward Network: Token-Level Processing ### What It Does After self-attention lets tokens exchange information, the FFN lets each token **independently process** its now-contextualized representation: $\text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2$ - `W_1`: `d_model x d_ff` (512 -> 2048) -- expand - `W_2`: `d_ff x d_model` (2048 -> 512) -- compress The 4x expansion-compression pattern is consistent across nearly all transformer variants. ### FFN as Key-Value Memory (Geva et al., 2021) A powerful reframe: **each FFN acts as a learned key-value memory**. - The rows of `W_1` are **keys** that match input patterns - The columns of `W_2` are **values** that emit vocabulary distributions - `ReLU(xW_1)` computes which keys the input activates - Multiplying by `W_2` retrieves and combines the corresponding values **Finding**: Lower layers' keys match superficial patterns ("is followed by 'the'"), while upper layers match semantic patterns ("is a country in Europe"). The FFN stores factual knowledge; attention routes information. **Visual**: The FFN is like a **lookup table with fuzzy matching**. Input pattern activates multiple memory slots (ReLU selects which ones fire), and the output is a blend of the retrieved values. ### Modern Variant: SwiGLU / GLU Most modern LLMs (LLaMA, PaLM, Mistral) replace ReLU with gated variants: $\text{FFN}_{SwiGLU}(x) = (\text{Swish}(xW_1) \odot xW_3)W_2$ This adds a third weight matrix (`W_3`) as a gate. Empirically better, at the cost of ~50% more parameters in the FFN. --- ## Residual Connections + Layer Normalization ### Why Residual Connections Each sublayer (attention, FFN) computes `LayerNorm(x + Sublayer(x))`. The `+ x` part is the residual connection. **Why it matters**: Without residuals, a 96-layer transformer would need to perfectly learn a 96-step function composition. With residuals, each layer only needs to learn a **small delta** on top of what's already there. Gradients flow cleanly through the skip connections during backpropagation. **Visual**: A highway system. Residual connections are express lanes that let information skip layers entirely. Each layer is an optional side road that adds refinement. ### Pre-Norm vs Post-Norm **Post-norm** (original 2017): `LayerNorm(x + Sublayer(x))` -- normalize after the residual add. **Pre-norm** (modern standard): `x + Sublayer(LayerNorm(x))` -- normalize before the sublayer. ``` Post-norm (original): Pre-norm (modern): x x ↓ ↓ Sublayer(x) LayerNorm(x) ↓ ↓ x + Sublayer(x) Sublayer(LayerNorm(x)) ↓ ↓ LayerNorm(result) x + Sublayer(result) ``` **Why pre-norm won**: Xiong et al. (2020) showed that post-norm produces **large, unstable gradients** near the output layer at initialization, requiring a careful learning rate warmup. Pre-norm keeps gradients well-behaved from the start, enabling training without warmup and achieving comparable performance with less hyperparameter tuning. **Modern practice**: Pre-norm with **RMSNorm** (a simpler, faster alternative to LayerNorm that skips mean-centering) is standard in LLaMA, Mistral, GPT-3, etc. --- ## Softmax Attention: Why Softmax? What Breaks? ### Why Softmax `softmax` converts raw attention scores into a **probability distribution** (non-negative, sums to 1). This means: 1. Attention weights are interpretable as "how much focus goes where" 2. The output is a convex combination of values (geometrically bounded) 3. It creates sharp, peaked distributions when scores differ (winner-take-most) ### What Happens with Very Long Context As sequence length grows: - The softmax distributes weight over more tokens - Individual attention weights get diluted ("attention entropy" increases) - The model struggles to sharply attend to the right tokens in a sea of candidates - The `QK^T` values can grow large, pushing softmax into near-one-hot saturation zones This is partly why long-context performance degrades even when you solve the O(n^2) compute problem. --- ## The Three Paradigms: GPT vs BERT vs T5 ![[transformer-three-paradigms.png]] ```mermaid graph TB subgraph "Encoder-Only: BERT" BE["Bidirectional<br>Every token sees ALL tokens"] BT["Training: Masked LM<br>Predict [MASK]ed tokens"] BU["Use: Classification,<br>NER, similarity"] end subgraph "Decoder-Only: GPT" DE["Causal / Autoregressive<br>Each token sees only PAST tokens"] DT["Training: Next token prediction<br>Predict token t+1 from tokens 1..t"] DU["Use: Text generation,<br>chat, reasoning"] end subgraph "Encoder-Decoder: T5" EE["Encoder: Bidirectional<br>Decoder: Causal + cross-attention"] ET["Training: Seq2seq<br>Corrupted input -> clean output"] EU["Use: Translation,<br>summarization"] end ``` ### The Attention Mask Difference This is the structural difference that defines each paradigm: ``` Encoder-only (BERT): Decoder-only (GPT): Encoder-Decoder (T5): Full attention Causal mask Encoder: full, Decoder: causal + cross-attention to encoder 1 1 1 1 1 1 0 0 0 0 Encoder Decoder 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 0 0 | 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 | 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 | 1 1 1 1 1 1 1 1 1 1 1 1 1 1 causal cross-attn "see everything" "see only past" "encode all, decode causally" ``` ### Training Objectives | Model | Objective | Example | |-------|-----------|---------| | BERT | Masked Language Model | "The [MASK] sat on the mat" -> predict "cat" | | GPT | Causal / Autoregressive LM | "The cat sat on" -> predict "the" | | T5 | Span corruption -> reconstruction | "The `<X>` sat `<Y>` mat" -> "`<X>` cat `<Y>` on the" | ### Why Decoder-Only Won For large-scale generative AI, decoder-only (GPT-style) dominated because: 1. **Simpler architecture**: One stack, one attention pattern. Fewer design decisions. 2. **Scaling behavior**: Causal LM scales more predictably. The same architecture handles 100M to 1T+ parameters. 3. **Generality**: Next-token prediction subsumes classification, QA, summarization, translation -- all tasks can be framed as text generation with the right prompt. 4. **Training efficiency**: Every token in the sequence provides a training signal (predict next token at each position). BERT only learns from ~15% of tokens (the masked ones). --- ## Transformer vs RNN vs CNN ![[transformer-vs-rnn-vs-cnn.png]] | Property | Transformer | RNN (LSTM/GRU) | CNN | |----------|------------|----------------|-----| | **Parallelism** | Full (all tokens simultaneously) | Sequential (token by token) | Full (within receptive field) | | **Context range** | Global (any token to any token) | Theoretically global, practically limited | Local (kernel size), stacked for wider | | **Complexity per layer** | O(n^2 * d) | O(n * d^2) | O(k * n * d^2) | | **Path length** | O(1) between any two tokens | O(n) between distant tokens | O(log_k(n)) with dilated convolutions | | **Training speed** | Fast (parallelizable) | Slow (sequential dependency) | Fast (parallelizable) | | **Memory** | O(n^2) attention matrix | O(n) hidden states | O(n) feature maps | ### Why Transformers Won 1. **Parallelism**: RNNs process tokens one-by-one. Transformers process all tokens simultaneously. On GPUs with thousands of cores, this is a massive speedup. 2. **Global context**: Every token directly attends to every other token. No information bottleneck through a fixed-size hidden state. 3. **Gradient flow**: O(1) path length between any two tokens (through attention). RNNs have O(n) path length, causing vanishing/exploding gradients over long sequences. ### What Transformers Are Bad At 1. **Long sequences**: O(n^2) compute and memory. A 1M-token context costs 1 trillion attention entries. 2. **Inference cost**: Autoregressive generation is sequential (one token at a time) and memory-bound (KV cache). The architecture's training advantage (parallelism) disappears at inference. 3. **No built-in state compression**: Every past token stays in the KV cache. There is no mechanism to "summarize" or "forget" old context -- it just grows linearly. 4. **Position sensitivity**: Without explicit positional encoding, the architecture is permutation-invariant. Encoding position remains a bolted-on solution. --- ## The Inference Problem: KV Cache and Memory Bottleneck ![[transformer-kv-cache.png]] ### Why Autoregressive Generation Is Memory-Bound During generation, a Transformer produces one token at a time. For each new token: 1. Compute the new token's Q, K, V 2. Attend to **all** previous tokens' K and V (stored in cache) 3. Run through FFN 4. Sample next token 5. Append new K, V to cache The actual matrix multiply for one new token is small. The bottleneck is **loading the entire KV cache from GPU memory (HBM) into compute units (SRAM)** at every step. ### KV Cache Growth ``` Token 1: Cache = [K1, V1] → 1 entry Token 2: Cache = [K1, V1, K2, V2] → 2 entries Token 100: Cache = [K1..K100, V1..V100] → 100 entries Token 10000: Cache = [K1..K10000, V1..V10000] → 10,000 entries ``` **Memory per token per layer**: `2 * num_heads * head_dim * sizeof(float16)` = 2 * 32 * 128 * 2 bytes = 16 KB (for a 32-head, 128-dim model like LLaMA-7B) **For a 32-layer model, 4K context**: 32 layers * 4096 tokens * 16KB = ~2 GB just for the KV cache. ### How GQA/MQA Reduce KV Cache | Variant | KV heads | Cache size (relative) | Quality | |---------|----------|----------------------|---------| | MHA (Multi-Head Attention) | `h` (e.g., 32) | 1x (baseline) | Best | | GQA (Grouped-Query Attention) | `h/g` (e.g., 8) | 1/4x | Near-MHA | | MQA (Multi-Query Attention) | 1 | 1/32x | Slight degradation | GQA (Ainslie et al., 2023) is the modern standard: groups of query heads **share** a single K/V head. LLaMA 2 70B uses GQA with 8 KV heads for 32 query heads, cutting KV cache by 4x. --- ## FlashAttention: The Tiling Trick ![[transformer-flash-attention.png]] ### The Problem Standard attention materializes the full `n x n` attention matrix in GPU HBM (high bandwidth memory). For long sequences, this matrix doesn't fit, and even when it does, reading/writing it is the bottleneck. ### The Insight: GPU Memory Hierarchy ``` SRAM (on-chip): ~20 MB ~19 TB/s bandwidth ← FAST, SMALL HBM (GPU memory): ~40-80 GB ~1.5 TB/s bandwidth ← SLOW, BIG Standard attention: Compute QK^T in SRAM, write full n×n to HBM, read back, apply softmax, write again... Multiple round-trips to slow HBM! ``` ### FlashAttention's Solution (Dao et al., 2022) **Tile the computation** so that each block of the attention matrix is computed, softmaxed, and applied to V **entirely in SRAM**, without ever materializing the full matrix in HBM. Key algorithmic tricks: 1. Process Q, K, V in blocks that fit in SRAM 2. Use **online softmax** (keep running max and sum) to compute exact softmax across tiles 3. Never write the `n x n` attention matrix to HBM **Result**: Exact attention (not approximate), but: - **2-4x faster** than standard attention - **Memory**: O(n) instead of O(n^2) - Enables training on 4x-16x longer sequences - 15% end-to-end speedup on BERT, 3x on GPT-2 **Visual**: Imagine reading a massive spreadsheet. Standard attention prints the entire spreadsheet, reads it row by row. FlashAttention processes it through a small window, one block at a time, accumulating results without ever printing the whole thing. --- ## Context Length: Challenges and Solutions | Approach | How it works | Achieves | |----------|-------------|----------| | **FlashAttention** | Tile attention in SRAM, avoid HBM writes | Same length, faster | | **ALiBi** | Linear distance penalty instead of positional embedding | Extrapolation beyond training length | | **RoPE** | Rotary embeddings with interpolation/NTK tricks | Extend 4-16x beyond training length | | **Ring Attention** | Distribute attention across devices in a ring topology, overlap communication with compute | Millions of tokens (scales with device count) | | **Sliding Window** (Mistral) | Each token only attends to last `w` tokens | O(n*w) instead of O(n^2), info propagates through layers | | **Sparse Attention** | Only attend to local + strided + global tokens | Subquadratic, but misses some patterns | ### Ring Attention (Liu et al., 2023) Distributes the sequence across multiple devices. Each device holds a chunk and sends its K/V blocks around a ring while simultaneously computing attention. The communication is fully overlapped with computation. Context scales linearly with device count -- 8 devices = 8x longer context. --- ## Scaling Laws: How Big Should Models Be? ![[transformer-scaling-laws.png]] ### Kaplan et al. (2020) -- The Power Laws Loss scales as a power law with three variables: ``` L(N) ~ N^(-0.076) (model parameters) L(D) ~ D^(-0.095) (dataset size in tokens) L(C) ~ C^(-0.050) (compute budget in FLOPs) ``` Key insight: **Model width and depth don't matter much** within a wide range -- only total parameter count matters. And larger models are more **sample-efficient** (learn more per token). Original recommendation: Train very large models on relatively modest data, stop well before convergence. ### Chinchilla (Hoffmann et al., 2022) -- The Correction **Kaplan was wrong about the optimal ratio.** Chinchilla showed: > For every doubling of model size, the number of training tokens should also be doubled. The **optimal ratio** is roughly **20 tokens per parameter**. | Model | Parameters | Training Tokens | Ratio | Verdict | |-------|-----------|----------------|-------|---------| | GPT-3 | 175B | 300B | 1.7:1 | Undertrained | | Gopher | 280B | 300B | 1.1:1 | Very undertrained | | Chinchilla | 70B | 1.4T | 20:1 | Optimal | Chinchilla (70B) **outperformed** Gopher (280B) on MMLU (67.5% vs 60%) despite being 4x smaller, because it was trained on 4x more data. **Practical implication**: Most LLMs before Chinchilla were **too big and undertrained**. You get better results from a smaller model trained on more data -- and the smaller model is cheaper to serve. --- ## Mamba and SSMs: The Alternative ### What Gap SSMs Fill Transformers have two fundamental limitations: 1. O(n^2) attention during training 2. O(n) KV cache growth during inference (no state compression) **State Space Models** (SSMs) process sequences with **O(n) compute** and **O(1) state** (fixed-size hidden state, like an RNN, but parallelizable like a Transformer during training). ### Mamba (Gu & Dao, 2023) The breakthrough: **selective state spaces** -- SSM parameters become input-dependent, allowing the model to selectively propagate or forget information based on content. | Property | Transformer | Mamba | |----------|------------|-------| | Training complexity | O(n^2) | O(n) | | Inference state | O(n) KV cache | O(1) fixed state | | Throughput | Baseline | 5x higher | | Max context | ~128K (practical) | 1M+ tokens | | Quality (3B params) | Baseline | Matches 2x larger Transformer | **The tradeoff**: Mamba cannot do arbitrary token-to-token lookback (it has compressed state, not full KV cache). For tasks requiring precise recall over long contexts ("What was the 47th word?"), Transformers still win. For tasks where compressed context suffices (most generation tasks), Mamba is dramatically more efficient. **Architecture note**: Mamba removes attention AND the separate FFN block entirely. It's a single, streamlined recurrent operation. This is a more radical departure than just "efficient attention." --- ## Key Improvements Since 2017: Summary | Innovation | Year | What Changed | Why It Matters | |-----------|------|-------------|----------------| | Pre-norm | ~2019 | LayerNorm before sublayer | Stable training without warmup | | RMSNorm | 2019 | Simpler norm (no mean-centering) | Faster, equally effective | | SwiGLU FFN | 2020 | Gated activation in FFN | Better quality per parameter | | RoPE | 2021 | Rotary positional encoding | Relative position, length extrapolation | | FlashAttention | 2022 | IO-aware tiled attention | 2-4x faster, O(n) memory | | GQA | 2023 | Grouped KV heads | 4-8x smaller KV cache | | Ring Attention | 2023 | Distributed attention across devices | Million-token context | | Mamba/SSMs | 2023 | Replace attention with selective state spaces | O(n) training, O(1) inference state | --- ## Related Notes - [[Attention Mechanism - Fundamentals]] -- the prerequisite: what attention is and why it was invented, before diving into self-attention - [[ChatGPT]] - [[Machine Learning]] - [[Neural Network]] - [[Natural Language Processing]] - [[Mamba - Selective State Space Models]] -- the SSM architecture with selective state spaces that achieves linear scaling - [[S4 - Structured State Spaces for Sequence Modeling]] -- the SSM architecture that proved long-range sequences don't need attention - [[Efficient Long Sequence Processing in Neural Networks]] -- comprehensive survey of techniques for handling long sequences (FlashAttention, sparse attention, SSMs, context extension) - [[Modern Training Techniques for Foundation Models]] -- PEFT, training strategies, and efficiency techniques for training and fine-tuning transformers