# GPU and Training Hardware - Fundamentals A systems-level mental model of why ML training needs specialized hardware, what the hardware stack looks like, and how it all fits together. Written for someone with strong systems thinking but zero GPU/ML background. --- ## Why GPUs for ML? The core operation of deep learning is [[Matrix Multiplication|matrix multiplication]] -- multiplying huge matrices of numbers together, billions of times. Each multiplication is simple arithmetic, but there are millions of them, and they are **independent** of each other. This is the key insight: CPUs are designed for **complex sequential tasks**. GPUs are designed for **simple parallel tasks**. ML is overwhelmingly the latter. A single training step for a language model involves: 1. Multiply input matrix (batch of data) by weight matrix -- **forward pass** 2. Compare output to expected answer -- **loss calculation** 3. Multiply gradients backward through every layer -- **backward pass** Each step is dominated by matrix multiplication. A 7B parameter model does roughly **14 billion multiply-add operations per token**, and you process thousands of tokens per step. --- ## CPU vs GPU Architecture ![[gpu-cpu-vs-gpu.png]] | | CPU | GPU | |---|---|---| | **Core count** | 8--24 powerful cores | 10,000--16,000 simple cores | | **Core design** | Complex (branch prediction, out-of-order execution, deep cache hierarchy) | Simple (basic ALU, minimal cache) | | **Strength** | One hard task fast | Many easy tasks simultaneously | | **Analogy** | One PhD mathematician solving equations one by one | 10,000 interns who can each multiply two numbers | | **ML relevance** | Bottleneck -- data loading, preprocessing | Where 95%+ of training compute happens | The PhD solves any equation faster than any single intern. But if you need to multiply 10,000 pairs of numbers, the interns finish in one round while the PhD takes 10,000 rounds. ML training is "multiply 10,000 pairs of numbers" repeated a billion times. --- ## NVIDIA GPU Hierarchy NVIDIA dominates ML hardware. Their naming tells you what tier you are looking at: ### Consumer (GeForce RTX) For gamers and hobbyists. ML people use them for prototyping. | Model | VRAM | FP16 TFLOPS | Price | Use case | |---|---|---|---|---| | RTX 4090 | 24 GB GDDR6X | 82 | ~$1,600 | Hobbyist training, fine-tuning 7B models | | RTX 4080 | 16 GB GDDR6X | 48 | ~$1,000 | Inference, small experiments | | RTX 5090 | 32 GB GDDR7 | ~104 | ~$2,000 | Best consumer option (2025) | ### Datacenter (Tesla / A-series / H-series / B-series) For serious training. This is what cloud providers and research labs buy. | Model | Architecture | VRAM | Memory BW | FP16 TFLOPS | Key feature | |---|---|---|---|---|---| | A100 | Ampere (2020) | 40/80 GB HBM2e | 2.0 TB/s | 312 | First MIG support (split 1 GPU into 7) | | H100 | Hopper (2022) | 80 GB HBM3 | 3.35 TB/s | 989 | Transformer Engine, `FP8` support | | B200 | Blackwell (2024) | 192 GB HBM3e | 8.0 TB/s | 2,250 | Chiplet design, `FP4` support | The generational leap pattern: each new architecture roughly **3x** the previous in LLM training throughput. ### What the letters mean | Letter | Architecture | Year | Named after | |---|---|---|---| | P | Pascal | 2016 | Blaise Pascal | | V | Volta | 2017 | Alessandro Volta | | A | Ampere | 2020 | Andre-Marie Ampere | | H | Hopper | 2022 | Grace Hopper | | B | Blackwell | 2024 | David Blackwell | See [[03-Resources/GPU]] for detailed Korean-language spec comparisons across generations. --- ## VRAM: Why Memory Matters More Than Compute ![[gpu-memory-hierarchy.png]] For large models, **memory is the bottleneck, not compute**. During training, VRAM must simultaneously hold: | What | Size (7B model, `FP16`) | Why | |---|---|---| | Model weights | ~14 GB | 7B params x 2 bytes each | | Gradients | ~14 GB | Same shape as weights | | Optimizer states (Adam) | ~28 GB | 2 copies: momentum + variance, 4 bytes each | | Activations | 2--10 GB | Depends on batch size and sequence length | | **Total** | **~60 GB** | For a "small" 7B model | This is why a single A100-80GB can barely train a 7B model, and why a 70B model requires multiple GPUs just to **fit**, regardless of speed. > [!tip] Rule of thumb > `FP16` training memory (GB) ~ 8--10x the parameter count in billions. > A 7B model needs ~60 GB. A 70B model needs ~600 GB. See [[VRAM]] for deeper dive on GPU memory types and OOM debugging. --- ## Mixed Precision Training Full `FP32` (32-bit floating point) is wasteful for ML. Most operations tolerate lower precision without hurting model quality. | Format | Bits | Range | Use case | Throughput vs `FP32` | |---|---|---|---|---| | `FP32` | 32 | High | Master weights, loss scaling | 1x (baseline) | | `FP16` | 16 | Limited (can overflow) | Forward/backward pass | 2x | | `BF16` | 16 | Same range as `FP32`, less precision | Preferred for training (no overflow) | 2x | | `FP8` (E4M3) | 8 | Very limited | H100+ Transformer Engine | 4x | | `FP4` | 4 | Minimal | B200 inference | 8x | The insight: `BF16` keeps the same exponent range as `FP32` (so no overflow/underflow), but uses 8 fewer bits for the fractional part. For gradient descent, this loss of precision is almost always fine. ``` FP32: [1 sign][8 exponent][23 mantissa] = 32 bits BF16: [1 sign][8 exponent][ 7 mantissa] = 16 bits ← same range, half the bytes FP16: [1 sign][5 exponent][10 mantissa] = 16 bits ← smaller range, more precision ``` Modern training uses **mixed precision**: keep a master copy of weights in `FP32`, do forward/backward in `BF16`, accumulate gradients in `FP32`. This gives ~2x speedup with negligible quality loss. See [[Modern Training Techniques for Foundation Models]] for where mixed precision fits in the training optimization stack. --- ## DGX Systems **DGX** is NVIDIA's brand for pre-built GPU server systems. Think of it as a "gaming PC" but for AI -- NVIDIA assembles the whole thing so you do not have to. | System | GPUs inside | Total VRAM | GPU interconnect | Price | |---|---|---|---|---| | DGX A100 | 8x A100-80GB | 640 GB | NVLink 600 GB/s | ~$200K | | DGX H100 | 8x H100-80GB | 640 GB | NVLink 900 GB/s | ~$300K | | DGX B200 | 8x B200-192GB | 1,536 GB | NVLink 1.8 TB/s | ~$500K+ | Key point: the 8 GPUs inside a DGX are connected via **NVLink**, which is 5--10x faster than PCIe. This matters for multi-GPU training where GPUs constantly exchange gradients. The analogy: DGX is a Genesis (완성형 서버), the GPU model (A100/H100) is the engine type. You buy the car, not just the engine. --- ## Multi-GPU Training ![[gpu-data-parallelism.png]] When a model does not fit on one GPU, or you want to train faster, you spread the work across multiple GPUs. Two main strategies: ### Data Parallelism Each GPU holds a **complete copy** of the model. Each GPU processes a **different batch** of data. After each step, GPUs synchronize their gradients (all-reduce). ``` GPU 0: Model copy → Batch 0 → Gradients → ┐ GPU 1: Model copy → Batch 1 → Gradients → ├→ All-Reduce → Update all copies GPU 2: Model copy → Batch 2 → Gradients → ┤ GPU 3: Model copy → Batch 3 → Gradients → ┘ ``` - Scales well (near-linear speedup) - Requires model to fit on a single GPU - Communication cost: gradient synchronization after each step ### Model Parallelism (Tensor / Pipeline) The model is **split across GPUs**. Each GPU holds a different piece. **Tensor parallelism**: Split individual layers across GPUs (e.g., each GPU does 1/4 of the attention heads). **Pipeline parallelism**: Split sequential layers across GPUs (GPU 0 gets layers 0--5, GPU 1 gets layers 6--11, etc.). ``` Pipeline parallelism: GPU 0: [Layers 0-5] → GPU 1: [Layers 6-11] → GPU 2: [Layers 12-17] → GPU 3: [Layers 18-23] ``` - Required when model does not fit on one GPU - More complex, lower utilization (GPUs wait for each other) - Often combined with data parallelism (hybrid approach) --- ## GPU Memory Hierarchy: HBM vs SRAM ![[gpu-memory-hierarchy.png]] Inside a single GPU, there are two levels of memory that matter: | Memory | Capacity | Bandwidth | Latency | Role | |---|---|---|---|---| | **HBM** (High Bandwidth Memory) | 40--192 GB | 2--8 TB/s | ~100 ns | Main GPU memory (where weights, activations live) | | **SRAM** (on-chip cache) | 20--50 MB | ~20 TB/s | ~1 ns | Tiny but 10x faster scratchpad | The bottleneck for attention: the standard attention algorithm reads Q, K, V from HBM, computes attention scores, writes them back to HBM, then reads them again to compute the output. This is **memory-bound** -- the GPU compute cores sit idle waiting for data. ### Why FlashAttention Matters [[FlashAttention]] restructures the attention computation to work in **tiles** that fit in SRAM, avoiding the round-trip to HBM. It never materializes the full N x N attention matrix in HBM. ``` Standard Attention: HBM → read Q,K → compute S = Q@K → write S to HBM → read S → softmax → write to HBM → read → ... (6 HBM round-trips) FlashAttention: HBM → read Q,K,V tile → compute everything in SRAM → write final output to HBM (2 HBM trips) ``` Result: **2--4x speedup** on attention, and uses **O(N) memory** instead of **O(N^2)** for the attention matrix. This is how modern models handle 128K+ context lengths without running out of VRAM. --- ## Practical Numbers: Time and Cost | Task | Hardware | Time | Cost | |---|---|---|---| | GPT-3 (175B) training | 10,000 V100 GPUs | ~34 days | ~$4.6M | | LLaMA 2 70B training | 2,000 A100-80GB | ~21 days | ~$2M (estimated) | | Fine-tune 7B model (LoRA) | 1x A100-80GB | 2--4 hours | ~$10 on cloud | | Fine-tune 7B model (full) | 4x A100-80GB | 8--12 hours | ~$100 on cloud | | Train methylGPT (7 GPUs) | 7x A100-80GB (DGX01) | ~2 days | (internal infra) | ### Cloud GPU Pricing (2025 approximate) | GPU | On-demand ($/hr) | Spot/Preemptible ($/hr) | |---|---|---| | A100-80GB | $3.50--4.00 | $1.50--2.00 | | H100-80GB | $8.00--12.00 | $3.00--5.00 | | RTX 4090 | $0.40--0.80 | $0.20--0.40 | --- ## methylGPT Context [[methylGPT]] trains on the team's DGX01 system (8x A100-SXM4-80GB). Key hardware details: | Parameter | Value | |---|---| | System | DGX01 | | GPUs available | 8x A100-80GB (SXM4) | | GPUs used for training | 7 (1 reserved for other workloads / MIG) | | Total training VRAM | 560 GB | | Training time | ~2 days per full run | | GPU interconnect | NVLink Gen 3 (600 GB/s) | | Training precision | Mixed (`BF16` forward/backward, `FP32` master weights) | The model uses 7 of 8 GPUs with data parallelism -- each GPU holds a full copy of the model (small enough to fit) and processes different batches. The A100's MIG feature allows the 8th GPU to be partitioned for other tasks simultaneously. --- ## NVIDIA GPU Family Tree ![[gpu-nvidia-family-tree.png]] --- ## Related Notes - [[03-Resources/GPU]] -- Detailed spec comparison (Korean) - [[VRAM]] -- Deep dive on GPU memory and OOM debugging - [[Modern Training Techniques for Foundation Models]] -- Training optimization stack - [[FlashAttention]] -- Memory-efficient attention algorithm - [[methylGPT]] -- Foundation model trained on DGX01 - [[Transformer]] -- The architecture that drives GPU demand - [[Backpropagation]] -- Why gradients need memory - [[왜 AI 추론이 엣지로 가는가 (개소리)]] -- Why this training hardware stays clustered in the cloud while inference flees to low-power edge chips