# Latency ![[latency-hero.png]] > [!summary] > Latency is the time elapsed from when a request is issued to when the result arrives back. It measures ONE operation, start to finish. The unit is time (microseconds, milliseconds, seconds). Lower is better. ## Latency vs Throughput (Contrast Pair) ![[latency-vs-throughput-bridge.png]] ^fu-20260626-1 | | Latency | [[Throughput]] | |---|---|---| | Question it answers | "How fast is ONE request?" | "How many requests per second?" | | Unit | time (ms, µs) | work / time (req/s, [[QPS]], FLOPS) | | Optimized by | CPU | GPU | | Analogy | How fast one car crosses a bridge | How many cars cross per hour | | Bottleneck | serial dependencies, cache misses | parallelism limits, memory bandwidth | | Trade-off direction | Lower latency often reduces throughput | Higher throughput often increases per-request latency | Key insight: **a system can have high throughput and high latency at the same time.** A conveyor belt of 1000 packages/hr with a 2-hour belt length has good throughput, terrible latency per package. ## CPU as a Latency Machine ![[latency-cpu-machine.png]] A CPU is architected to minimize latency for a single thread: - Out-of-order execution: reorders instructions to avoid stalls - Branch prediction: speculatively executes the likely path - Large L1/L2/L3 caches: keeps hot data close to the core - High clock frequency (3-5 GHz): shrinks the time per clock cycle On an [[A100]] / [[DGX]] box, the CPUs (AMD EPYC, 128 cores) handle orchestration, preprocessing, and any workload that needs fast round-trip responses. The GPU handles the bulk throughput. Neither does the other's job well. ## Real Numbers (A100 box reference) ![[latency-real-numbers.png]] | Operation | Typical latency | |---|---| | L1 cache hit | ~1 ns | | L3 cache hit | ~10-40 ns | | [[03-Resources/Fundamentals/DRAM]] access | ~60-100 ns | | [[HBM]] (A100 GPU) | ~100 ns | | [[NVLink]] GPU-to-GPU | ~1-2 µs | | [[PCIe]] CPU-to-GPU transfer | ~5-10 µs | | Network round-trip (LAN) | ~100 µs | | Network round-trip (WAN) | ~10-100 ms | ## Why It Matters ![[latency-why-it-matters.png]] Latency is the user-facing metric. A model inference pipeline on a DGX box can have enormous [[Throughput]] (hundreds of requests/s processed in parallel) but if one request takes 2 seconds to come back, that user experience is bad. Latency is what the human feels; throughput is what the machine measures. For real-time inference, serving, and interactive APIs, latency is the binding constraint. For batch training and data processing, [[Throughput]] is. Watch out for [[Tail Latency]]: the p99 of your latency distribution is often 5-10x the median. Optimizing for mean latency while ignoring p99 hides the worst user experiences. ## Related - [[Throughput]], the complementary metric; the two are in constant tension - [[QPS]], a throughput metric for request-serving systems - [[Tail Latency]], the slow end of the distribution; p99/p999 - [[Memory Bandwidth]], often the root bottleneck behind high latency - [[HBM]], high-bandwidth memory in the A100; reduces GPU memory latency - [[PCIe]], the bus that adds CPU-GPU transfer latency - [[03-Resources/Fundamentals/DRAM]], main memory; much higher latency than CPU caches