step 05 / 19Part I Models (optional)

S5 Inference and Serving

What this chapter covers

Reasoning models (← S4 Post-training and Reasoning) generate many thinking tokens. This chapter explains their cost and latency, and how to reduce both. Text generation is usually limited by the speed of reading weights from memory. That helps explain output-token prices, the wait before a long prompt produces its first token, and why serving systems batch requests.

After this chapter you can

Use example numbers to show whether a request’s time goes to prefill or decode, and what problem the KV cache and batching each solve. (lecture and self-check)
Explain why API output tokens cost more than input tokens, and why a long prompt makes the first token arrive later. (core)
Given a model and hardware, estimate how much memory the weights and KV cache need and the theoretical upper limit on decode speed. (core)
Tell which speedups change the output and which don’t, and name the cost of each. (advanced)
Write a minimal inference server and measure TTFT, TPOT and throughput. (advanced; the core exercise only measures TTFT and TPOT)

Lecture

A request has two phases: prefill processes the input in parallel, then decode generates the answer one token at a time. Their bottlenecks differ. Start with a complete request, then examine what determines its cost and speed.

★ One example, end to end

The chapter uses one fictional model and GPU, with round numbers for easier calculation. Real hardware and models have different values; the calculations work the same way.

	Setup
Model “Example 8B”	8 billion parameters, stored in bf16 (2 bytes per parameter)
GPU	48 GB of GPU memory; memory bandwidth 1,000 GB/s (it can move 1,000 GB out of memory per second); compute 100 TFLOPS (100 trillion floating-point operations per second)
Request	2,000 input tokens (system prompt, conversation history, question); the model answers with 500 tokens

In time order:

Load: the weights are 8 billion × 2 bytes = 16 GB. Once they’re in GPU memory, 32 GB is left.
Prefill: the 2,000 input tokens are computed together. Processing one token takes about 2 operations per parameter (one multiply, one add; ← S2 Transformers and Decoding), so the total is 2 × 8 billion × 2,000 = 32 trillion operations. At 100 trillion per second, that’s about 0.32 seconds.
First token appears: the user has waited about 0.32 seconds (not counting queueing). This wait is called TTFT.
Decode: every token generated after that requires reading the 16 GB of weights from memory once: 16 GB ÷ 1,000 GB/s = 16 ms. 500 tokens is 500 × 16 ms = 8 seconds.
The tally: reading 2,000 tokens took 0.32 seconds, 0.16 ms each on average; writing 500 tokens took 8 seconds, 16 ms each. Output tokens are 100 times slower than input tokens.

Drawn to scale: the 0.32 seconds of prefill equals exactly 20 decode steps. 500 output tokens take 500 steps, about 8 seconds; the figure shows only the beginning.

This 100× gap is the root of why API price lists charge several times more for output than for input. The real price gap isn’t 100×, because serving systems decode many people’s requests together and share the cost (see How do you serve many users at once?). It also shows why reasoning models are slow: thinking for 5,000 tokens takes 5,000 × 16 ms = 80 seconds of decode alone.

These times are theoretical lower bounds. How do you measure latency and throughput? explains why real runs are slower. First, why does a GPU capable of 100 trillion operations per second need 16 ms to decode one token?

★ What limits GPU speed?

A GPU has two limits: how fast it can compute (compute), and how fast it can move data (memory bandwidth). Whichever you hit first sets the speed.

Memory has two tiers. The weights and the KV cache live in GPU memory (HBM, often just called VRAM), which is large but takes time to move data to the compute units. Next to the compute units there’s a very small, very fast on-chip memory (SRAM) that only holds the small chunk of data currently being computed. For every computation, data has to move from HBM into SRAM first.

Arithmetic intensity: how many operations per byte moved. The example GPU can compute 100 trillion operations and move 1,000 GB per second. Dividing one by the other gives 100: you need at least 100 operations per byte moved to keep the compute busy. Do fewer and the compute units sit waiting for data; that’s memory-bound. Only above that are you compute-bound. Modern data center GPUs need several hundred operations per byte to saturate their compute.

Applying this to LLMs. In bf16 each parameter takes 2 bytes, and processing one token does 2 operations on it, so if each read of the weights processes only 1 token, the intensity is 1. If one read of the weights processes n tokens at once, the intensity is n. In other words, intensity is roughly how many tokens share one read of the weights.

Phase	Tokens per weight read	Intensity	State on the example GPU
Prefill (2,000 input tokens)	2,000	about 2,000	far above 100, compute-bound
Decode (single request)	1	about 1	far below 100, memory-bound

Decode uses only about 1/100 of the compute limit; the rest of the time it’s waiting for the weights to arrive. That’s where the 16 ms comes from: the speed is set by “weight size ÷ bandwidth” and has almost nothing to do with compute.

For one request, decode speed depends on memory bandwidth rather than TFLOPS. Two GPUs with a 2× compute difference but the same bandwidth generate text at nearly the same speed for one user. When choosing a machine to run a model locally, compare memory bandwidth first.

Going deeper: the roofline formula and what this estimate ignores

Roofline: plot both limits on one chart, with intensity on the x-axis and achievable compute speed on the y-axis. At low intensity, speed rises in proportion to intensity (the bandwidth slope); past the turning point (100 for the example GPU) it hits the ceiling (the flat compute line). Decode sits at the very bottom of the slope.
Roofline formula: achievable compute speed = min(peak compute, bandwidth × intensity). The turning-point intensity = peak compute ÷ bandwidth.
What’s ignored: the estimate above counts only the weights. Each decode step also reads that request’s KV cache (next section) and writes a small amount of intermediate results; prefill also has the computation of the attention scores themselves, which grows with the square of the input length when the input is long.
Specs aren’t reality: the official bandwidth and compute figures are peaks that real code usually doesn’t reach; quantized formats also spend a little extra time restoring values (dequantizing).
Reading official specs: compute is often quoted with sparsity or at lower precision, which makes the number several times larger than dense bf16 compute.

★ What does the KV cache store, and how big is it?

The KV cache stores intermediate results already computed for earlier text, so each decode step only processes the new token. The price is memory: for a long conversation, the KV cache can be bigger than the model weights.

Why you need it. Without a cache, writing output token 501 means recomputing all 2,500 previous tokens; writing output token 502 means recomputing 2,501. The work snowballs with length. But thanks to the causal mask (← S2 Transformers and Decoding), the keys and values of earlier tokens aren’t changed by later tokens, so they can be computed once and reused.

What it stores. Attention is split into several heads. For every layer and every head, each token stores a key vector and a value vector, each of head-dimension length. Many modern models use GQA, where several query heads share one set of keys and values, so the number of sets stored is the number of KV heads, called num_key_value_heads in config.json. Putting it together:

KV cache per token = 2 (key and value) × layers × KV heads × head dimension × bytes per value

Working it out for Example 8B. Assume it has 32 layers, 8 KV heads, a head dimension of 128, and a KV cache also in bf16 (2 bytes):

Per token: 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB, about 0.13 MB).
The example request (2,000 input + 500 output = 2,500 tokens): 2,500 × 131,072 ≈ 330 million bytes, about 0.33 GB.
A long 32K-token (32,768-token) conversation: 32,768 × 131,072 ≈ 4.3 GB.

After loading the weights, 32 GB is left. Counting only the KV cache, that holds about 97 example requests at once, or 7 long 32K conversations. If this model didn’t use GQA and had 32 KV heads per layer, the KV cache would be 4 times larger, a single 32K conversation would need about 17 GB, and 32 GB would hold only 1. That’s why S2 Transformers and Decoding covers GQA and MLA as ways to shrink the KV cache.

Decode reads the KV cache too. For every token generated, attention looks back at all the earlier text, so besides the weights, the request’s whole KV cache is read once. The longer the conversation, the more each step reads, and TPOT slowly grows. But the weights are usually still the bulk of it, so the change is much smaller than for TTFT.

The KV cache does not cache “answers.” It only stores intermediate results for one request’s own earlier text, and it’s freed when the request ends. If two users ask exactly the same question, the model still computes each separately (unless the beginnings match exactly and prefix caching kicks in; see later). Weights that fit in GPU memory do not guarantee long contexts either: you need room for the KV cache.

Before you build: estimating parameter count and KV cache from config.json

Units: KiB = 1,024 bytes, GiB = 1,024³ bytes; GB = 10⁹ bytes. Have your calculator output bytes and convert separately, so you don’t mix them up.

Fields to read for the KV cache: num_hidden_layers (layers), num_key_value_heads, head_dim (when it’s missing, usually hidden_size ÷ num_attention_heads), and torch_dtype or whichever number format you specify.

Parameter count: for Llama/Qwen-style models, add up the size of every matrix (d = hidden_size, h = num_attention_heads, h_kv = num_key_value_heads, d_h = head dimension, m = intermediate_size, V = vocab_size, L = layers):

Part	Parameters per layer
Attention q and output matrices	d × (h·d_h), one each
Attention k and v matrices	d × (h_kv·d_h), one each
SwiGLU MLP	3 × d × m
Two RMSNorms	2 × d (tiny)

Then add the embedding table V × d, the final norm, and the output layer: when tie_word_embeddings is true, the output layer shares the embedding matrix and is counted once; only when it’s false do you add another V × d. Some models have other tiny norm parameters that barely affect an estimate at the “hundreds of millions” level.

Check it with a made-up config: d = 4,096, h = 32, h_kv = 8, d_h = 128, m = 14,336, V = 128,000, L = 32, embeddings not tied.

Attention per layer: 2 × 4,096 × 4,096 + 2 × 4,096 × 1,024 = 41,943,040
MLP per layer: 3 × 4,096 × 14,336 = 176,160,768
Norms per layer: 8,192; per-layer total 218,112,000, or 6,979,584,000 for 32 layers
Embedding and output layer: 2 × 128,000 × 4,096 = 1,048,576,000; final norm 4,096
Total 8,028,164,096, about 8 billion: the size of Example 8B.

Trap: some checkpoints store a separate output layer (lm_head) in the file even though the config says tie_word_embeddings: true, for example Qwen3-0.6B. So don’t infer parameter count from file size.

MLA: models that use MLA, like DeepSeek’s, store compressed vectors, so the KV formula above doesn’t apply; you have to compute it separately from fields such as kv_lora_rank.

★ How do you serve many users at once?

If decode is stuck on reading weights, let many requests share the same read. This is the core of how serving systems save money; the hard part is that requests vary in length and KV cache memory is easy to waste.

How batching saves. Decode 32 requests together and each step reads the weights only once (16 GB), plus 32 separate KV caches. Assume each averages 0.33 GB:

Each step reads 16 + 32 × 0.33 ≈ 26.6 GB, about 26.6 ms, and produces 32 tokens at once.
Throughput goes from 62.5 tokens per second for a single request to about 1,200 per second, and the cost per output token drops to about 1/19.
The price: each person’s TPOT goes from 16 ms to 26.6 ms.

Two things cap how large it can get: the KV cache filling memory (as computed in the previous section, about 97 example requests), or the tokens processed per step growing until you become compute-bound (about 100 on the example GPU).

Static vs continuous batching. The simplest approach is to fill a batch and run it all to the end together, called static batching. The problem is uneven lengths: if 4 requests need 50, 100, 200 and 500 output tokens, the whole batch runs 500 steps, and the slots of the three that finished early sit idle. Only (50 + 100 + 200 + 500) ÷ (4 × 500) = 42.5% of the slots are actually used. Continuous batching (the iteration-level scheduling introduced by Orca) reschedules after every generated token, filling in a new request as soon as one finishes, so slots almost never sit empty.

PagedAttention: don’t reserve the KV cache in one big block. Early systems reserved one contiguous block per request, sized for the maximum length it might use. If every request reserves enough for 32K (4.3 GB), 32 GB holds only 7; but most requests actually use only 2,500 tokens, and sized by actual use 97 would fit. vLLM’s PagedAttention borrows the idea of virtual memory from operating systems: it splits the KV cache into fixed-size blocks (say 16 tokens each) and allocates a block only when it’s needed, so the only waste is the unfilled part of the last block. Multiple requests can also share blocks with identical content.

RadixAttention: share prefixes automatically. Many requests start the same way: the same system prompt, the earlier turns of the same multi-turn conversation, the tool descriptions an agent carries at every step. SGLang’s RadixAttention records every request’s prefix in a radix tree (a tree that branches by prefix), so the shared beginning has its KV cache computed only once and shared by everyone.

Common serving engines.

Engine	Good for	Highlights
vLLM	Serving many users on GPUs	PagedAttention, continuous batching
SGLang	Same, especially multi-turn conversations and agents	RadixAttention reuses prefixes automatically
llama.cpp, Ollama	Running models locally	Uses the GGUF quantized format; runs on Macs and ordinary computers

Batching raises total throughput, but each request has a slightly longer TPOT. Choose the batch size based on the cost and latency you can accept.

★ How do you measure latency and throughput?

Measure latency as two separate things, “how long until the first token” and “how fast tokens come after that,” and measure system efficiency by throughput. The three pull against each other, and your report also has to look at the slowest requests.

Metric	Meaning	Numbers from the example
TTFT (time to first token)	From sending the request until the first token appears; equals queueing + prefill	about 0.32 s (excluding queueing)
TPOT (time per output token)	Average interval between later tokens; sets how fast streamed text appears	16 ms alone, 26.6 ms at batch 32
Throughput	Tokens per second produced by the whole system; reflects cost efficiency	62.5 per second alone, about 1,200 at batch 32
Goodput	Counts only requests that meet the latency targets (SLOs)	see below

A request’s total latency ≈ TTFT + output tokens × TPOT. The example request at batch 32 takes 0.32 + 500 × 0.0265 ≈ 13.6 seconds, of which prefill is only a small part, so shortening the output usually cuts latency more than shortening the input.

How it changes with prompt length. TTFT is roughly proportional to input length: 4 times the input means 4 times the prefill work. With very long inputs, the cost of attention grows with the square of the length (each token is compared with every earlier token), so TTFT grows even faster. TPOT is insensitive to input length; it only grows slowly because each step reads more KV cache.

The gap between the theoretical limit and measurements. “Weight size ÷ bandwidth” is the upper limit on decode speed. Measurements are usually noticeably slower: real bandwidth doesn’t reach the spec, each step also reads the KV cache, each step has fixed software overhead, and quantized models have to restore values. When comparing, the ratio “measured ÷ theoretical” means more than the absolute numbers.

Goodput. Say the targets are TTFT under 1 second and TPOT under 50 ms. The system completes 10 requests per second, 3 of which miss the targets, so goodput is 7 per second. A bigger batch can make throughput look better, but more requests miss the targets and goodput actually drops.

P50 and P99. P50 is the median; P99 is the value that 99% of requests are faster than. Of 100 requests, 98 have a TTFT of 0.4 seconds and 2 have 12 seconds: the average is about 0.63 seconds, which looks fine; P50 is 0.4 seconds, but P99 is 12 seconds.

Report P50 and P99 latency as well as the average; users notice unusually slow requests. TTFT spikes often come from queueing behind long prompts in prefill, or waiting for space in a full KV cache.

★ Does speeding up change the output?

First sort speedups into two kinds: those that guarantee the same output as before (or the same probability distribution), and those that trade a little quality for speed or memory. Before using one, ask “which kind is it?” and you’ll know whether you need to re-evaluate quality.

Method	Kind	How it speeds things up	Cost
FlashAttention	Doesn’t change output	Reorders the attention computation so intermediate results stay in SRAM and less data moves; mathematically the same result	Needs a specialized kernel (a dedicated program that runs on the GPU)
Prefix caching	Doesn’t change output	Reuses the KV cache of an identical beginning, skipping that part of prefill	Takes memory; the beginning must match exactly
Speculative decoding	Doesn’t change the output distribution	A small model guesses first, the large model verifies in one pass	An extra small model to run; no gain when guesses are often wrong
Quantization	Approximate	Stores weights in fewer bits, so they read faster and take less memory	Quality may drop
Distillation, pruning	Approximate	Switch to a smaller model (distillation ← S4 Post-training and Reasoning) or remove unimportant weights	Quality may drop, and it needs retraining

Speculative decoding. This exploits exactly the fact that decode is memory-bound: one read of the weights takes about as long whether it processes 1 position or 5. So let a cheap small model (the draft model) guess the next 4 tokens, then have the target model check all 4 positions in one pass, costing about one decode step. Suppose the first 3 guesses are right: accept those 3, plus the 1 token the target model itself produces at the first wrong position, and this round yields 4 tokens that would otherwise take 4 steps. With the correct acceptance rule (see the expandable section below), the output’s probability distribution is exactly the same as using the target model alone. The higher the hit rate, the bigger the payoff; when the batch is already large and the GPU is close to compute-bound, the premise that “verification is nearly free” no longer holds and the benefit shrinks.

Quantization. Switch Example 8B to 4-bit (0.5 bytes per parameter) and the weights go from 16 GB to about 4 GB (a bit more with the scale factors). The decode limit goes from 16 ms per token to about 4 ms, and there’s more room for the KV cache. Common approaches:

GPTQ: adjusts the quantized weights layer by layer to minimize the error in each layer’s output.
AWQ: uses activations to find important weight channels (a column of the weight matrix) and protects them.
GGUF: the file format llama.cpp uses, with many quantization levels; the most common format for running models locally. Most models Ollama downloads by default are 4-bit.

After quantization, retest quality on your own eval data because the effect varies by task. The 4× speedup from 4-bit is an upper bound; measured gains are lower.

Going deeper: the speculative decoding acceptance rule and how FlashAttention works

Acceptance rule: for each guessed token x, the draft model gives it probability q(x) and the target model gives p(x). Accept with probability min(1, p(x) / q(x)); on a rejection, sample a replacement token from max(0, p − q) renormalized (so it sums to 1), and discard all later guesses. If every guess is accepted, the target model adds one more token at the last position. You can prove that every token produced this way follows the target model’s distribution p. Under greedy decoding the rule simplifies to: accept a guessed token if it equals the target model’s argmax.

Variants: you don’t need a separate model. Some approaches add a few small prediction heads to the target model to guess several tokens at once; others look for matching spans in the prompt to use as guesses (good for tasks that copy large chunks of input, such as rewriting code).

FlashAttention: the straightforward implementation writes the whole L × L attention score table back to HBM, then reads it out for the softmax and the multiplication by V, so data movement grows with the square of the length. FlashAttention splits Q, K and V into small tiles, moves only one small tile into SRAM at a time, and accumulates the result with a softmax that updates as it goes (online softmax), so the score table is never written back to HBM. The number of operations is the same; data movement drops sharply. Floating-point operations happen in a different order, so values may differ very slightly, but mathematically it’s the same result.

★ How does prompt caching work?

Two API features are inference-level mechanisms underneath: prompt caching is cross-request reuse of the KV cache, and constrained decoding restricts the candidate tokens at each decode step.

Prompt caching. An API’s prompt caching is prefix KV reuse: when the beginning of this request exactly matches some recent request, the server pulls out the stored KV cache and skips that part of prefill. If the first 1,500 of the example request’s 2,000 input tokens (system prompt and tool descriptions) hit the cache, only the remaining 500 need prefill, TTFT drops from 0.32 seconds to about 0.08 seconds, and those 1,500 tokens skip most of their computation, which is why cached input tokens are cheaper.

The cache only matches identical prefixes: change one character at the beginning (for example, putting the current time at the very top of the system prompt) and all the KV cache after it is different, so the whole thing has to be recomputed. The cache also expires after a period without use. How to arrange context to raise the hit rate → S11 Context Engineering.

Constrained decoding. At each decode step, work out from the JSON schema or grammar “which tokens are allowed right now,” set every other token’s probability to 0, and choose from the remaining candidates. For example, if the output so far is {"title": and the schema says title is a string, this step can’t pick a number or {, only a token that can start a string. So as long as generation ends normally, the output follows the grammar. The cost is a bit more computation per step, and it can push the model toward tokens it wouldn’t otherwise choose (in practice → S7 Prompts, Structured Output and Workflows).

Your code must still check for complete JSON. Even with constrained decoding, max_tokens can cut off the output, or the model can refuse.

○ What else is there to serving at scale?

Since prefill and decode behave completely differently (one compute-bound, one memory-bound), you can put them on different GPUs and tune each separately; this is called prefill/decode disaggregation (for example DistServe, Mooncake). The KV cache can also be tiered: systems such as Mooncake and LMCache move KV cache that isn’t needed right now to CPU memory or disk and bring it back when needed, which is cheaper than redoing prefill. CacheBlend lets segments that aren’t at the beginning (such as RAG retrieval results) reuse precomputed KV caches too. To write kernels like FlashAttention yourself, learn CUDA or Triton.

Exercises

Concepts route (no code): after “Try it first”, you can skip to the self-check.

Try it first (about 10 minutes)

1. GPU A: 200 TFLOPS compute, 1,000 GB/s bandwidth. GPU B: 100 TFLOPS compute, 2,000 GB/s bandwidth. Running Example 8B (16 GB of weights) for a single user chatting, which one generates text faster? Which one gets the first token out faster (2,000 input tokens, as in the lecture example)?

Check your answer

Generation (decode) depends on bandwidth: A takes 16 ms per token, B 8 ms, so B is twice as fast. The first token (prefill, 2,000 input tokens) depends on compute: A takes about 0.16 seconds, B about 0.32 seconds, so A is faster.

2. One request has 4,000 input tokens and 200 output tokens; another has 200 input and 4,000 output. Using the example’s numbers, which has the longer total latency, and why?

Check your answer

The first: prefill about 0.64 seconds + decode 200 × 16 ms = 3.2 seconds, about 3.8 seconds in total. The second: prefill about 0.03 seconds + decode 4,000 × 16 ms = 64 seconds. The second is far longer: output tokens are written one at a time, each requiring a read of the weights; input is computed all together.

Core: An inference calculator and local measurements

Goal: predict inference memory and speed with formulas, then verify by measuring, to build intuition for the numbers.

Steps

Write a calculator. Input: the model’s config.json (layers, hidden size, attention heads, KV heads, head dimension, MLP intermediate size intermediate_size, vocabulary size, whether input and output embeddings are tied tie_word_embeddings), the number format, and a context length. Output: an estimated parameter count, weight size, KV cache size per token, and total KV cache for the given length.
Add the hardware’s memory bandwidth as an input and output the theoretical upper limit on decode speed (bandwidth ÷ weight bytes read per token).
Check it with Qwen3-0.6B’s config, then pick a model you’re interested in.
Run a small model locally with Ollama (or llama.cpp), generate 128 tokens from the same prompt, repeat 20 times, and pull the prefill and decode times from the responses.
Run prompts of 4 lengths (for example about 256, 1K, 4K and 8K tokens) 5 times each and record TTFT and TPOT.

Acceptance criteria

For Qwen3-0.6B (bf16), the calculator gives a KV cache of 114,688 bytes per token and about 600 million parameters (input and output embeddings counted once).
A table: theoretical decode limit, median and P90 of measured TPOT, the ratio between them, and one sentence explaining the gap.
A chart: TTFT against prompt length; explain how TTFT and TPOT each change with prompt length, and why.

Hints

TypeScript: call Ollama’s HTTP API with fetch; the calculator itself needs no packages.
Python: requests or the ollama package; read config.json with json.
Ollama’s API returns prompt_eval_count, prompt_eval_duration, eval_count and eval_duration (in nanoseconds). Note that prompt_eval_duration only counts prompt tokens that weren’t cached, and prompt_eval_cached_count is how many were read from the cache. When measuring prefill, put the part that changes (for example a random string) at the start of the prompt: if you only change the end, the whole beginning still hits the cache. Check that prompt_eval_cached_count is 0 each run.
When using Ollama, raise the context length first (see S6 LLM APIs and Application Basics): the default may be only 4K, and the 4K and 8K prompts in step 5 will definitely hit it.
Look up memory bandwidth in the official specs: for Apple Silicon it’s on each model’s tech specs page, for NVIDIA in the GPU’s datasheet.
For quantized models (such as Ollama’s default 4-bit), a parameter isn’t 2 bytes; convert accordingly when computing the limit.
If you get stuck:
- Reread: KV cache (including the parameter count formulas in “Before you build”), What limits GPU speed?, How do you measure latency and throughput?
- Further: the “Introduction to Inference” and “KV Cache” chapters in Part I of zero-to-sglang

Advanced: Model project M5: KV cache, quantization and a mini inference server

Goal: implement the key inference components yourself on Qwen3-0.6B, and show each component’s effect with measured numbers.