Part III: Inference Internals & Production Serving

Arithmetic intensity (FLOPs per byte read) is proportional to the number of tokens processed in parallel. Single-token decode has AI near 1 (well below GPU compute-memory crossover), while multi-token prefill achieves much higher AI.

TTFT measures how long the user waits before seeing any response; it is dominated by the prefill phase. TPOT measures the per-token generation speed; it is dominated by decode latency and directly sets the streaming speed the user perceives.

Each of the 500 output tokens requires a separate decode step. With 200-token prefill (one pass) versus 500 decode steps, the workload is clearly decode-heavy. Maximizing batch size and reducing per-token memory access time has the most impact.

Separate prefill and decode fleets can each be optimized independently (e.g., beefy prefill GPUs, many decode GPUs). The cost is that after prefill completes, the KV cache must be transferred to the decode node, adding network latency and bandwidth pressure.

Without caching, every decode step re-runs attention over the full growing sequence. Step t processes (S + t) tokens through O(n^2) attention, summing to roughly S*T total attention work over T steps.

The attention equation is softmax(Q K^T) V. At each new decode step the query comes from the new token and attends to all previous keys and values. The old queries are never referenced again so caching them provides no benefit.

Each layer stores one K and one V matrix per token, each of shape (H, D). That is 2 x H x D values per layer per token. Multiplying by L layers and P bytes per value gives the total.

KV cache size scales as L x 2 x H x D x P per token. A 70B model has more layers L and a larger head dimension D than a 13B model. Both factors multiply directly, making the total cache size roughly proportional to the parameter count.

In memory-bound decode, the bottleneck is reading model weights from HBM. A single forward pass reads the weights once regardless of batch size, so serving N users in one pass costs 1/N of the weight-read bandwidth per user.

Every padded position requires computation (attention, feedforward) that produces no useful result. In a batch where the longest sequence is 1024 and the average is 256, roughly 75% of the compute is spent on padding.

In static or dynamic batching the batch runs until the longest request finishes. Requests that complete early leave their GPU capacity idle. Continuous batching inserts new requests at any iteration boundary, keeping the batch full and GPU utilization high.

A large prefill inserted into the batch changes the workload from memory-bound decode to compute-bound prefill for one step, stalling ongoing decode requests. Chunked prefill (splitting prefill across N iterations) keeps each iteration's compute profile stable.

A request with a 300-token completion still reserves space for max_seq_len tokens (e.g., 32768). At roughly 10 GB per max-length slot for a 70B model, even a few requests exhaust GPU memory despite using a fraction of it.

Just as virtual memory lets a process address a sparse logical address space backed by physical pages allocated on demand, PagedAttention lets each request address a logical KV sequence backed by physical GPU memory blocks allocated one block at a time.

Multiple requests can reference the same physical block via their block tables with a reference count. When a request needs to write beyond the shared prefix it gets a new private block — exactly the copy-on-write semantics from OS fork.

PagedAttention's primary win is packing more concurrent requests into the same HBM by eliminating wasted reserved-but-unused space. Higher concurrency means higher batch size, which directly improves memory-bound decode throughput.

For large S the attention matrix is huge (e.g., S=8192 -> 256M fp16 values = 512 MB). Even if FLOPs are manageable, reading and writing this tensor to HBM bandwidth-limits the operation. FlashAttention eliminates this read/write.

The online softmax algorithm (Milakov and Gimelshein, 2018) updates a running max and normalization factor tile-by-tile, allowing exact softmax without ever holding the complete S x S matrix. FlashAttention uses this to tile computation entirely in SRAM.

When a kernel is bandwidth-bound (as attention is for typical sequence lengths), the runtime is approximately bytes_transferred / bandwidth. Halving HBM traffic halves runtime even if FLOPs are unchanged.

In v1, warp-level reductions introduced synchronization overhead within each tile computation. FA2 reorganizes which dimensions each warp handles to minimize barriers and keep all warps busy, improving occupancy and reducing idle cycles.

Minimum decode latency is approximately model_size / HBM_bandwidth. Quantizing from 2 bytes per weight (BF16) to 0.5 bytes per weight (INT4) reduces model size by 4x, which directly reduces the bandwidth-bound latency floor by 4x.

Not all weights matter equally. AWQ observes that channels with large activation scales amplify quantization errors. By scaling those channels up before quantization (and compensating at inference), it reduces their effective error without changing the weight bit-width.

Activations have per-channel magnitude variation that makes uniform INT8 quantization lossy. SmoothQuant rescales each channel so the activation range shrinks and the weight range grows proportionally, making both easier to quantize to INT8 with low error.

Weight quantization errors can be measured and compensated during offline calibration on representative inputs. KV cache values depend on the specific prompt and generation context, so their dynamic range varies unpredictably, making it harder to choose safe scale factors.

The acceptance-rejection scheme uses a corrected resample that provably yields draws from the target distribution. Draft tokens whose probability ratio passes are accepted; otherwise a corrected sample is drawn from the target.

Decode is memory-bound, meaning the GPU is stalled waiting for weights from HBM regardless of how many tokens it processes. Verifying K tokens in one forward pass costs nearly the same as verifying 1, so K correct guesses means K tokens for the price of one.

EAGLE attaches a lightweight autoregressive draft head that consumes the target model's feature vectors, enabling much higher acceptance rates because the draft is conditioned on the same representation the target model uses.

Acceptance rate depends on how well the draft model predicts the target's next token. Over long generations, distribution drift accumulates, especially on creative or low-perplexity regions. The draft model's predictions become less aligned with the target.

TP's all-reduce after every attention and FFN block generates a synchronization per layer that saturates a slow inter-node link. NVLink within a node handles this cheaply; cross-node PCIe or InfiniBand makes it prohibitive.

Replica-based data parallelism scales throughput by routing extra requests to idle copies of the full model. Each replica is independent, so no inter-GPU synchronization is needed during inference, unlike TP or PP.

TP=2 within a node exploits fast NVLink to halve per-request latency. When the model is too large for one node, PP is added across nodes because it passes activations only at pipeline boundaries, tolerating higher inter-node latency.

SP shards the sequence dimension across GPUs, distributing the O(s^2) attention memory and compute. This is only meaningful when the sequence itself is so long that even after TP splitting, the activation tensors overflow HBM during prefill.

SGLang's RadixAttention organizes cached KV blocks as a radix tree. Different requests that share a common prefix share the path from root to the divergence point, while their unique suffixes branch off as separate child nodes, maximizing reuse.

LRU naturally protects a hot system prompt because it is touched by every request and thus always appears recently used. Pinning or high LRU priority ensures that the most-shared prefixes survive cache pressure from unique per-user suffixes.

A long fixed system prompt is identical across every request, guaranteeing a hit for those tokens on every request after the first warm-up. Short unique user turns add only a small unique suffix that must still be prefilled, but the large shared portion is free.

PagedAttention's block granularity means fully matching blocks are reused directly. The first diverging block triggers a copy-on-write (or is simply recomputed), ensuring correctness while maximizing the reused portion.

1000 tokens/sec * 3600 sec/hr = 3.6M tokens/hr. $3/hr divided by 3.6M tokens/hr = $0.83 per million tokens. This is the fundamental capacity-planning formula.

With reserved or owned hardware you pay for GPU-hours regardless of load. At 10% utilization you are paying 10x the per-token cost versus 100% utilization. The API charges only for tokens consumed, so it wins below the break-even utilization.

Decode is memory-bandwidth-bound at arithmetic intensity near 1 FLOP/byte. The limiting factor is how fast weights can be streamed from HBM to compute units each step, not the number of FLOPs the GPU can perform.

In the memory-bandwidth-bound regime tokens-per-second scales linearly with HBM bandwidth. 4.8 / 3.35 = 1.43x, so the H200 produces roughly 43% more decode throughput for the same model size.

Long prefill requests are compute-bound and block the scheduler. Short requests that arrive while a long prefill is in flight must wait in queue, producing high TTFT tail even though the mean (dominated by the common short case) looks fine.

Little's Law is a queueing theory identity: average queue length equals arrival rate times average sojourn time. For an LLM serving system it lets you reason about how increasing load (lambda) inflates end-to-end latency (W) as the system fills up.

Unlike a web service where request cost is roughly fixed, LLM output length is unknown until the EOS token is generated. A request generating 2000 tokens occupies batch slots for 40x longer than one generating 50 tokens, creating enormous variance in resource consumption.

8k-token prefills are expensive compute-bound operations. At 10% of traffic, they appear frequently enough to impact p95 TTFT directly by queuing shorter requests behind them. TPOT is affected less directly since decode cost depends on the decoding request's KV cache length, not the prefill request.

(336/14)^2 = 576 patch tokens. These are concatenated with the text prompt and processed together in prefill. A typical text prompt of 50 tokens becomes a 626-token sequence, making prefill more than 10x more expensive for the same text query.

Early fusion (used in LLaVA, Qwen-VL, etc.) injects image tokens directly into the token sequence. Cross-attention fusion (used in Flamingo, etc.) keeps image representations in a separate stream and adds special cross-attention layers, which can be more parameter-efficient but architecturally more complex.

Each image in a VL request adds hundreds of tokens to the prefill phase. This makes the prefill much more compute-intensive relative to the decode, sharpening the asymmetry that disaggregation exploits. The benefit scales with how much heavier prefill is versus decode.

30 frames times 256 tokens equals 7680 image tokens per request. At this scale the KV cache demand per request is enormous and directly competes with memory needed for other concurrent requests, creating severe memory pressure that text workloads avoid.

GQA reduces KV heads from 64 to 8, an 8x reduction. Since per-token KV cache size is proportional to n_kv_heads, this directly cuts the cache size by 8x, allowing 8x more concurrent users or 8x longer contexts at the same memory cost.

MQA is an extreme compression that forces all query heads to attend to the same key-value representation. Different heads can no longer specialize to different aspects of the context, which reduces model expressiveness and hurts quality on tasks requiring diverse attention.

MLA stores a single low-rank latent vector per token and reconstructs K and V at attention time. The latent dimension can be much smaller than the full KV dimension, so the per-token cache footprint is proportional to the latent rank rather than n_kv_heads times head_dim.

The standard GQA uptraining recipe (Ainslie et al. 2023) initializes each grouped KV head by mean-pooling the original MHA KV heads in the corresponding group. This preserves signal from all original heads and provides a warm start for continued training.

All expert weights must be loaded into GPU memory even though each token only uses 2 experts. When the batch size per expert is small (low load), the memory-bandwidth-per-token ratio is much worse than a dense model with the same active parameter count.

If the router consistently prefers a few popular experts, those experts become bottlenecks while others sit idle. In expert parallelism each GPU handles a subset of experts, so one overloaded expert GPU stalls the entire batch, degrading throughput.

With many large experts, storing all experts on every GPU would require enormous memory. EP assigns different experts to different GPUs and routes tokens across GPUs via all-to-all communication. TP is still used within dense layers (attention), but the FFN experts are handled by EP.

More experts with higher top-k allows each token to aggregate knowledge from a wider set of specialists, improving model quality. The challenge is that more experts spread across more GPUs increases all-to-all communication complexity and requires careful load balancing to avoid stragglers.

PI compresses positions from the extended range into the training range by dividing position indices by the scale factor. The model has seen all resulting positions during training, so it can handle them. However, compression makes nearby positions closer together in angle, reducing the model's ability to differentiate them.

YaRN's insight is that different RoPE frequency bands serve different roles. High-frequency dimensions track local token order and need interpolation at long context; low-frequency dimensions track long-range structure and should be left alone. This mixed strategy preserves quality better than applying uniform scaling.

Ring attention assigns each GPU a contiguous shard of the sequence. Keys and values are rotated around the ring of GPUs in a pipeline, so each GPU computes partial attention scores for its query shard against all key-value shards. The sequence sharding makes memory linear in the number of GPUs.

Even with context extension techniques, models trained with long-context fine-tuning have seen far fewer examples near the maximum length. Quality degrades near the edges of the context window, a phenomenon called the 'lost in the middle' effect. Advertised context length is an upper bound, not a quality guarantee.

Prefill is compute-bound at AI ~700 while decode is memory-bandwidth-bound at AI ~1. Running them on the same GPU means each phase competes for resources it does not need, reducing utilization of both. Disaggregation lets each phase run on hardware optimized for its bottleneck.

Decode must attend over all previously computed tokens. The KV cache built during prefill encodes those tokens and must transfer from the prefill GPU to the decode GPU. At hundreds of kilobytes per token for large models, this transfer cost is non-trivial and must be hidden by network bandwidth.

The benefit grows with how much prefill dominates over decode. VL workloads add hundreds of image tokens per request, making prefill extremely expensive relative to decode. Disaggregation lets prefill GPUs be heavily utilized for compute while decode GPUs are optimized for memory bandwidth, yielding large throughput improvements.

More prefill capacity means more requests complete prefill and are ready for decode simultaneously. If the decode pool has not scaled up yet, these requests queue at the decode tier, inflating TPOT and end-to-end latency even though TTFT per request improves.

Recompute cost scales with prefill compute while fetch cost scales with cache size and DRAM bandwidth. For long sequences on underpowered hardware, fetching is faster. For short sequences or compute-rich hardware, recomputing avoids the DRAM round-trip entirely.

In a multi-replica deployment, two replicas may independently compute the KV cache for the same popular system prompt. RDMA lets one replica read the already-computed cache from another's CPU DRAM or HBM over a fast network, avoiding duplicate prefill work across the fleet.

NVMe has 100-microsecond latency and gigabytes-per-second bandwidth, making it suitable for KV caches that are accessed infrequently but need to persist. Multi-turn conversations that sit idle between user messages are ideal: the cache is too large for HBM but needs to survive until the next turn.

A semantic hash maps similar content to the same bucket, allowing cache reuse even when the exact token sequence differs. This handles cases like system prompts rewritten with synonyms or slight reformatting that would miss a token-exact hash but produce nearly identical KV representations.

The naive implementation writes the full S = QK^T matrix to HBM, reads it back for softmax, then writes again. For long sequences this is terabytes of HBM traffic. FlashAttention tiles the computation so the intermediate matrix is kept in fast SRAM, dramatically reducing HBM round-trips.

Triton exposes a tile-based abstraction in Python where the programmer thinks in blocks of data rather than individual threads. This hides warp divergence, shared memory bank conflicts, and register allocation while still generating near-optimal PTX, making kernel authorship accessible to ML engineers.

torch.compile traces the computation graph and can fuse adjacent elementwise operations with the custom kernel, eliminating intermediate tensor writes to HBM between ops. The custom kernel alone executes in isolation; torch.compile integrates it into the broader operator fusion pipeline.

CUTLASS templates are highly parameterized C++ components that correspond to concepts like warp tiles, shared memory layouts, and pipeline stages. Getting correct and fast CUTLASS code requires understanding how the GPU hardware executes matmuls, making it powerful but demanding compared to Triton.