Serving, session by session

Session 2 · Models — what's inside decides what's slow

Inference engineering makes models faster and cheaper without sacrificing quality — and both halves of that demand intuition for what's under the hood. The architecture is the workload.

Why this session exists: every serving number you'll ever chase is downstream of architectural facts. TTFT is long because attention over the whole prompt is expensive; decode drips because weights must be re-read per token; a flag like --kv-cache-dtype fp8 only makes sense once you know what the KV cache physically is. This session opens the box: how a transformer computes, where its bytes live, and the one ratio — ops:byte — that locates every bottleneck.

Intuition. A neural network is an assembly line: identical simple stations, arranged in stages, each transforming the part before passing it on. No station is smart; the arrangement is.

Mechanism. The unit is a node: multiply input by learned weights, add a bias, pass it on. Nodes group into layers; a layer's outputs — its hidden states — feed the next layer. The workhorse operation is the matmul (matrix multiplication): an input vector times a weight matrix, plus bias — that is a linear layer. Because chained matmuls collapse into one, each is followed by an activation function, a cheap non-linearity that lets depth mean something. Networks compose: an encoder turns raw input into an internal representation; a decoder generates output from one. Modern LLMs are decoder-only; encoder–decoder survives in other modalities (Whisper encodes audio, decodes text).

An LLM stacks three parts: an embedding layer turning tokens into vectors; dozens-to-hundreds of transformer blocks (each: an attention sublayer, a feed-forward network, normalization); and the LM head, an output layer producing logits — one score per vocabulary token. Sampling turns logits into a token, steered per request by temperature (reshape scores), top-k / top-p (restrict the candidate pool); temperature 0 or top-k 1 makes selection deterministic. Generation loops until the model emits the stop token or hits a length limit. All of today's generative LLMs are causal language models — next-token predictors over what came before.

Trade-offs. Two facts to keep. The feed-forward networks are the bulk of the parameters, attention the second-largest — but attention is where inference difficulty concentrates. And architecture is destiny for tooling: a model's config.json names its architecture (read Qwen3MoeForCausalLM as family Qwen, generation 3, mixture-of-experts, causal LM), and engines optimize architectures, not checkpoints — sizes, base/instruct variants, and LoRA fine-tunes of one architecture all inherit the same optimized paths. Session 1's "prefer popular architectures" advice, mechanized.

Intuition. In "I decided to write a source because I thought it would be easy, but it was actually hard," you resolve "it" to writing-a-source without thinking. Attention gives the model that skill: a mechanism for scoring how much every token should influence the current one.

Mechanism. Scaled dot-product attention takes three matrices derived from the sequence: Q (queries) for the token being processed, K (keys) and V (values) representing all prior tokens; scores Q against K, normalizes, and mixes V accordingly. Attention sublayers are multi-head — several attention operations in parallel, each free to learn a different relationship type. LLMs use causal self-attention (Q, K, V from one sequence, no peeking ahead); image and multimodal models add cross-attention (Q conditioned on another sequence, like a text prompt). Because each token attends to every previous token, attention is quadratic in sequence length — except in practice decode is linear, thanks to the KV cache: stored K and V for every previous token, built during prefill, appended and re-read every decode step, resident in GPU memory. Session 1 promised its definition; this is it.

Trade-offs. Even linear gets expensive, and attention is delicate — small numerical errors compound across tokens — so optimization splits into two families. Implementation improvements keep the math exact but execute it better: FlashAttention restructures attention to slash memory reads/writes, hand-fused per GPU generation (the H100 kernel is not the B200 kernel) — tens of thousands of lines replacing a five-line algorithm, most valuable in compute-bound phases; PagedAttention stores the KV cache in fixed-size blocks addressed through a lookup table, so it no longer needs one contiguous memory slab (session 9 pages through it). Algorithmic changes trade a little quality for better complexity, e.g. sliding-window attention attending only to the last w tokens — O(N·w) instead of O(N²), with w commonly 8–32K.

Intuition. Instead of one enormous generalist brain, a hospital: many specialists and a triage desk that routes each case to a couple of them.

Mechanism. MoE (mixture of experts) makes the feed-forward layers sparse: hundreds of smaller expert matrices replace one giant one, and a tiny learned router picks a few experts per layer, per token. Qwen3-235B-A22B activates 22B of its 235B parameters per token — the router choosing 8 of 128 experts at each of 94 layers. The label active parameters (the A22B) is the per-token compute cost; total parameters is the memory cost.

Trade-offs. MoE shines for single-request local inference — frontier knowledge at small-model per-token compute. But in batched production serving, different requests light up different experts, so expect nearly all parameters hot unless you run expert parallelism at serious scale (sessions 5 and 10). Dense models stay efficient below ~32B parameters, and narrow-domain models gain little — the whole model is effectively one expert.

Intuition. A chef who chops far faster than the runner restocks the pantry spends the day waiting. Whether a kitchen is knife-limited or pantry-limited isn't philosophy — it's a ratio you can compute.

Mechanism. A GPU has a compute speed (operations/second) and a memory bandwidth (bytes/second); their quotient is the hardware's ops:byte ratio — e.g. an H100 in FP16: 989 teraFLOPS against 3.35 TB/s ≈ 295 operations per byte moved. An algorithm has an arithmetic intensity: total operations it performs divided by total bytes it moves. Compare the two on a roofline model: intensity below the hardware ratio and you're pinned to the diagonal bandwidth ceiling (memory-bound); above it, the flat compute ceiling (compute-bound).

Trade-offs — why the phases differ. Prefill loads the weights once and runs huge matrix–matrix multiplications over the whole prompt: mountains of math per byte → high intensity → compute-bound. Decode re-reads the weights (and the growing KV cache) for every single token, doing only slim vector–matrix work each step: low intensity → memory-bandwidth-bound. Session 1 asserted this asymmetry; the roofline proves it. It also explains batching's magic: a batch of 32 decode steps re-uses each loaded weight byte 32 times, multiplying intensity and sliding decode rightward toward the roof. Image and video generation, which attend over the entire latent object every denoising step, sit with prefill on the compute side.

Apply to your code

Your ablation is a roofline tour. PROFILE=worst runs decode at batch 1 — deepest into memory-bound territory, which is why its tokens/sec is a fraction of hardware peak. Each step up (baseline → best, 64 → 256 sequences) raises decode's arithmetic intensity — the mechanism behind your repo's 22.5× batch-32 result. quant's FP8 weights + FP8 KV halve the bytes in decode's denominator: same math, half the memory traffic, intensity doubles. And your L4's much lower memory bandwidth vs an H100 means its knee sits differently — session 3 computes it.

New terms this session

arithmetic intensity

Operations performed per byte of memory moved by an algorithm; compared against hardware ops:byte to locate the bottleneck.

attention (Q, K, V)

Mechanism scoring how much each prior token (keys/values) should influence the current one (query); multi-head runs several in parallel.

causal language model

Predicts the next token from previous tokens only; all current generative LLMs.

embedding layer / LM head

Input layer mapping tokens to vectors; output layer mapping hidden states to per-token logits.

FlashAttention

Exact-attention kernel family minimizing memory reads/writes, hand-fused per GPU generation; biggest wins in compute-bound phases.

logits · temperature · top-k · top-p

Per-token scores from the LM head, and the sampling dials that reshape or restrict them before a token is drawn.

matmul / linear layer / activation

Matrix multiply of input by learned weights (+bias); the non-linearity between them that makes depth meaningful.

MoE · expert · router · active parameters

Sparse feed-forward design: a learned router activates a few expert matrices per layer per token; active params = per-token compute, total params = memory.

ops:byte ratio

A GPU's compute speed divided by its memory bandwidth — the knee of its roofline (H100 FP16 ≈ 295).

PagedAttention

KV cache stored in fixed-size blocks addressed via a lookup table, eliminating the need for contiguous memory.

roofline model

Chart of achievable performance vs arithmetic intensity: diagonal bandwidth ceiling, flat compute ceiling.

sliding-window attention

Attend only to the last w tokens: O(N·w) cost, small quality trade.

stop token

Special token whose generation ends the output sequence.

transformer block

The repeated hidden unit of an LLM: attention sublayer + feed-forward network + normalization.

Session 3 · Hardware — the spec sheet is a roofline in disguise

GPUs are throughput machines: thousands of simple workers doing one uniform operation on independent data. Every serving number you will ever hit is bounded by three lines on a spec sheet — FLOPS, gigabytes, and bytes per second.

Why this session exists: session 2 gave you the demand side (arithmetic intensity); hardware is the supply side. A CPU is a master craftsman — brilliant at complex sequential work; a GPU is a factory floor — tens to hundreds of thousands of threads doing simple math in lockstep, which is exactly what matmul-shaped inference wants. Inference engineers work above the silicon, but choosing and sizing accelerators well requires a mental model of what's inside the box. (One scoping note: the material, and this stage, focus on NVIDIA datacenter GPUs — the market's center of gravity — rented in the cloud from hyperscalers or neoclouds (GPU-first providers like CoreWeave or Nebius); on-premise and air-gapped (physically isolated) deployments exist for enterprises and governments.)

Mechanism — compute. A GPU is organized into Streaming Multiprocessors (SMs), each containing three kinds of engines: CUDA cores for scalar math, Tensor Cores for vector/matrix math, and SFUs (special function units) accelerating operations like sin and log (softmax leans on them). For inference, the number that matters is Tensor Core throughput, because Tensor Cores execute MMA — matrix multiply-and-accumulate (A×B + C), the primitive under every layer. Two spec-sheet traps: quote dense FLOPS, not sparse — the sparse figure assumes 2:4 structured sparsity (half the values are zero, skippable), roughly doubling the number, and default inference is dense; and compare FLOPS at identical precision, since throughput roughly doubles each time precision halves (FP16 → FP8 → FP4).

Mechanism — memory. On-package sits VRAM — the GPU's main memory, todayHBM (high-bandwidth memory, generations HBM3/3e/4), a form of DRAM measured in tens–hundreds of GB. On-chip sits a small amount of much faster SRAM as caches: L0 per Tensor Core, L1 per SM, L2 shared across SMs (an H100: 256 KB L1 per SM, 50 MB L2 total). Memory bandwidth is the peak rate VRAM feeds the cache hierarchy — the denominator of session 2's ops:byte ratio.

Trade-offs — the buying rules. Three rules fall straight out of the roofline. For compute-bound work (prefill-heavy, image/video generation): buy FLOPS. For decode tokens/second at low-to-medium batch: buy bandwidth — the H200 beats the H100 for TPS on identical compute because it moves 4.8 vs 3.35 TB/s. For fitting at all: VRAM must hold the weights plus at least ~50% headroom for KV cache (more for long context, big batches, or video), or you'll meet the OOM (out-of-memory) error — at load time if the weights don't fit, mid-flight if the headroom doesn't.

Mechanism. GPU names decode as letter = architecture generation (named for scientists, refreshed every 1–2 years), number = size within it. Working engineers live in the 3–5 newest generations; Turing (T4) and Ampere (A10, A100) linger in legacy corners, while today's deployments run Lovelace, Hopper, or Blackwell:

Trade-offs. Each generation ships both raw gains and inference features — and kernels chase architectures: FlashAttention 3 exploits Hopper's asynchrony, FlashAttention 4 leans on Blackwell's async tiling; new silicon takes about a year to fully saturate with optimized software, so judge new architectures on your own benchmarks, not launch slides. Lovelace's missing NVLink is its defining constraint: L-series GPUs can't do efficient multi-GPU tensor parallelism, only slower schemes — one reason your L4 world is single-GPU by design. Two more pieces of the map: Grace (and now Vera) are NVIDIA's ARM CPUs paired with GPUs on superchips (GH200, GB200), whose NVLink chip-to-chip link moves up to 900 GB/s between CPU and GPU memory — several times PCIe — making CPU-side offload of KV caches and LoRA weights actually fast. And Rubin, the 2026 generation, bakes this stage's ideas into silicon: HBM4 for the decode-bound, plus the Rubin CPX — a separate chip specialized for compute-bound prefill. Disaggregation, in hardware.

Mechanism. The atomic rental unit is the instance: GPU(s) plus host CPU, host RAM, storage, networking, and interconnect — and any of the six can be your bottleneck or failure point, not just the GPU. Even "the same GPU" varies: the SXM board form factor outperforms PCIe cards (A100 SXM carries ~5% more memory bandwidth), and providers assemble instances to their own tastes — read the fine print. Scaling up: the standard multi-GPU unit is the node — eight GPUs joined by NVLink (point-to-point, ~900 GB/s per GPU on Hopper, ~1,800 on Blackwell) coordinated through NVSwitch (all-to-all). Between nodes, the standard is InfiniBand at up to 400 Gb/s per NIC — the fastest node-to-node fabric, yet an order of magnitude below NVLink; not every cloud provides it, so verify. Above nodes sit rack-scale systems: the GB200 NVL72 fuses 72 Blackwell GPUs + 36 Grace CPUs into one NVLink domain — the machine behind session 1's Dynamo numbers.

Trade-offs. That bandwidth staircase — NVLink ≫ InfiniBand ≫ Ethernet — is the terrain map for every parallelism and disaggregation choice in sessions 5 and 10: techniques that chat constantly between GPUs must live inside NVLink; techniques that ship data occasionally can cross InfiniBand. And when the problem inverts — GPU too big for the model — MIG (multi-instance GPU) splits an A100/H100/H200/B200 at the hardware level into up to seven slices (an H100's 132 SMs divide into 7 compute slices; a 3-slice MIG gets ~3/7 of compute and up to half the VRAM). For a 3B-parameter model, two MIG fractions often beat one whole GPU on efficiency.

Every challenger bets on one edge: extreme memory bandwidth for decode (Cerebras wafer-scale, Groq's SRAM-based LPU), power efficiency (Furiosa, Qualcomm), or platform integration (Google TPU, AWS Inferentia/Trainium, plus AMD's MI-series as the nearest like-for-like rival). All share three uphill battles: rebuilding the software stack without CUDA, manufacturing at the frontier, and distribution. Competition is good for you — more supply, more leverage. At the other extreme, local (edge) inference runs on the user's own device: zero network latency, offline independence, privacy, and zero marginal serving cost — against weak hardware, thermal limits, a fragmented support matrix, and battery drain. Desktop-land shows the capacity-vs-speed fork neatly: an RTX 5090 has 32 GB of fast memory; an Apple M3 Ultra offers 512 GB of unified memory, slower but vast. Build for the median user's device, not the enthusiast's.

The choices above compress into one guard-clause tree:

Apply to your code

Your entire ablation lives inside Fig S3.2's default box. TENSOR_PARALLEL_SIZE=1 isn't a choice so much as Lovelace physics — no NVLink, no efficient tensor parallelism. gpu_memory_utilization 0.90 → 0.92 is the headroom dial from concept 1: each point reclaimed is KV room, i.e. concurrency. Dropping max_model_len caps the worst-case KV a single request can claim — the same lever from the VRAM-sizing rule. And the quant profile's FP8 weights halve the bytes in the napkin's denominator: your single-stream ceiling roughly doubles, and every batched token rides cheaper memory traffic. If you ever outgrow the L4, the tree says your next stop is a single H100-class GPU — not two L4s.

New terms this session

dense vs sparse FLOPS

Raw matrix throughput vs the ~2× marketing figure assuming 2:4 structured sparsity; inference defaults to dense.

HBM / VRAM · L1 / L2 cache

The GPU's high-bandwidth main memory (DRAM, GBs), and the small fast on-chip SRAM tiers per-SM and shared.

InfiniBand / NIC

The standard node-to-node fabric (up to 400 Gb/s per network interface controller) — fastest between machines, ~10× slower than NVLink.

instance

The cloud's atomic GPU rental: GPU(s) + host CPU, RAM, storage, network, interconnect — any component can bottleneck.

MIG

Multi-instance GPU: hardware partitioning of one big GPU into up to seven isolated fractional instances.

MMA / Tensor Core / CUDA core / SFU / SM

Matrix multiply-accumulate, the engines that run it, the scalar and special-function units beside them, and the streaming multiprocessor that houses them all.

node

The standard multi-GPU unit: eight GPUs joined by NVLink/NVSwitch.

NVLink / NVSwitch

Point-to-point GPU interconnect (~0.9–1.8 TB/s per GPU) and the all-to-all switching layer over it.

NVL72

Rack-scale system fusing 72 GPUs + 36 Grace/Vera CPUs into one NVLink domain.

OOM

Out-of-memory failure — at load if weights don't fit, mid-flight if KV headroom runs out.

SXM vs PCIe

GPU board form factors; SXM variants carry higher bandwidth and power.

superchip / Grace / Vera / unified memory

CPU+GPU packages with a ~900 GB/s CPU↔GPU link (fast KV/LoRA offload); Apple's variant pools one large memory for both.

Session 4 · Software — the ladder from CUDA to Dynamo

Hardware iterates in years; software in weeks. NVIDIA's moat is as much CUDA's ecosystem as silicon — and your job lives near the top of a four-rung abstraction ladder you must understand all the way down.

Why this session exists: most inference engineering is configuring engines and orchestrating GPUs, not writing GPU code. But every flag you set is a handle on machinery one or two rungs below, and debugging without a mental model of the adjacent rungs is guesswork. The ladder: CUDA (talk to the GPU directly) → deep-learning frameworks (Python over CUDA) → inference engines (configurable, pre-optimized serving) → Dynamo (orchestrate engines at datacenter scale). Key stewards to know: NVIDIA (CUDA up to Dynamo), Hugging Face (the model registry, transformers/diffusers, safetensors), the Linux Foundation (PyTorch, vLLM), LMSYS Org (SGLang).

Mechanism. CUDA is NVIDIA's platform and programming model for parallel GPU work, best understood by parts: a CUDA kernel is a function that runs parallelized on the GPU (whenever you read "kernel," substitute "a piece of code written for the GPU"); a CUDA graph is a recorded DAG of kernels and operations, replayable to strip launch overhead from repeated workflows; the driver and runtime are the low-level and developer-facing interfaces beneath and beside them. CUDA isn't a language — kernels are written in C++ and compiled (via nvcc) into CPU + GPU code. Nobody starts from zero: BLAS, the decades-old linear-algebra specification, arrives as cuBLAS (its most-used routine, GEMM — general matrix–matrix multiply — powers every linear layer), cuDNN supplies neural-net primitives, template libraries like CUTLASS and CuTe give building blocks for high-performance kernels (FlashAttention 3 is built on CUTLASS), and FlashInfer packages optimized attention and sampling kernels for LLM serving.

Trade-offs. You will probably never write a kernel — but kernel selection is real work. Kernels are welded to hardware, with values hard-coded against a specific GPU's bandwidth and Tensor-Core layout: an H100 kernel underuses a B200; a B200 kernel may not run on Hopper at all. Frameworks pick kernels automatically, but experts swap plugins — e.g. DeepGEMM, DeepSeek's FP8 GEMM kernels for Hopper, dropped in for one hot matmul shape (and needing a plan when you change GPUs). The signature memory trick is kernel fusion: two back-to-back kernels waste a write-then-read round trip through VRAM, so re-implement them as one. Fusing multiply-by-2 with multiply-by-3 into multiply-by-6 halves the memory traffic; real fusions combine matmul + bias + activation, and during bandwidth-bound decode every avoided round trip is speed. Compilers fuse the easy cases automatically; the sophisticated ones (FlashAttention) are fused by hand.

Mechanism. PyTorch — created at Meta, now Linux Foundation — is the industry-standard Python package for tensor operations: write performant CPU/GPU code in Python, drop to CUDA plugins only where needed; its autograd makes it dominant for training, and torch.compile makes it serious for inference, performing automatic kernel selection and fusion targeted at a specific GPU. (TensorFlow has faded; JAX remains a sharp-edged research favorite.) Model weights ship as safetensors, Hugging Face's format that stores only tensor data — no executable code on load, unlike legacy pickle-style formats — memory-mapped for fast, safe loading across dozens of shard files. ONNX instead bundles weights with an execution graph for portability, feeding runtimes like ONNX Runtime or NVIDIA's TensorRT compiler (still strong for image/video models). Hugging Face's transformers and diffusers are reference implementations — perfect for reading a model's input/output spec or notebook tinkering, wrong for production serving.

Trade-offs. torch.compile cannot fuse plugin kernels like FlashAttention or DeepGEMM — and LLM serving is mostly plugin kernels — so its sweet spot is rare architectures and long chains of lightweight ops, not mainstream LLMs. Hence the industry's bifurcation: hand-written compiled PyTorch for control, or a prebuilt engine for convenience, with weights increasingly shipped safetensors-only, skipping intermediate representations entirely.

Mechanism. An inference engine packages the whole runtime layer — scheduler, batching, KV management, optimized kernels, an OpenAI-compatible server — behind flags. Three compete seriously, all Apache 2.0, all shipping continuous batching, quantization, speculation, prefix caching, parallelism, and disaggregation out of the box:

Trade-offs. vLLM (2023, UC Berkeley → PyTorch Foundation) trades peak performance for breadth — the fastest path to a solid server for almost any open model, and the sane choice on smaller or older GPUs where compiled engines buy little. SGLang (Dec 2023, LMSYS) pairs a fast backend with a composable frontend, partners directly with DeepSeek/Qwen/Kimi on features like multi-latent attention, and has invested hardest in giant-MoE, multi-node NVL72-class serving; its RadixAttention automatically re-uses cached prefixes across requests — decisive when your traffic shares openings. TensorRT-LLM wins raw performance via NVIDIA-engineered kernels — but mind the versions: 0.x is the old TensorRT plugin, 1.x (summer 2025) is a standalone PyTorch-based engine — and demands the most engineering. One principle governs all three: more constraints, more performance — the narrow tool beats the broad one when you can afford its constraints.

NVIDIA Dynamo (announced GTC, March 2025; Apache 2.0) orchestrates any of the three engines across a datacenter: KV-cache-aware routing (send a request to the worker already holding its prefix), prefill/decode disaggregation with independent scaling, and multi-node parallelism for the largest MoE models — plus an SLA-based planner that scales prefill and decode workers against your declared TTFT and TPS targets in real time. The rule of thumb the material gives: the more scale you have, the more techniques become available — and the inverse: below the traffic volume where disaggregation and KV-routing pay for themselves, Dynamo is overhead; use engines directly. This is the software behind session 1's GB200 numbers.
github.com/ai-dynamo/dynamo Dynamo 1.0 (Apr 2026)

Without precise benchmarks, "optimization" is a mood. Gold standard: shadowing — mirroring live production requests onto the test system. When you must simulate instead, match production on four axes: sequence lengths (ISL/OSL, input and output token counts — TTFT and memory ride on ISL), traffic volume and pattern (concurrency, with jitter — randomized arrival timing — to mimic reality), request contents (they drive cache hit rates and speculative-draft acceptance), and inference parameters (temperature, reasoning effort) at production values. Tools: SGLang's genai-bench, NVIDIA GenAI-Perf, Locust for load. And borrow eval datasets (MMLU, GSM8K, HumanEval, SWE-bench) as benchmark inputs — they double as varied realistic traffic and a quality spot-check that your optimizations didn't dent the model. Maximize against unrealistic inputs and production will disappoint you on schedule.

Apply to your code

Your stack is this ladder, literally: PyTorch under vLLM, vLLM under vllm serve, Modal standing in for the rung Dynamo occupies at larger scale. enforce_eager=True in worst disables CUDA graphs — you're paying launch overhead every decode step, deliberately; best re-enables them (concept 1's replay trick) and adds async_scheduling, the engine preparing batch N+1 while N runs. Day-0 Gemma support is vLLM's breadth thesis in action. And your benchmark scripts should now be audited against concept 5: fixed ISL/OSL sweeps, jittered arrivals, and an eval spot-check after the quant profile — matching production shape is the difference between a number and a truth.

New terms this session

BLAS / cuBLAS / cuDNN / GEMM

The classic linear-algebra spec, its CUDA implementation, the neural-net primitive library, and the general matrix–matrix multiply at the heart of every linear layer.

CUDA kernel / graph / driver / runtime

A GPU function; a recorded, replayable DAG of GPU work (kills launch overhead); and the low-level vs developer-facing interfaces beneath them.

CUTLASS / CuTe / FlashInfer / DeepGEMM

Template libraries for building high-performance kernels; a packaged library of LLM-serving kernels; DeepSeek's FP8 GEMM kernels for Hopper.

Dynamo

NVIDIA's engine-agnostic orchestration layer: KV-aware routing, disaggregation, multi-node parallelism, SLA-driven scaling.

in-flight batching

TensorRT-LLM's name for token-level continuous batching.

ISL / OSL · jitter · shadowing

Input/output sequence lengths; randomized request timing; mirroring live traffic onto a test system — the anatomy of an honest benchmark.

kernel fusion / kernel selection

Merging adjacent kernels to delete memory round trips; choosing among hardware-specialized kernel implementations.

ONNX / ONNX Runtime / TensorRT

A portable format bundling weights + execution graph, and the runtimes that execute or compile it.

RadixAttention

SGLang's automatic prefix re-use: cached KV shared across requests with common openings via a radix-tree index.

safetensors

Weights-only, memory-mapped serialization — no executable code on load; the dominant model file format.

torch.compile

PyTorch's compilation step: automatic kernel selection + fusion targeted at a GPU; can't fuse plugin kernels, so limited for mainstream LLMs.

Session 5 · Techniques — the five families of go-fast

Inference is one of the rare fields where a paper can be in production within weeks. This session is the arsenal: quantization, speculation, caching, parallelism, disaggregation — and the judgment about when each one pays.

Why this session exists: sessions 2–4 built the physics and the tools; this is the applied research that exploits them. Two governing principles from session 1 sharpen here. Constraints buy performance — disaggregation literally works by constraining each engine to one phase. And a new corollary: the more traffic you have, the more optimizations you can afford — KV-aware routing, high parallelism, and dynamic disaggregation only pencil out across many GPUs. Techniques also interact: quantizing the KV cache relieves a disaggregation bottleneck (symbiosis), while big batches starve speculation of the spare compute it feeds on (conflict). Finding the balanced set takes patient experimentation — the material recounts a Baseten engineer scripting through 77 configurations before a non-obvious combination doubled a customer's TPS.

Intuition. Round π to 3.14 and cube it: 30.96 instead of 31.01. Round to 3 and cube it: 27. Small precision losses compound through repeated math — and inference is nothing but repeated math. Quantization is the art of shrinking numbers without letting the rounding echo.

Mechanism. Models train in a native format — usually BF16/FP16. Post-training quantization (PTQ) converts finished weights (and optionally more) to a lower-precision format; quantization-aware training (QAT) bakes low precision in during training (GPT-OSS shipped in MXFP4, Kimi K2 Thinking in INT4) — but with open weights, PTQ is your lever, via tools like NVIDIA's ModelOpt whose outputs run on all three engines. Halving precision helps both phases: prefill gets Tensor Cores with ~2× FLOPS, decode moves half the bytes — though overheads mean each step down buys ~30–50% in practice, not 2×. A format is defined by precision (bits), type (integer vs floating point), and a scale factor mapping values back up; together these set dynamic range (smallest-to-largest representable value) and granularity (how many values share one scale factor). Floating point beats integer for inference because its exponent bits (an FP8 E4M3 value = sign + 4 exponent + 3 mantissa bits) preserve the outliers that matter. Granularity climbs from per-tensor to per-channel to per-block: the Blackwell-era microscaling formats put a scale factor on every 32 values (MXFP8/MXFP4), and NVFP4 tightens to blocks of 16 plus a global scale — Blackwell applies scale factors inside the Tensor Cores to offset the materialkeeping. (Local-inference land uses GGUF's dynamic mixed-integer quants down to ~1.58-bit average — heroic for laptops, wrong for quality-sensitive production.)

Trade-offs. Sensitivity is a ladder: linear-layer weights (least sensitive) → activations → KV cache (moderate — its errors compound token-to-token) → attention itself (most sensitive; softmax stays native in all but the most aggressive schemes; first and last layers are often skipped too). Quality gates are non-negotiable: compare perplexity (how "surprised" the model is by known-good text — want no meaningful rise), a public benchmark, and your own evals against the original weights, accepting only noise-level deltas — the production standard is zero perceptible loss. FP8/MXFP8 is today's sweet spot; NVFP4 is the promising frontier; and quantization is a dial, not a switch — weights-only at FP8 is a gentler setting than everything-at-FP4. If your domain can't risk quality at all, relax: every other technique in this session is lossless.

Intuition. Solving a sudoku is hard; checking a filled-in one is easy. For an LLM, generating a token is solving; verifying a proposed token is checking. Decode leaves compute idle while weights stream from memory — speculation spends that idle compute on cheap guesses the model then checks in bulk.

Mechanism. A speculator produces draft tokens; the target model (the one you're accelerating) validates them in a single forward pass, accepts the correct prefix, and generates one token of its own: N accepted drafts ⇒ N+1 tokens per pass. Uplift rides on three factors — draft cost, draft length, and acceptance rate — with acceptance decaying along the draft (one rejection discards everything after it), so aim for short, high-confidence sequences. Higher temperature hurts (distributions get harder to guess), subject matter shifts acceptance, and speculation improves TPS/ITL only — never TTFT. The algorithms: draft–target pairs a small same-family model (≥10× smaller; fine-tune/distill it toward the target for better acceptance) — easiest to adopt, most overhead (a second model's weights, KV, and orchestration). Medusa grafts 2–4 extra decoder heads onto the target — historically important, rarely used now. EAGLE is the general-purpose champion: a purpose-built sub-1B draft head that reads the target's own hidden states (early, middle, late layers) and emits up to ~8 drafts at high acceptance, attached to the same module so no CPU round-trips — the go-to if you can train the head. N-gram speculation needs no model at all: build a dictionary of token sequences from the prompt during prefill, propose matching suffixes during decode — drafts can exceed 10 tokens and it beats EAGLE wherever output closely echoes input (code completion and revision, editing); lookahead decoding generalizes it by generating its own n-grams at the cost of extra compute.

Trade-offs. Speculation is a low-batch luxury: at high batch sizes compute saturates and engines dynamically disable it. It trades some throughput and cost for per-user speed — the exact opposite trade from batching, which is why the two negotiate.

Mechanism. Within a request, KV caching is table stakes — every engine does it, or decode would recompute the whole past per token. The technique is re-use across requests: prefix caching keeps the KV for shared openings, so a request matching a cached prefix skips prefill for those tokens — this is exactly why pay-per-token APIs price "cache hit" input tokens cheaper. The catch is autoregression: the match runs from token 1 to the first novel token and stops — two prompts identical except for their first word share nothing. Hence a context-engineering law: put novel tokens as late as possible (system prompt and documents first, the user's fresh question last). The savings compound in exactly the workloads that dominate production — long system prompts (agents, chatbots, RAG scaffolds), code completion's thousands of lines of shared context, document Q&A, and multi-turn chat that replays the whole transcript each turn. (Re-using mid-prompt chunks breaks positional assumptions; research systems like LMCache's CacheBlend selectively recompute to make it work — session 10 returns to this.)

Trade-offs — where the cache lives. KV is precious and VRAM is finite; engines let you set the fraction of post-weights memory reserved for it, and it will fill, forcing evictions and misses. The escape is tiered offload:

Keep hot blocks high, demote cold ones (Dynamo's KV Block Manager provides the plumbing); Grace-class superchips make G2 unusually fast. And once you run multiple replicas, routing must become cache-aware: send the user's next turn to the replica already holding their prefix, not just the least-busy one — or build a global G4 cache so any replica can eventually fetch any sequence and autoscaling doesn't vaporize warm state. Finally, "long context" has an operational definition: a sequence is long when its KV cache is big enough to cause trouble (commonly past 32K–128K tokens) — attention becomes the top VRAM consumer, and your defenses are the now-familiar trio: FlashAttention, PagedAttention, and chunked prefill (split a huge prompt into chunks scheduled alongside decode so one whale doesn't stall the boat — session 9 details it). Benchmark with genuinely long inputs or you'll discover this in production.

Mechanism. Sizing first: at FP8, ~1B parameters ≈ 1 GB, so DeepSeek-V3.1's 671B params OOM a single 192 GB B200 instantly; even 4×B200 (768 GB) merely fits weights with no room for KV — which often claims 80%+ of post-weight VRAM — so real serving takes a full 8-GPU node. Estimate: precision × parameters + expected KV, then round up to an instance size (and often go beyond the minimum for KV room and latency). Three strategies split the work, and the interconnect hierarchy from session 3 decides which fits where:

Trade-offs. TP is the single-node default (dense or MoE); mixed deployments run TP for attention + EP for the sparse MoE layers. Across nodes over InfiniBand: dense → TP8PP2 (TP inside, PP between); MoE → EP16-style layouts, whose replicated routers and lighter communication tolerate the slower fabric — TP8PP2 for per-user latency, wide EP for total throughput. (Context parallelism, splitting the sequence itself, is rare in LLMs but essential for video.) And the standing caution: unless weights + KV genuinely exceed a node, extra GPUs are usually better spent on horizontal replicas — or on the next concept.

Mechanism. Disaggregation fuses three ideas you now own: prefill and decode have opposite bottlenecks; specialization wins; and parallelism works if you respect interconnects. So: a prefill engine ingests the prompt, builds the KV cache and first token; ships the KV over the interconnect; a decode engine generates the rest — each engine tuned to its phase (prefill typically runs lower TP than decode). Production systems use conditional disaggregation: the request lands on decode first, which keeps it local if the prefix is cached or the input is short, and forwards genuine heavy prefill — far better for messy real traffic. Fleets are written xPyD (5P3D = five prefill, three decode workers), and Dynamo makes the ratio dynamic: a prefill queue, ISL-after-cache routing thresholds, and NIXL-based KV transfer (including a kernel that transposes KV blocks between different TP layouts).

Trade-offs — when it pays. Three conditions, per the material: serious volume (≈100M–1B+ tokens/day depending on model size), a large model (≥~100B params), and prefill-heavy traffic with long inputs. Missing the first two → you're burning hardware for marginal gains; missing the third → horizontal replicas beat it. The reference text case: a frontier LLM behind a code editor — huge, varied contexts from thousands of simultaneous developers. Which is precisely the shape of the GB200 deployments from session 1's field report.

Apply to your code

Your quant profile walks Fig S5.1's third exit — FP8 weights + FP8 KV on an L4 (Lovelace supports FP8) — so run the three quality gates against best before trusting it, and expect the KV half of the win to show up as higher max concurrency, not just faster tokens. prefix is concept 3 verbatim: measure your cache hit rate, and audit your prompt templates for the novel-tokens-late rule — in your fintech pipeline, documents and system scaffold belong before the per-request question. Your repo's IntelligentRouter is one upgrade away from cache-aware routing: add prefix-affinity to its scoring. TP/EP/disaggregation remain read-only knowledge on a single L4 — but n-gram speculation is a live experiment for code-shaped or revision-heavy traffic, since it needs no trained head and vLLM ships it behind a flag.

New terms this session

acceptance rate · draft/target model

Fraction of proposed draft tokens the target model verifies; the small proposer and the model being accelerated.

cache-aware routing

Sending requests to the replica already holding their KV prefix instead of merely the least-loaded one.

chunked prefill

Splitting long prompts into chunks scheduled alongside decode so one huge prefill doesn't stall everyone (detailed in session 9).

conditional disaggregation · xPyD

Decode-first routing that only forwards heavy prefill to prefill workers; the notation for fleet ratios (5P3D).

dynamic range · granularity · scale factor · E4M3

A format's value span; how many values share one scale factor; the multiplier mapping quantized values back; FP8's sign/4-exponent/3-mantissa layout.

EAGLE · Medusa · n-gram / lookahead

Hidden-state-fed draft head (the general-purpose choice); extra decoder heads (historical); dictionary-based drafting from the prompt (king of code editing) and its self-generating cousin.

G1–G4 tiers · KVBM

The KV storage ladder — VRAM, host RAM, local SSD, networked SSD — and Dynamo's block manager for moving KV between tiers.

microscaling (MXFP8/MXFP4) · NVFP4

Blockwise scale factors every 32 values; NVIDIA's 4-bit format with 16-value blocks plus a global scale.

perplexity

How surprised a model is by reference text; the quickest post-quantization quality check.

PTQ / QAT · calibration

Quantizing after vs during training; the pass that computes scale factors to preserve accuracy.

TP / EP / PP · all-reduce · topology-aware parallelism

Splitting tensors, experts, or layer-stages across GPUs; TP's per-layer synchronization; designing splits around the interconnect hierarchy.

Session 6 · Production — from a fast replica to a real service

A perfectly tuned engine on one GPU is a lab result. Production is everything wrapped around it: packaging, scaling, surviving hardware, watching it, and the client code everyone forgets to optimize.

Why this session exists: sessions 2–5 optimized the runtime layer; this is the infrastructure layer from session 1's stack, made concrete. The arc: package the replica (containers) → multiply it (autoscaling) → globalize it (multi-cloud) → operate it (testing, cost, observability) → and finally cross the wire to the client, whose overhead lives inside your latency budget whether you own that code or not.

Mechanism. A container is a running, isolated environment; an image is the executable package it runs from; a Dockerfile is the recipe; a registry (Docker Hub is to images what Hugging Face is to weights) stores and distributes them. Containers share the host's Linux kernel — lightweight by design — and images stack in layers: a base image (start from vLLM's or SGLang's official ones, not from scratch), your additional layers (dependencies, code, config), and a thin ephemeral container layer whose runtime writes vanish on termination.

Trade-offs. Inference dependency chains are long and brittle — a working build is an achievement worth freezing. An image pins together the CUDA toolkit/driver versions, Python packages (torch, transformers), the engine version, and system packages (ffmpeg for audio/video work). Two disciplines: pack light (images run to many GB; every extra layer slows deploys and cold starts) and pin exact versions (uv/poetry/pip will then reproduce the same build forever, immune to upstream breakage). Day-0 support for a hot new model usually means building on nightlies — expect to rebuild on stable within weeks. NVIDIA's NIMs (inference microservices) are pre-built containers — a flexible multi-LLM flavor and per-model max-performance flavors — useful as starting points or references; for maximum control, build your own from a leaner base.

Mechanism. The goal: never miss the latency SLA, never pay for idle GPUs. The machinery is Kubernetes: a cluster with a control plane (routing and scaling decisions) and a worker plane (the containers), running N replicas per model — where N is the thing autoscaling adjusts. Two families of scaling signal exist — utilization (GPU compute/memory) and traffic (in-flight requests) — and they can disagree: a burst of very long prompts can saturate GPUs while request counts sit flat, so scale on the signal that tracks your bottleneck. Scaling up means paying the cold start, a four-step ladder, each step optimized separately:

Trade-offs. Step 1 is mostly your provider (and contract). Steps 2–3 are bytes × bandwidth: pack light, and note quantized weights load faster too. Weights now dwarf images — load them separately, and from storage physically near the GPU (same datacenter), not over Hugging Face egress or an S3 hop, because hundred-GB models want GB/s. Step 4 bites when compilation is involved: TensorRT-LLM and compiled PyTorch take minutes to build engines — cache the built engine, and load it only onto the exact GPU/CUDA/dependency environment it was built in. Once replicas multiply, two different components share traffic: a router answers "where should this request go" (request-level) and a load balancer answers "where could it go" (system-level) — and naive even-splitting fails because requests aren't even: one 10,000-token prompt among 100-token traffic unbalances everything, and some requests have a better home (KV-cache-aware routing to the replica holding the prefix; LoRA-aware routing to the replica holding the adapter). Behind it all sits a queue — FIFO by default, priority if you want paid users ahead of free — which must drain to new replicas the moment they come online. Scale to zero closes the loop: zero replicas when idle, wake on traffic — legitimate for dev, periodic business-hours agents, and batch jobs, given fast cold starts and robust queueing; but if you're using it to make a latency-sensitive, lightly-used product affordable, session 1's tree is talking to you: you may not be ready for dedicated infrastructure at all. Finally, compound AI pipelines (voice-activity detector → transcriber → LLM) need independent component scaling — each stage right-sized and scaled on its own — but kept in one cluster: at 10 ms intra-cluster vs 50 ms inter-cluster per hop, a 5-step pipeline crossing clusters burns 200 ms — a fifth of a one-second SLA — on networking alone.

Mechanism. Past one cluster, you build the global version of Kubernetes' split: one control plane (deployment + global scaling decisions) over many workload planes (serve traffic, scale locally, report demand) — separated so any plane's failure leaves the others serving. GPU procurement spans hyperscalers (AWS/GCP — premium, broad), neoclouds (CoreWeave/Nebius — GPU-first), and resellers, bought three ways: reserved (months–years, discounted, the baseline), on-demand (flexible, pricey), and spot (cheap, pre-emptible on minutes' notice) — real fleets blend all three, spread worldwide for user proximity. Geo-aware load balancing then keeps requests near users: rule of thumb, ~5 ms per time zone crossed, so New York→San Francisco costs ~15 ms each way — real money against a 300 ms budget.

Trade-offs — reliability and compliance are why you bother. GPUs fail, constantly: the Llama 3 team logged 419 unexpected interruptions across 16,000 GPUs in 54 days — about one failure per 50,000 GPU-hours, and a single 8-GPU node run for a year is 70,000 GPU-hours. Treat health at the node level (one bad GPU predicts its neighbors): detect, cordon, cycle. Above hardware: provider maintenance and outages, answered by active-active (multiple regions serving simultaneously, seamless continuation) or active-passive (hot standby, cut over on failure). Security guards three assets — user data, model weights (a trade secret), and the infrastructure itself; the cheapest win is not storing user data you don't need, the rest is standard hardened-container practice validated by pen tests. Compliance rides multi-cloud too: SOC 2 / HIPAA generally require compliant providers underneath you, and data-residency rules (Canadian data stays in Canada) become solvable with a cluster near Toronto and one near New York.

Mechanism. Test end-to-end, not just per-replica: manual scripts, load tests, and shadow traffic (copy live requests to the candidate system) — remembering tests burn real GPU money, so sample shadows and keep load tests short. On cost: comparing per-token API prices to GPU-hours directly is a trap (batch sizing, traffic saturation, and sequence lengths all warp it) — instead convert both sides to total cost over at least a week, and add engineering time to the dedicated side for a true TCO. Observability watches, at minimum: request volume, input/output sizes, response codes, latency percentiles (TTFT, TPS, E2E at P50/P90/P99), replica counts (serving and starting), utilization across CPU/RAM/GPU/VRAM, and queue depth — together, because they explain each other: a latency spike reads completely differently against rising volume vs rising input lengths. Add server logs and audit logs, and pipe everything into the tools your team already wakes up to — Grafana, Datadog, PagerDuty — never a silo.

Mechanism. Every request has two sides, and the client side (browser, agent, app — usually the OpenAI SDK or a framework like LangChain/LiteLLM) spends your budget too: establishing a session costs tens of milliseconds, and a TLS handshake alone can eat 10%+ of a 300 ms P95 SLA — so re-use sessions (good SDKs do silently; your custom client must). When nobody's waiting, flip to asynchronous inference: fire-and-forget requests acknowledged immediately, results delivered to a webhook, with timeouts in hours instead of minutes — the right shape for bulk documents and corpus embedding. When the payload is continuous media, HTTP request/response stops fitting: WebSockets carry unstructured real-time streams (audio in/out; mind the per-server connection cap), gRPC carries schema-enforced service-to-service streams (validation makes it slightly slower). Text chat needs neither — streamed HTTP is enough.

Apply to your code

Modal is your control-plane-in-a-box: its scale-to-zero is concept 2's version (fine for your bursty dev traffic — and the warning about leaning on it for latency-sensitive products echoes session 1's tree). Your fp8-ckpt profile is a pure cold-start play: pre-quantized weights are half the gigabytes at step 3 and skip quantize-at-boot work at step 4 — measure boot time across profiles, not just steady-state TPS. Note the opposite trade hiding in best: CUDA-graph capture adds startup seconds to buy steady-state ITL — a cold-start vs warm-speed dial. And audit your Prometheus dashboard against concept 4's checklist: you have latency percentiles and tokens/sec; add queue depth, replica/boot state, and ISL/OSL distributions so a latency spike explains itself.

New terms this session

active-active / active-passive

Multi-region reliability postures: all regions serve simultaneously vs a hot standby that takes over on failure.

async inference · webhook

Fire-and-forget requests acknowledged instantly, results delivered by callback; timeouts in hours — the throughput-side client pattern.

cold-start ladder

Procurement → image load → weights load → engine start; the full price of new capacity.

container / image / Dockerfile / registry / layers

Running environment; its executable package; the build recipe; the distribution hub; and the base→additions→ephemeral stack images are made of.

control plane / worker (workload) plane

Decision-making vs traffic-serving halves of a cluster — and, globally, of a multi-cloud fleet.

independent component scaling

Right-sizing and autoscaling each stage of a multi-model pipeline separately — inside one cluster.

NIM

NVIDIA's pre-built inference containers: flexible multi-LLM or per-model max-performance flavors.

reserved / on-demand / spot

The three GPU purchase modes: long-term discounted baseline, flexible premium, pre-emptible discount.

router vs load balancer

"Where should this request go" (per-request, cache/LoRA-aware) vs "where could it go" (system-level evening-out).

scale to zero

Idling to zero replicas and waking on traffic; needs fast cold starts and robust queueing.

shadow traffic · TCO · audit logs

Copying live requests to a test system; total cost of ownership including engineering time; the record of changes to the service.

WebSockets / gRPC

Bi-directional streaming for unstructured real-time data vs schema-enforced service-to-service communication.

Session 7 · System design — build one, and the frameworks stop being magic

Source 2 opens by making you construct a serving service from scratch. Not to replace vLLM — to expose the components vLLM automates: request handling, batching, streaming, scheduling, resource management.

Why this session exists: frameworks abstract away architectural trade-offs, and you can't reason about performance, cost, or scalability through an abstraction you've never seen underneath. the serving literature's §3 builds a minimal single-model service, layers on batching and streaming, generalizes it into a reference architecture, then does it all again for multi-model serving. Every production system you'll ever touch is a refinement of these skeletons.

Mechanism. Six components: an API server (HTTP endpoints, request/response); an LLM engine — the conductor that initializes everything and orchestrates each request end-to-end; a workload manager (request queue + batch decisions — where batching strategy lives); a model executor (spawns and coordinates workers, communicating over a task queue and result queue); a model worker that loads and runs the model in its own process; and a model manager (weight loading and caching).

Trade-offs — why the process split. GPUs are expensive; the cardinal sin is letting one idle while a CPU tokenizes, validates, or serializes. So model execution is isolated into dedicated processes bound to specific GPU devices, while the web service stays on CPU handling concurrency and orchestration — all the CPU-shaped chores moved out of the GPU's way. This "seems like overkill" in a toy, and is standard practice in every real system.

Mechanism. Each prompt in each web request becomes a Sequence object with its own UUID — this is the load-bearing abstraction: it decouples the user's request from model execution, freeing the backend to regroup, reorder, and prioritize prompts however it likes. The workload manager keeps an incoming queue and an active set; get_next_batch fills the active set FIFO up to a cap (batch_size = 4 in the toy); the engine loops add → batch → execute → update until every sequence in the request finishes, then maps outputs back to callers by ID. Individual requests may wait a bit; the service as a whole produces far more — session 1's trade, now in code.

Trade-offs. FIFO-with-a-cap is deliberately naive: real batch configuration depends on the model, prompt characteristics, traffic shape, and hardware, and production engines replace this loop with continuous batching, dynamic scheduling, and memory-aware KV management (session 9's whole agenda). The skeleton's value is that when you meet those techniques, you'll know exactly which function they replaced.

Mechanism. The batch loop above returns nothing until the whole batch finishes — terrible UX. The fix exploits autoregression: since generation is token-by-token anyway, stream each sequence's new token to its client after every step while continuing to batch internally. New prompts join the active batch between steps: at T0 the batch is [P1]; at T1, P2 arrives and the batch is [P1,P2]; at T2, P3 joins while P1 finishes and exits. Delivery uses an async queue per client draining into an HTTP SSE (server-sent events, text/event-stream) response.

Trade-offs. Look at what you just built: requests entering and leaving a running batch at token granularity. That is continuous batching's essence, hand-rolled — proof that the industry-standard technique is less a trick than the natural consequence of taking streaming and batching seriously at once.

Mechanism. Generalizing, a production single-model service must deliver the classic six (low latency; high throughput in QPS/TPS; horizontal scalability; reliability; resource efficiency; observability) plus LLM-specific demands: giant memory footprints, KV cache management across requests and sessions, streaming, and batching of wildly variable-length workloads. The architecture answers with three separated concerns. Part A, infrastructure management: package serving as a replicable unit (container/Pod), delegate scaling, restarts, allocation, and monitoring to a platform (Kubernetes, cloud), and put a load balancer in front. Part B, the serving frontend: web interface + business logic — auth, integrations (model metadata, audit logs, billing), model download/config, request validation and batching, rate limiting. Part C, the serving backend: the inference engine itself (vLLM, Triton), an isolated process focused purely on execution performance.

Trade-offs. The stated reason for the separation is the deepest lesson in the section: LLM serving requirements evolve constantly and are model-specific (KV behavior shifts with attention variants; lengths shift with use cases) — so isolate the evolving parts (backend, model-aware logic) from the stable parts (infrastructure, business plumbing). Note how this rhymes with source 1's stack: frontend ≈ tooling, part A ≈ infrastructure, backend ≈ runtime — two sources, one shape.

Mechanism. When you have many models (sizes, versions, task-specific, per-customer fine-tunes), one service per model wastes GPUs on idle tenants. A multi-model service adds: a model store (metadata: framework, version, how to load), a model manager holding an LRU cache of loaded workers (evict least-recently-used when full), and a model engine that instantiates framework-appropriate workers on demand. Workflow: request names a model_id → cache hit? serve : fetch metadata → create worker → register (evicting if needed) → infer → respond. It shines when models are many and usage is sparse — 1,000 customer models with ≤200 ever hot at once, or scheduled jobs that load, run three hours, unload.

Trade-offs. Both real-world pains are UX: model cold starts (an unloaded model costs seconds to tens of seconds — download, load, maybe evict — and under load can cascade into timeouts) and hot-model scaling (replicating a suddenly-popular model is awkward when every instance has its own private cache). Two reference designs trade against them. Cost-optimized: shared multi-model instances plus a routing layer that maps models→instances, routes to already-loaded copies, tracks per-model replicas, and bin-packs models onto minimal servers — cheap, but reactive (always catching up to traffic) and operationally intricate. Latency-optimized: a dedicated, pre-provisioned instance group per model, created via a provisioning service before traffic arrives — no cold starts, independent scaling, simpler ops; the bill is over-provisioning models that never get hot. And multi-model thinking applies to LLMs too: prefix-cache-aware routing is "route to the instance holding the state," and multi-LoRA serving hot-swaps many adapters over one shared base model.

Apply to your code

This section is a mirror: your tutorial repo is its architecture. Your FastAPI server with health probes = the API server; your ModelServer = engine + executor; your LRU ModelManager juggling OPT-125m→2.7B on 24 GB = concept 5's model manager, cache and evictions included; your IntelligentRouter = part A's routing layer with a complexity dimension bolted on. Two upgrades fall straight out of the reading: add model-affinity to the router (route to the replica that already has the model loaded — the cost-optimized design's key move), and instrument model cold starts separately from request latency so the LRU's UX price is visible on your Grafana board. Your Modal offline-batch class, meanwhile, is concept 2's loop distilled to its essence.

New terms this session

API server / LLM engine / workload manager / model executor / model worker / model manager

The six-component skeleton: HTTP edge; orchestrating conductor; queue-and-batch brain; cross-process coordinator; the GPU-bound inference process; the weight loader/cache.

bin packing

Loading models onto the minimum number of servers to maximize utilization.

instance group / provisioning service

A dedicated, pre-created pool of replicas for one model, and the service that creates it before traffic arrives.

LRU cache · model cold start · hot model

Evict the least-recently-used loaded model when full; the seconds-scale price of summoning an unloaded model; the model suddenly needing replication.

model store / model metadata

The registry describing each model — framework, version, how to load — that drives worker creation.

multi-LoRA serving

Hot-swapping many LoRA adapters over one shared base model — multi-model economics at adapter granularity.

Sequence

The per-prompt tracked object (own UUID) that decouples web requests from execution and enables regrouping, prioritization, and continuous batching.

serving frontend / serving backend

Business-logic layer (auth, integration, validation, batching policy) vs the isolated, performance-only inference engine.

SSE · QPS · task/result queue

Server-sent events for HTTP token streaming; queries per second; the inter-process channels between executor and worker.

Session 8 · Challenges — why serving breaks, with the numbers to prove it

Source 2's bridge section: the same physics as sessions 2–3, re

Why this session exists: without the fundamentals, optimization degenerates into trial-and-error that gets stuck in local optima — you learn how a flag helps without why. And the stakes are business-shaped, in three ways. Customer experience: latency and satisfaction are inversely linked with sharply diminishing returns — 20 s → 1 s TTFT changes a product's fate; 0.1 s → 0.01 s changes nothing human-perceptible, so trade that surplus back for throughput. Less obviously, an optimization budget can be spent on quality: the headroom you win can fund a 70B model at the latency you formerly needed an 8B for. Cost: inference already consumes more AI chip spend than training and the gap is widening — training is an upfront investment; inference bills every query forever, and agentic workflows compound multiple model calls per user action. Feasibility: a sales agent stable all year can see Black Friday demand surge 400%, and optimized models that run on lower-grade, more available chips unlock regions where H-class GPUs simply can't be had.

The section's GPU tour adds a sharp lesson to session 3: "the same GPU" isn't. The H100 SXM and H100 NVL variants differ across the whole sheet (1,979 vs 1,671 dense FP16 teraFLOPS). And notice the two sources disagree on H100 FP8 by exactly 2× — 1,979 vs 3,958 teraFLOPS — because the larger figure is NVIDIA's with-sparsity number and the smaller is dense. You learned that trap in session 3; here it is in the wild, between your own two reference texts. Verify variant, verify dense, verify precision.

Mechanism. Weights travel disk → CPU RAM → GPU HBM, then stay cached in HBM — because the bandwidth ladder forbids anything else: SSD ~0.5–14 GB/s, CPU memory ~50–200 GB/s, GPU memory ~300 GB/s–3 TB/s. Serve from anywhere lower and every request eats a transfer delay no user will forgive. Sizing is two numbers: parameter count (usually in the model's name) × bytes per parameter (in config.json's torch_dtype): FP32 = 4 B, FP16/BF16 = 2 B, FP8/INT8 = 1 B. Llama-2-7B in BF16: 7B × 2 B ≈ 14 GB — and the model's shard files on Hugging Face sum to ~13 GB. The estimate works.

Mechanism. Fitting the weights isn't fitting the workload. The KV cache claims its share per token:

Trade-offs. That total is what your "memory left after weights" must hold, minus a reserve for activations (intermediate tensors). The material's A10-vs-L40S table makes it concrete: after the 14 GB model, an A10 (24 GB) has 10 GB left → max batch ≈ 4 at seq 4,096; an L40S (48 GB) has 34 GB → batch ≈ 16 — the pricier chip wins on cost per request. Memory also grows during generation as sequences lengthen, so provision for peak-at-end or meet the OOM mid-reply. Rule of thumb: start with GPU memory ≈ 2× model size. (The formula's "KV heads" term is why architecture matters: MQA, GQA, and DeepSeek's MLA — coming attractions — all shrink exactly that factor.)

Mechanism. W&H rebuild the roofline with a pizza oven: memory-bound = oven half empty, dough arriving too slowly; compute-bound = dough plentiful, oven at capacity. Their worked knee: L40S = 362 TFLOPS ÷ 864 GB/s ≈ 419 FLOPS/byte. Then the formula that pins everything down — for a matmul of [M,K]×[K,N]:

Now insert LLM shapes. The input tensor is [batch, sequence, hidden]; take batch 1 and a hidden dim h = 4,096. Prefill multiplies with M = s (the whole prompt): long prompts push intensity past the knee — compute-bound, as promised. Decode multiplies with M = 1: intensity = h²/(h + h² + h) ≈ 0.5 FLOPS/byte, regardless of sequence length. Not 50. Not 5. One half — three orders of magnitude under the knee.

Trade-offs — the two therapies. The diagnosis dictates the medicine: compute-bound → reduce FLOPS (better kernels, lower-precision math, fewer operations); memory-bandwidth-bound → reduce bytes moved (quantize, batch to re-use loaded weights, fuse kernels, speculate). Every technique in sessions 5, 9, and 10 is one of these two prescriptions wearing different clothes.

Apply to your code

Run concept 3 on your own model: open its config.json, read num_hidden_layers, num_key_value_heads (Gemma-class models use GQA, so it's the KV-head count — not attention heads — that enters the formula), and head_dim, and compute your MB-per-token; then your max_model_len ladder (32,768 → 16,384 → 10,000) turns into literal gigabytes of reclaimed batch room, and fp8 KV halves the per-token figure outright. Your gpu_memory_utilization 0.90 → 0.92 is the "memory left after weights" pool being widened by two points. And the napkin explains your repo's 22.5× batch-32 benchmark at the physics level: B climbing the intensity curve on a GPU whose knee it can never actually reach.

New terms this session

activations (serving sense)

Intermediate tensors created during a forward pass; a memory reserve alongside weights and KV.

hidden dimension · tensor shape

The width h of each token's vector; input shape [batch, sequence, hidden] — whose M-dimension collapses to 1 in decode.

KV-cache-per-token formula

2 × layers × KV heads × head_dim × bytes — the number that turns max batch × max seq into gigabytes.

knee / crossover point

The hardware ops:byte ratio where the roofline's diagonal meets the flat top (L40S ≈ 419; L4 ≈ 806 in FP8).

MHA / MQA / GQA / MLA

Attention-head layouts: full multi-head; one shared KV head; grouped KV heads; DeepSeek's latent compression — each shrinking the formula's KV-head factor.

peak memory

The end-of-generation high-water mark KV growth produces; the thing OOMs are measured against.

SXM vs NVL (variant discipline)

Same-name GPUs with different sheets — and the dense-vs-sparsity 2× that separates the two sources' H100 numbers.

torch_dtype

The config.json field naming a model's native precision, hence bytes per parameter.

Session 9 · Essential optimizations — the levers every deployment pulls

Session 5 surveyed the arsenal; this session opens the hood on the tier that's on by default everywhere: batching's full evolution, chunked prefill's scheduling truce, PagedAttention's memory trick, and prefix caching done properly.

Why this session exists: "essential" here means table stakes — continuous batching and PagedAttention are enabled in effectively every production LLM deployment, and prefix caching is now default-on too. Understanding their mechanics is the difference between tuning knobs and understanding a scheduler.

Intuition. You run a river ferry: 10 seats a boat. One person, one boat = delightful passengers, bankrupt business (that's decode at batch 1: the model reads billions of parameters to emit one token). Wait for a full boat = efficient, but the first arrival may wait forever. The whole history of LLM batching is refinements of this dilemma.

Mechanism. Batching's win is decode-side: a batch of 3 generates 3 tokens per read-through of the weights — arithmetic intensity artificially multiplied (session 8's napkin). Prefill barely needs the help; past ~1,000 input tokens it saturates compute alone. The evolution: static batching (client- or server-side) waits to fill a fixed batch — fine offline, fatal online, where 9 requests can sit five minutes waiting for a 10th. Dynamic batching adds a second dial: max delay time — dispatch when the batch fills or the clock runs out, whichever first (the ferry with a 5-minute rule). Good enough for classic ML; LLMs break it, because variable output lengths mean the whole batch waits for its slowest member. Continuous batching fixes that with iteration-level scheduling: the batch is recomposed every token step — finished sequences leave immediately, queued ones join immediately (exactly what your session-7 streaming loop discovered by accident). Two knobs govern it, and you must satisfy both: max-num-seqs (how many sequences ride at once — the decode-side cap) and max-num-batched-tokens (the per-iteration token budget — the prefill-side constraint; set it too low and prefill can't feed the GPU enough parallel tokens).

Mechanism. Continuous batching creates a collision: request 1 is mid-decode when requests 2–3 arrive wanting a long prefill. Prioritize prefill (it's TTFT, the metric chatbots live on) and request 1's decode stalls — its ITL spikes for the whole duration of someone else's prompt. Mix a whole prefill into the batch and the stall barely improves: one prefill step dwarfs a decode step. Chunked prefill is the truce: slice the long prompt into chunks sized like decode steps and interleave — decoders keep ticking every iteration while newcomers' prefill advances chunk by chunk.

Trade-offs. The ledger is exact: ITL improves and throughput usually rises (idle gaps get filled); TTFT lengthens (the newcomer's prefill is stretched out) and end-to-end can tick slightly worse from per-chunk overhead. The chunk size lives inside max-num-batched-tokens: set it to the max model length and you've disabled chunking; set it tiny and overhead eats you. It is, in other words, a direct SLA dial between your ITL and your TTFT.

The theory is sessions 2 and 4 (fusion, FlashAttention, GQA/MLA shrinking the KV term of session 8's formula); the practice is that attention backends are selectable: vLLM via environment (VLLM_ATTENTION_BACKEND=FLASHINFER), SGLang via --attention-backend {flashinfer|fa3|triton|…} — and engines choose sensibly by default (SGLang: FlashInfer on pre-Hopper GPUs, FlashAttention 3 on Hopper). The material' field advice: no clear-cut universal answer exists; start with defaults, exhaust the higher-leverage levers first, and treat kernel swapping as the last experiment, not the first.

Mechanism. Output lengths are unknowable in advance, so naive engines pre-allocate each request's KV as one contiguous max-length slab — and the vLLM paper measured the carnage: only 20.4–38.2% of KV memory actually held token state; the rest was fragmentation and reservation waste. PagedAttention imports OS paging: carve KV into fixed-size blocks (pages) of a few tokens each, allocate them anywhere, and resolve through a per-request block table — the prompt-plus-completion can live in blocks 7, 1, and 3, the last block half-filled and still growing. Result: "near-zero waste," and with it, default-everywhere status alongside continuous batching.

W&H's quantization treatment lands where session 5 did (with useful hands-on AWQ and KV-quantization walkthroughs to copy). The new material is pruning: models are over-parameterized, so remove redundancy — structured (whole sections), unstructured (individual weights), and the practical middle, 2:4 semi-structured sparsity: zero 2 of every 4 consecutive values, which sparse Tensor Cores (Ampere onward) accelerate at literally 2× matmul speed. Neural Magic's Sparse Llama 3.1 claims 98% accuracy recovery with ~30% higher throughput and ~20% lower latency on vLLM — promising, still maturing. And now session 3's spec-sheet warning closes its loop: the "sparse FLOPS" number is real silicon for models pruned this way, and marketing fluff for everyone else.

Mechanism. Classic response caching (Triton hashes the whole request → returns stored output) collapses on free-form text — humans phrase one question a hundred ways. Prefix caching matches prefixes instead: keep each request's KV in GPU memory after completion (rather than discarding), evict LRU when space runs out, and any new prompt sharing an opening skips that much prefill. SGLang's RadixAttention structures the materialkeeping as a radix tree (a trie) living in CPU memory, each node mapping to KV blocks on GPU, LRU applied recursively at the leaves — so "You are a helpful assistant…" is one shared node with a thousand branches. The two killer scenarios: multi-turn chat, where every turn replays the whole transcript (without caching, TTFT grows with every exchange), and long-context serving, where a stable mega-prompt turns 100k-token prefills into cache hits. Engines now enable it by default at near-zero overhead — even a 5% hit rate is pure profit.

Trade-offs — hit rate is an engineering discipline. Static content at the front, dynamic at the back (the session-5 law, now with teeth): the material shows that changing "Document" to "Documents" — one character — is a full cache miss. For RAG: byte-identical formatting, stable chunk ordering, deduplication; and partial hits still pay ("matched through Document 3:" is a win). Scaling it: reserve generous KV room per instance, and once you go horizontal, generic load balancing (round robin, least-connection, utilization-based) destroys prefix locality — you need affinity routing, e.g. consistent hashing of prefixes to instances. Both sources arrive at the same place: the serving text called it cache-aware routing; W&H build it from the LB primitives up.

Apply to your code

Your tuned profile's max_num_batched_tokens=8192 is Fig S9.1's dial — on your L4, run the honest experiment: fixed traffic, sweep the budget {2,048, 8,192, max_model_len}, and watch TTFT and ITL trade places at concurrency 16. Your repo's near-zero-KV-growth benchmark is concept 4 measured — you can now explain its mechanism, block table and all. And take the one-character lesson to your fintech pipeline today: byte-stable system scaffold and document formatting, dynamic user text last, then read your prefix hit-rate metric before and after — that single audit is often worth more than a hardware upgrade.

New terms this session

2:4 semi-structured sparsity · pruning

Zeroing 2 of every 4 consecutive weights so sparse Tensor Cores double matmul speed; the broader family of removing redundant weights (structured/unstructured).

block table / page

PagedAttention's per-request lookup mapping logical token positions to fixed-size KV blocks scattered anywhere in memory.

chunked prefill

Slicing long prompts into decode-sized chunks interleaved with decode steps: smoother ITL and better utilization for longer TTFT.

consistent hashing / affinity routing

Pinning prefixes to instances so horizontal scaling doesn't destroy cache locality; contrast round robin and least-connection balancing.

fragmentation

Wasted, unusable memory from contiguous pre-allocation — the disease PagedAttention cures.

iteration-level scheduling

Continuous batching's core: recompose the batch every token step; finished sequences exit, queued ones join.

max delay time

Dynamic batching's second dial: dispatch on full batch or expired clock, whichever first.

max-num-seqs / max-num-batched-tokens

The sequence-count cap (decode-side) and per-iteration token budget (prefill-side) that jointly govern the scheduler.

radix tree / RadixAttention

A trie of prompt prefixes (CPU-side) whose nodes map to GPU KV blocks, with recursive LRU eviction — SGLang's prefix-cache engine.

response caching

Hashing whole requests to reuse outputs — effective for fixed inputs, weak against free-form text; the foil that motivates prefix caching.

Session 10 · Advanced optimizations — past one GPU, past one phase

The deep end: speculative decoding with real benchmark numbers, the full parallelism family, prefill–decode disaggregation as an engineering problem, and the KV cache promoted to a first-class citizen of the serving stack.

Why this session exists: everything so far assumed the model fits on one GPU and both phases share it. Past ~100B parameters, or past the point where a shared GPU can hit your SLOs, neither holds. This section supplies what session 5's survey deliberately deferred — the mechanisms, the flags, and the measured results.

Mechanism (the missing depth). The analogy is two-stage retrieval: a cheap model filters a million candidates to a thousand, a strong model ranks the rest — speculation does this per token. Acceptance is probabilistic: if the draft proposed a token at probability 0.6 and the target scores it 0.8, accept; if the target scores it 0.4, accept with probability 0.4/0.6. On rejection, everything after the rejected token is discarded (autoregression makes it moot) and the target samples a replacement from a modified distribution that excludes the rejected token — this is the proof-backed guarantee that output is identical to non-speculative decoding. Lossless, always. The drafting ladder, in order of increasing sophistication: an existing same-family small model (same tokenizer; quantize it aggressively — the target is your safety net; distill it from the target if you can train, for acceptance-rate alignment); self-drafting, where the target drafts for itself — Medusa's extra heads propose parallel candidates per future position and assemble candidate sequences, while EAGLE predicts the target's future hidden states rather than tokens (steadier), with EAGLE-2 adding a dynamic draft tree that adapts speculation length to text predictability and EAGLE-3 fusing features from multiple layers — the current champion, at the cost of training; and the n-gram method — build a lookup table of token sequences from the prompt itself ("a quick brown → fox"), propose matching continuations, zero extra model.

Tuning K. Engines expose per-position acceptance: a K=6 run showing [0.8, 0.7, 0.6, 0.5, 0.10, 0.02] is telling you positions 5–6 are wasted — lower K. Typical sweet spot: 4–8; highly predictable generation (structured output, agent function-calling) tolerates 16–32; n-gram's near-zero draft cost means it profits even at low acceptance.

Limitations, consolidated. Decode-only; counterproductive when compute-bound (long-prefill workloads, big batches — it then trades throughput for latency); operationally fiddly to co-host two models on one GPU, which is why the industry has drifted toward self-drafting and n-grams; and a static K can't track shifting traffic — adaptive speculation is an active research front. Best case: latency-sensitive, low prefill:decode ratio, small real-world batches.

Mechanism. Four members. Data parallelism (DP) is horizontal scaling: replicate whole instances behind a router; strategies escalate from round robin → least-connections → latency-based → cache-aware, and modern LLM routers weigh a whole signal panel: KV locality, cache-hit percentage, per-instance KV usage, raw input length, queued and in-flight request counts, tokens already generated, and each request's SLO budget (see NVIDIA Dynamo's KV Router and the llm-d router). TP splits width (every layer sliced across GPUs; partial results merged — communication every layer); PP splits depth (contiguous layer-stages; communication only at stage boundaries — but unbalanced stages create pipeline bubbles where downstream GPUs idle). Expert parallelism (EP) serves MoE models (Mixtral, DeepSeek-V3, GPT-OSS) by partitioning experts across GPUs and dispatching each token only to the GPUs hosting its selected experts — complementing TP/PP rather than replacing them.

Trade-offs — the flowchart logic. First question: do you need multi-GPU at all? HBM feeds compute at ~3 TB/s; NVLink moves ~900 GB/s between GPUs — roughly 3× slower — so scaling is sub-linear (a second GPU doubles memory and FLOPS, not performance), and one bigger GPU beats several small ones when it suffices. Second: quantize before you shard — FP8 halves the footprint, W4A16 quarters it. Then: one NVLink node → TP (TP=8 on an 8×H100 box is the classic setup); a PCIe-only multi-GPU box → TP barely works (PCIe is device-to-host plumbing, ~10 GB/s class) — keep TP tiny, consider PP or a different instance; genuinely hyperscale → TP inside nodes, PP across them (the TP8PP2 pattern), managed via Ray for multi-node vLLM. Both sources have now handed you the same layout from different directions.

Mechanism. Chunked prefill interleaves the phases but can't erase their interference — they still share silicon with opposite utilization profiles. Disaggregation physically separates them, which buys two things: independent TTFT/ITL optimization (scale prefill workers for input-heavy traffic, decode workers for output-heavy — per your input:output ratio) and stable, predictable ITL, since no prefill ever barges into a decode GPU. The make-or-break cost is KV cache transfer — prefill's output must reach the decode worker faster than it would take to just recompute it.

Trade-offs. Same-node placement is confining (same hardware for both phases; caps prefill's TP width), so serious deployments go inter-node with RDMA plus four overhead-hiding tricks: stream the KV in chunks (like video) rather than all-at-end; make transfers asynchronous so communication overlaps compute; exploit that KV is layer-local and ship it layer-by-layer as prefill finishes each layer; and shrink it (quantize/compress). Done well, transfer overhead drops under 1% of per-request latency (Wang et al., 2025). When to use it: the material's flowchart lands where the serving text's did — large models, heavy load, and genuine independent-TTFT/ITL tuning needs; everyone else should keep the simpler aggregated setup.

Mechanism. The frontier workloads — coding copilots tracking whole repos, agents with long histories, tenant knowledge bases — pose a choice: RAG (retrieve relevant chunks per query; short prompts, but chunk variety kills cache hits) versus CAG, cache-augmented generation (load the whole corpus as a cached long context; prefix caching is its naive form). Two shifts make CAG real: 100k–1M-token windows, and models increasingly beating the lost-in-the-middle positional bias. The material's worked comparison, with a 500-token system prompt cached in both: RAG prefills 10×500 fresh chunks + 500-token question = 5,500 non-cached tokens; CAG holds 100k tokens cached and prefills only the 500-token question — so a 5 s RAG TTFT becomes ~0.5 s (before even counting embedding + vector-search time). But on vendor APIs, cached input costs 10–25% of regular input (GPT-5: $1.25 vs $0.125/M; Gemini 2.5 Pro and Claude Sonnet 4 similar with storage/write surcharges) — and 100,500 cached tokens at GPT-5 rates ≈ $0.013/request versus RAG's ≈ $0.007. CAG is twice as fast and, via API, roughly twice the price.

Trade-offs — self-hosting flips the economics. On your own GPUs, cache storage is nearly free if you have somewhere to put it — enter LMCache, which makes KV a first-class citizen: tiered offloading (CPU RAM ≈ 3× the KV space, local SSD ≈ 50×, Redis/S3 beyond — figures from an AWS p5.48xlarge) lets one instance hold, say, four tenants' long contexts and swap them GPU↔CPU on demand — a 4× instance saving where any single tenant can't saturate a replica; CacheGen compresses KV into distribution-aware bitstreams that transfer fast and decompress cheaply; and CacheBlend attacks prefix caching's mid-prompt blindness by pre-computing per-chunk KV and blending them with selective recomputation of a small token fraction — making RAG chunks themselves cacheable regardless of order. None of this obsoletes RAG: enormous, dynamic, unbounded corpora still want retrieval; the point is that the KV cache is now a designed, tiered, routed subsystem — exactly the G1–G4 picture the serving text drew, here with the open-source parts named.

Apply to your code

On a single L4, TP, EP, and disaggregation stay read-only knowledge — but concept 1 is runnable this afternoon: your fintech traffic (documents in, extractions out) is reference text n-gram territory, since outputs echo inputs. Add an eighth profile to your ladder with the improved-n-gram config, then benchmark at concurrency 1 and 16 and watch for the material's crossover — and pull the per-position acceptance stats before touching K. Separately, sketch the CAG math for your static compliance documents: at self-host prices with LMCache-style CPU offload, your 3× KV space is free RAM you already rent from Modal.

New terms this session

CAG · lost in the middle

Cache-augmented generation — the whole corpus as reusable cached context; the positional bias (strong start/end, weak middle) long-context models are outgrowing.

CacheGen / CacheBlend / LMCache

Distribution-aware KV bitstream compression; per-chunk KV blending with selective recompute (cacheable RAG chunks); the system elevating KV to managed, tiered infrastructure.

data parallelism (DP) · router signals

Replicating whole instances behind a router; the modern signal panel — KV locality, hit %, cache usage, input length, queue depths, tokens generated, SLO budget.

modified distribution

The adjusted distribution the target resamples from after rejecting a draft token — the mechanism behind speculation's losslessness.

per-position acceptance · prompt-lookup window

The engine telemetry ([0.8, 0.7, …]) that tunes K; the n-gram match-length bounds (min/max) in vLLM's config.

pipeline bubble

Idle downstream stages when PP's assembly line has a slow station — the price of PP's low communication.

RDMA · layer-wise KV transfer

Direct remote memory access (InfiniBand-class) that disaggregation needs across nodes; shipping each layer's KV as prefill completes it to hide transfer behind compute.

self-drafting · draft tree

The target model drafting for itself (Medusa heads; EAGLE hidden-state prediction); EAGLE-2's dynamic structure adapting speculation length to predictability.

W4A16 · Ray

4-bit weights with 16-bit activations (quarter-size models); the distributed framework coordinating multi-node vLLM.

Session 11 · Capstone — the whole stage in one afternoon

W&H's closing lab runs everything you've learned against one model (Qwen3-14B), one GPU (an L40S), and a benchmark harness — and the results vindicate four sessions of theory with uncomfortable precision. This session is the lab, plus the synthesis of the stage.

Why this session exists: optimization is a moving target — the "best" strategy changes with environment, and resources don't allow brute-forcing every option. What survives is a method. The lab optimizes online token throughput for a single instance (throughput = cost, since tokens are the billing unit), while honestly flagging the standing tension: peak throughput and minimum latency conflict, so the real objective is maximum throughput within an acceptable latency band — session 1's SLO thinking, now operationalized.

Mechanism. (1) Examine the hardware: nvidia-smi first, always — driver/CUDA compatibility, performance state (P8 idle, P0/P1 under load), and two diagnostic pairings worth memorizing: low utilization under load = batching/scheduling inefficiency; high power with low throughput = memory or kernel bottleneck. (2) Generate representative traffic — dataset selection is the optimization target: the whole exercise tunes the system for a traffic shape, so the shape must be real. The lab uses ShareGPT (genuine conversational lengths) plus a synthetic prefix-repetition dataset with controllable prefix length, suffix length, and unique-prefix count — a tunable cache-stress instrument. Traffic is driven by vllm bench serve, which controls request rate, burstiness, and max concurrency while collecting metrics. (3) Define metrics before measuring: from the full menu, the lab picks four — total token throughput, output token throughput, mean TTFT, mean ITL. (4) Start the server and read the logs (next concept — the logs are half the lesson). (5) Baseline benchmark. (6) Quantized benchmark. (7) Workload-specific techniques — LMCache-style KV reuse for prefill-heavy traffic, speculative decoding for decode-heavy, plus the knob cluster you now know by heart (gpu-memory-utilization, max-model-len, block-size, max-num-seqs, max-num-batched-tokens). (8) Distributed benchmark — deliberately last.

Mechanism. vLLM's boot log is a balance sheet. Qwen3-14B in BF16 on the 46 GB L40S: weights 27.5 GiB, KV budget 11 GiB = 72,064 cacheable tokens, and the log's own verdict — "maximum concurrency for 40,960-token requests: 1.76×." The model devours 65% of the GPU; the workload is KV-starved before the first request arrives (session 8's 2×-rule violated in the wild). Swap in the AWQ 4-bit checkpoint: weights 9.36 GiB, KV 29.15 GiB = 191,056 tokens, concurrency 4.66× — the ledger transformed. The benchmarks then merely confirm what the ledger foretold: ShareGPT throughput jumps 2.7× (474 → 1,280 total TPS) with TTFT down 42% (103.6 → 59.3 ms). Two more results complete the picture: the same baseline server fed prefix-repetitive traffic produced 1,123 TPS vs ShareGPT's 474 at identical latency — no flags changed, just session 9's defaults (prefix caching, continuous batching, block sharing) meeting traffic shaped to reward them; and nvidia-smi during runs read 97% utilization — the saturation check.

Trade-offs. AWQ here is weights-only 4-bit: the win is data movement and freed memory (bandwidth therapy + capacity therapy from session 8), not cheaper math — activation quantization is the compute-side lever. And a quality gate still applies before you ship a 4-bit model to fintech customers: session 5's three checks are not optional.

Mechanism. Step 8 runs Qwen3-14B-AWQ on two multi-GPU boxes: a g6e.12xlarge (4× L40S, PCIe only) and a p4d.24xlarge (8× A100, NVLink). On the g6e, the single-GPU setup beats TP2 and TP4 on both throughput and TTFT — despite the L40S being the stronger single-GPU inference chip — because every all-reduce crawls over PCIe. On the p4d, the ordering flips: TP4 wins, halving TTFT from 66 to 33 ms. Exactly the session-3/session-10 prediction: parallelism inherits the interconnect.

Trade-offs — the kicker. Even on the NVLink box, TP is not "free performance": four independent replicas (one per GPU) deliver nearly triple the total throughput of TP4 sharing one model (9,816 vs 3,926 TPS). So the true benefits of vertical scaling are exactly two — fit models that don't fit, and cut per-request latency (which no amount of horizontal replication can do) — while horizontal scaling wins throughput, fault tolerance, and simplicity. Production default: horizontal, single-GPU replicas; go vertical on demonstrated need. Both sources, one verdict.

The material close with the five tensions that survive every technology cycle, each of which you now own at mechanism depth: throughput vs latency (batching's waiting time — sessions 1, 9); memory efficiency vs model quality (quantization's dial and gates — sessions 5, 8); hardware utilization vs flexibility (aggressively tuned configs overfit one GPU and one traffic shape — session 11's own AWQ-on-PCIe results); vertical vs horizontal scaling (fit + latency vs throughput + resilience — sessions 3, 10); and static vs adaptive serving (fixed configs are predictable but brittle; frameworks increasingly self-tune batch size, cache policy, and scheduling from live metrics — the field's next section). The method that navigates them: define the scenario → craft representative traffic and honest metrics → establish a baseline with the general wins → add workload-specific techniques → scale deliberately → iterate, forever.

Apply to your code — the graduation exercise

Run the eight steps on your own stack, verbatim: nvidia-smi on your Modal L4 and note the diagnostic pairs; build two benchmark datasets — one shaped like your fintech traffic, one prefix-repetition with unique-prefix count set to your number of document templates; adopt the four metrics; then, before any traffic, capture vLLM's memory ledger for every PROFILE in your ladder and tabulate weights / KV gigabytes / cacheable tokens / the concurrency line — that table alone will rank your profiles almost perfectly. Benchmark baseline vs your FP8 profiles and expect the AWQ story in miniature; bolt on the session-10 n-gram experiment as your workload-specific step; and skip step 8 with a clear conscience — your L4 is the single-GPU exit of Fig S11.1, and the lab just proved that exit is where most production systems belong.

And with that, the stage's three goals are closed: you can whiteboard a full serving system and defend each component (sessions 1, 6, 7), argue every major trade-off from physics rather than folklore (2–5, 8–10), and you hold a repeatable method plus a profile-by-profile experiment plan for your own code (this session). The two sources end in the same place deliberately: measure, don't guess — and the measurement only means something if the traffic is yours.

New terms this session

adaptive serving

Frameworks self-tuning batch size, cache policy, and scheduling from live traffic and hardware metrics — the answer to overfit static configs.

AWQ

Activation-aware weight quantization; the lab's 4-bit weights-only checkpoint that tripled KV space and 2.7×'d throughput.

burstiness

The benchmark-traffic knob controlling arrival clumpiness at a fixed average rate — real users don't arrive uniformly.

memory ledger (boot log)

The engine's startup accounting — weights GiB, KV GiB, cacheable tokens, worst-case concurrency — that predicts benchmark outcomes before any request.

nvidia-smi · P-state

The first command of every session: driver/CUDA versions, utilization, memory; P8 = idle, P0/P1 = full-speed under load.

prefix-repetition dataset

Synthetic cache-stress traffic with controllable prefix length, suffix length, and unique-prefix count.

ShareGPT

The real-conversation dataset standing in for chatbot-shaped production traffic.

vertical vs horizontal scaling

More GPUs per model (fit + per-request latency) vs more single-GPU replicas (throughput + fault tolerance) — production defaults to horizontal.