Think of your service as an airport. Uvicorn — the server program that accepts network connections and speaks HTTP — is air-traffic control: it handles planes (connections) arriving on the runway and speaks the radio protocol. FastAPI is the terminal building: gates (routes — URL paths mapped to handler functions) where each flight is processed. Dependency injection — a system where the framework builds and hands your handler the things it declares it needs — is the ground crew: every gate that says "I need fuel and a baggage cart" gets them delivered, without the gate knowing where fuel comes from. And lifespan — code that runs exactly once at startup and once at shutdown — is opening the airport in the morning and locking it at night.

The point of the analogy: the terminal never flies planes. Your FastAPI app never runs inference. It receives, checks, routes, and hands off.

FastAPI is not a server — it's a framework that plugs into a standard called ASGI (Asynchronous Server Gateway Interface: the Python contract between web servers and web apps that supports async code, streaming, and WebSockets). ASGI replaced the older WSGI (the synchronous predecessor — one request in, one response out, no streaming mid-response), and that difference is exactly why FastAPI can hold a token stream open while serving other users. FastAPI itself is a thin, typed layer over Starlette (the ASGI toolkit providing routing, middleware, responses), which is run by Uvicorn.

Now the lifecycle of one request through that stack — this exact sequence is the skeleton that S2 (concurrency), S3 (streaming), and S4 (the gauntlet) all hang off:

Two mechanisms deserve their own snippets, because the code is the concept. First, lifespan — the once-per-process bracket where expensive shared things (an HTTP connection pool, config) are created:

# the app-wide setup/teardown bracket
@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.http = httpx.AsyncClient(base_url=VLLM_URL)  # create once…
    yield                                                  # …serve requests…
    await app.state.http.aclose                          # …clean up once

app = FastAPI(lifespan=lifespan)

Second, a dependency — a plain function whose return value FastAPI injects wherever it's declared:

def get_http(request: Request) -> httpx.AsyncClient:
    return request.app.state.http            # hand out the shared client

HttpDep = Annotated[httpx.AsyncClient, Depends(get_http)]

@app.post("/v1/chat")
async def chat(req: ChatRequest, http: HttpDep):  # injected, testable, no globals
    ...

Why this matters beyond tidiness: in S5 you'll override get_http in tests to point at a fake model server — dependency injection is what makes the whole service testable without a GPU.

FastAPI's overhead is noise next to inference. Framework routing + validation costs on the order of a millisecond per request; your Gemma endpoint takes hundreds to thousands of milliseconds to generate. The app layer is never your latency problem — until it blocks (S2) or buffers (S3).

Know why the boundary exists — the material's own list. §2 closes with "FastAPI Limitations" and it reads as a spec of what vLLM does instead: inefficient model memory management, thread limits, the GIL, no micro-batching of inference requests, no CPU/GPU workload splitting. Multiple Uvicorn workers can't share one in-memory model (each process would load its own copy makes this explicit). Every item on that list is a reason inference lives across the boundary, and none of it stops FastAPI being excellent at the app-layer jobs.

Structure decays without intent. Flat single-file apps are fine to learn with; the material's "modular structure" (routers/, services/, schemas/, dependencies.py) is what survives a second feature. The lab uses it from the start.

Version reality (July 2026): FastAPI is on the 0.13x line (0.138 shipped June 2026) with monthly minors, requires Python 3.10+, and recent releases refactored router internals (routers are now preserved, routes can be added after inclusion) and added optional frontend-serving. Treat the material's install instructions as historical; pin versions with a lockfile (the lab uses uv).

A Pydantic model is a Python class that declares the exact shape of data — field names, types, and rules — and rejects anything that doesn't fit. Think of it as customs at the border. Every traveler (request) is inspected against a declared manifest before entering the country (your logic). Contraband — a 2-million-token prompt, a negative max_tokens, a role of "admin" in a chat message — is turned away at the border with a precise citation of what rule it broke, instead of being discovered later as a mysterious crash deep inland (or worse, forwarded to your GPU to burn money).

Same machinery, both directions: validation is checking incoming data against the model; serialization is converting your Python objects into outgoing JSON. One class defines both sides of the contract, and FastAPI turns it into live documentation.

Declaring a model is declaring the contract — this snippet is the mechanism itself. It's the request shape your lab uses, deliberately a product-shaped subset of the OpenAI chat schema your Modal endpoint speaks:

class Msg(BaseModel):
    role: Literal["system", "user", "assistant"]   # enum, not free text
    content: str = Field(min_length=1, max_length=32_000)

class ChatRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")      # unknown fields → 422
    messages: list[Msg] = Field(min_length=1, max_length=64)
    max_tokens: int = Field(default=512, ge=1, le=2048)   # cap what GPU pays for
    temperature: float = Field(default=0.2, ge=0, le=2)
    thinking: bool = False                         #  Gemma flash ↔ thinking switch

What happens with it at runtime:

When declarations aren't enough, validators — functions attached to a field or the whole model that run custom checks or transforms during validation — take over (source §4): a @field_validator can strip control characters from content; a @model_validator can enforce cross-field rules like "the first message must be the system role" (which, you already know, is also what keeps your vLLM prefix cache hitting — contract design and inference performance meet here). Computed fields add derived read-only outputs, and pydantic-settings applies the same validation to environment variables, so a missing VLLM_BASE_URL kills the process at startup with a clear error instead of a 3 a.m. 500.

Validation is effectively free here. Pydantic v2's core is compiled Rust (pydantic-core), benchmarked 5–17× faster than v1 on typical models; validating a chat request costs microseconds against a generation that costs seconds. Never skip validation "for performance" on an LLM API.

Coercion vs strict is a real choice. By default Pydantic coerces ("42" → 42) — friendly for humans and sloppy clients. strict=True rejects instead — better for machine-to-machine contracts. Pick per model, deliberately.

extra="forbid" cuts both ways. On your product API it catches client typos (temprature) instantly. But if you ever promise "OpenAI-compatible" to consumers, forbidding unknown fields will break SDKs that send new optional params — compatibility layers must be tolerant readers (accept and ignore unknowns), which is exactly how vLLM and NIM behave. Product API: forbid. Compatibility API: ignore.

Where it breaks: validation cost does become visible on huge payloads (multi-MB document uploads) — bound sizes first (max_length, body-size limits at the proxy). And a schema can't judge meaning: "content is a string ≤ 32k chars" passes prompt injections happily. Semantic safety is a different layer — S4's guardrails.

Two words first. Concurrency is making progress on many tasks in overlapping time — juggling. Parallelism is literally executing several tasks at the same instant on separate cores — a subset of concurrency. The material's picture: one fast-food owner alternating between the register and the grill is concurrent; hiring three cooks who grill simultaneously is parallel.

Now the inversion that makes this session simple. In your inference stage, a request was compute-bound (limited by processor speed) and memory-bound (limited by how fast data moves to compute — the KV-cache story you know). But your gateway never does that work — it sends bytes across the boundary and waits. From the app layer's chair, a 4-second generation is indistinguishable from a slow database: it is I/O-bound (limited by waiting on input/output — network, disk, another service). The material says it directly: relying on an external model API is "the only instance where you can treat AI inference workloads as I/O-bound." Waiting is cheap. One small CPU process can hold hundreds of waits — if you wait correctly.

Python offers three ways to overlap work, and the GIL (Global Interpreter Lock — CPython's rule that only one thread executes Python bytecode at a time within a process) decides what each is good for: async I/O (one thread, tasks yield at await — best for many waits), multithreading (several threads, GIL serializes Python but releases during I/O waits — fine for waits, useless for Python-level compute), and multiprocessing (separate processes, true parallelism, separate memory). Your gateway is wait-shaped, so async I/O is its native tool; multiprocessing appears only as replicas — more container instances behind a load balancer.

Why not just run many workers and stay sync? You can — it's simple and correct — but each worker process holds one request's wait at a time, so 200 concurrent 4-second generations need ~200 processes' worth of memory for work one async process could hold. Worse, the material point: separate processes share no memory, so anything loaded per-process is loaded N times. For a thin gateway that's tolerable; it's fatal for in-process models — one more brick in the boundary.

GIL, 2026 status: free-threaded CPython (PEP 703) has arrived as an opt-in build — experimental in 3.13, officially supported in 3.14 — but it targets CPU-parallel threading, which your I/O-bound gateway doesn't need. Async remains the right tool here regardless of the GIL's fate.

Async's real cost is discipline, not performance: every library in the hot path must be async-capable (httpx not requests, asyncpg not psycopg2's sync mode, redis.asyncio), and one lapse triggers the failure mode Concept 2 is about.

The event loop is a single-threaded scheduler that runs pieces of tasks, parking any task that's waiting and resuming it when its I/O completes. Picture one waiter with a perfect notebook: they take table 3's order, hand it to the kitchen, and — crucially — never stand at the kitchen window. They serve tables 4 through 60, and when the bell rings for table 3, the notebook says exactly where to resume. A coroutine is a function defined with async def whose execution can pause at each await and resume later — a notebook entry. await is the moment the waiter walks away from the window.

The thread pool is a fixed crew of worker threads the framework keeps around for code that insists on standing and waiting — a few old-school runners who do stand at the window so the head waiter doesn't have to. FastAPI's rule: async def handlers run on the loop; plain def handlers are shipped to the pool. Both keep the restaurant serving. The catastrophe is a third thing: the head waiter deciding to stand at the window personally. Every table stops being served at once.

The material three-endpoint experiment, condensed — this code is the whole lesson:

@app.get("/fast")                     # ✓ coroutine + async client
async def fast:
    return await async_client.chat.completions.create(...)   # loop parks it; serves others

@app.get("/slow")                     # ✓ sync handler + sync client
def slow:
    return sync_client.chat.completions.create(...)          # FastAPI ships it to the thread pool

@app.get("/block")                    # ✗ THE bug: async def, blocking call inside
async def block:
    return sync_client.chat.completions.create(...)          # runs ON the loop → whole server stalls

Two escape hatches when a sync-only library must run inside async code: await asyncio.to_thread(fn, …) / run_in_threadpool (push just that call onto the pool and await its result), or wrap fan-out I/O with asyncio.gather(*coros) (run many coroutines concurrently and collect all results) — which the material uses to fetch many URLs at once in its web-scraper project, and which your lab reuses below.

The pool is small on purpose: the default AnyIO thread pool under FastAPI is 40 threads. Forty concurrent sync requests and the 41st queues — which is graceful degradation, but with 4-second generations that's 10 req/s ceiling per instance for sync handlers. Async handlers have no such ceiling on waits; their limit is upstream capacity (and S4's deliberate limits).

Detection beats belief. A blocked loop looks like "the service randomly hangs under load." Tools: PYTHONASYNCIODEBUG=1 logs slow callbacks; py-spy dump --pid … shows exactly which line the loop thread is stuck on; and your p99 will scream long before your p50 does (S5 makes this measurable).

Blocking ≠ only network. CPU work on the loop blocks it identically: parsing a 100-page PDF, tokenizing megabytes, json.loads on a huge body. Push heavy CPU to to_thread (GIL releases for much C-backed work) or to a worker process — Concept 3.

Some work shouldn't hold a connection open: enriching 5,000 tickers, generating a batch of images, re-indexing documents. The universal shape is the dry cleaner: you hand over the shirts, get a ticket immediately, and come back later. In HTTP that's 202 Accepted (the status code meaning "received and queued, not done") plus a job id, then either polling (the client periodically asks "is it done?") or a push notification when ready. FastAPI gives you a zero-infrastructure starter — BackgroundTasks, work scheduled to run in the same process after the response is sent — and the honest question of this concept is when that stops being enough and a task queue (a broker like Redis holding jobs that separate worker processes pull and execute) takes over.

Your stack already has the queue's big brother: gemma_modal.py's OfflineEnricher class — a Modal function that loads the engine in-process and chews a whole JSONL batch. For heavy offline work, the gateway's correct move is often "submit to Modal, track the call id" rather than running anything itself: Modal is your durable worker tier. The material's warning stands either way : background work still shares the machine — schedule the coordination in the gateway, run the weight elsewhere.

Plain HTTP is a vending machine: one coin, one complete item. Generation breaks that shape — the item is produced over seconds. The menu of fixes, as couriers:

• Short polling — the client asks "done yet?" on a timer. Phoning the post office every five minutes. Simple, chatty, mostly empty answers.
• Long polling — the server holds each request open until it has something, then the client immediately asks again. Staying on hold until there's news; hang up, redial. One message per call.
• SSE (Server-Sent Events) — one HTTP response the server keeps open, pushing many messages down it; strictly server→client. A ticker-tape subscription: it prints as events happen, you never talk back on that wire. This is how LLM tokens flow across the industry — the material notes ChatGPT itself rides SSE.
• WebSocket — an upgraded connection where both sides send messages anytime. A phone call. Full-duplex, its own protocol after an HTTP handshake.
• gRPC streaming — typed remote procedure calls over HTTP/2, with client-, server-, or bidirectional streams defined in a schema (reference source §8). A dedicated business line with pre-printed forms — superb between your own services, not a browser native.

SSE's two famous catches. (1) The browser's EventSource only issues GET — no body — so real chat (long histories) uses POST + manual stream reading via fetch, trading away free auto-reconnect (the material walks both; the lab uses POST). (2) Over HTTP/1.1, browsers cap ~6 connections per origin — a few open SSE tabs can starve a site; HTTP/2 multiplexing (reference) removes the cap and is table stakes in 2026.

WebSocket's cost is operational, not conceptual: every proxy, LB, and gateway on the path must pass the upgrade; reconnect, heartbeat, and resume logic are yours to write (SSE gives them free via retry:/Last-Event-ID); and long-lived stateful connections make rolling deploys and autoscaling stickier. Pay it when you need the second direction mid-stream — voice, "stop generating" without a second request, human-approves-tool-call agent UIs.

The material's design rule survives every transport : one streaming entry point, with body/params selecting behavior server-side — not an endpoint per feature. State stays on the server; clients stay dumb.

Your gateway is a relay commentator: it listens to the raw feed from the stadium (vLLM's SSE) and re-broadcasts to your audience — same events, your voice. It can drop internal chatter, rename things, add its own markers ("that was the reasoning; here's the answer"), and if a listener hangs up, it tells the stadium to stop the feed. Two streams, one loop: read a chunk upstream → transform → write a chunk downstream, repeated per token.

First, the bytes. An SSE stream is UTF-8 text: each event is a few field: value lines ended by one blank line — that blank line is the delimiter that makes it a protocol:

Now the relay itself — an async generator (an async def function that yields values over time; each yield becomes one chunk written to the client) bridging the two streams. This skeleton is the mechanism (full version in the lab):

async def relay(payload: dict, http: httpx.AsyncClient):
    async with http.stream("POST", "/v1/chat/completions", json=payload) as r:
        async for line in r.aiter_lines:            #  stream IN from vLLM…
            if not line.startswith("data: "): continue
            if line == "data: [DONE]":
                yield "event: done\ndata: {}\n\n"; return
            delta = json.loads(line[6:])["choices"][0]["delta"]
            if txt := delta.get("reasoning_content"):
                yield f"event: reasoning\ndata: {json.dumps({'t': txt})}\n\n"
            if txt := delta.get("content"):           #  …transform…
                yield f"event: token\ndata: {json.dumps({'t': txt})}\n\n"

@app.post("/v1/chat/stream")
async def chat_stream(req: ChatRequest, http: HttpDep):
    return StreamingResponse(relay(build_payload(req), http),   #  …stream OUT
        media_type="text/event-stream",
        headers={"Cache-Control": "no-store", "X-Accel-Buffering": "no"})

Note what the amber layer just earned its keep doing: it split reasoning_content from content into named events (your product's stream shape), hid vLLM's raw chunk schema from clients, and set the two headers that keep intermediary proxies honest — Concept 3's subject. Backpressure — the slower side pacing the faster side so buffers don't grow without bound — comes free here: each yield awaits the socket write to the client, and while paused, you're not pulling from vLLM either.

Passthrough vs re-emit. Forwarding vLLM's bytes untouched is simplest and cheapest — right when you promised OpenAI-compatibility. Re-emitting typed events (above) buys a stable product contract and reasoning separation, at the cost of one JSON parse+dump per chunk — microseconds against a ~10–100 tokens/sec stream; negligible until you proxy thousands of concurrent streams, at which point you're choosing a Go/Envoy data plane anyway (S5).

Get usage before [DONE]: send "stream_options": {"include_usage": true} upstream and vLLM's final chunk carries token counts — feed them to S4's budget tracker; without it, streaming responses are invisible to cost accounting.

Ordering & loss: SSE inherits TCP's in-order delivery per connection; what it can't survive is the connection. If mid-stream death must be resumable, you need id: per event plus server-side replay state — real work. Most chat products instead make regeneration cheap and idempotent. Decide which product you are.

Your stream is water through a hose with many couplings — CDN, load balancer, ingress, server, the model host's own edge. Any coupling can quietly do one of three hostile things: pool the water (a buffering proxy — an intermediary that collects the whole response before forwarding it — turning your live stream into one big splash at the end), auto-shutoff on stillness (an idle timeout — the proxy closing connections that carry no bytes for N seconds — fatal during a long prefill pause), or ignore a dropped hose end (the client left, but nobody told the stadium, so the GPU keeps generating tokens into the void — you pay for every one).

The cruel part: none of these exist on localhost. This is the canonical "worked in dev, dies in prod" session.

Two closing judgments. Heartbeat cadence: every 15–30s covers every default above at ~0 cost; tie it to your tightest hop. Resumability: Last-Event-ID replay demands server-side stream state — the queue-backed pattern (generate into a store, serve SSE from it) that 2026 guides recommend for mobile-heavy apps; adopt it when reconnect-with-context is a product requirement, not before.

Authentication (verifying who is calling) and authorization (deciding what that identity may do) are the hotel front desk: the keycard proves you're a guest (authn); which floors it opens is authz. Three keycard styles matter here: an API key — a long random secret you issue, store hashed, and look up per request — a plain card the desk checks against its ledger (instantly revocable); a JWT — a JSON Web Token: signed claims (who, what scopes, until when) the server verifies without any lookup — a card with your details embossed and a tamper-proof seal (stateless, but valid until expiry even if you want it dead); and OAuth 2 — the protocol for obtaining tokens via a third party ("sign in with GitHub") without ever seeing passwords. Everything downstream — limits, budgets, caches, audit — hangs off the identity this step establishes.

In FastAPI, auth is just a dependency (S1's machinery doing security work) — resolve identity or raise, and every protected route declares it:

async def current_key(authorization: Annotated[str, Header]) -> ApiKey:
    token = authorization.removeprefix("Bearer ").strip
    key = await keys.lookup(hash256(token))        # store HASHES, never raw keys
    if key is None or key.revoked:
        raise HTTPException(401, "invalid key", headers={"WWW-Authenticate": "Bearer"})
    return key                                     # carries: owner, scopes, tier

KeyDep = Annotated[ApiKey, Depends(current_key)]   # → chat(req, key: KeyDep, …)

Authorization then reads the identity: the material tour — RBAC (role-based: permissions attach to roles like "admin"/"member"), ABAC (attribute-based: rules over attributes — tier, org, time), ReBAC (relationship-based: "may access documents they own/share") — collapses, for a model API, into scopes on the key: chat, chat:thinking, batch, admin. A one-line check per route ("batch" in key.scopes) covers most products; graduate to a policy engine when relationships appear.

API keys vs JWT is revocation vs lookups: keys need a (cached) store hit but die instantly; JWTs verify offline but live until exp — so keep them short-lived (minutes–hours) and pair with refresh tokens. Machine-to-machine LLM APIs overwhelmingly use keys (every provider you've called does); JWTs shine when a human session already exists. Hygiene that's non-negotiable: keys hashed at rest, transmitted only in headers (query params leak into logs — the material XSS caution for GET applies doubly), prefix-identifiable (gg_live_…) so leaked keys are traceable, and rotation supported from day one. LLM twist: identity gates capabilities that cost differently — thinking mode burns multiples of flash mode's tokens, so it's a scope, not a boolean anyone can flip.

Rate limiting is rejecting requests beyond N per time window, per identity — the nightclub bouncer with a per-hour wristband count. Its siblings: throttling — slowing processing instead of rejecting (letting people queue outside); load shedding — deliberately dropping excess when the system nears saturation, to protect everyone already inside (fire-marshal cap); and quota — a longer-horizon allowance, like tokens per day (monthly membership). The LLM twist that changes everything: one guest can drink a hundred times more than another. A request with 30k prompt tokens and max_tokens=2048 is not one unit of load — so mature LLM APIs limit requests per minute AND tokens per minute/day. Requests protect the door; tokens protect the bar.

Token-aware flow: pre-check an estimate before calling (prompt chars/4max_tokens as worst case) against remaining quota, then post-debit actuals from the usage the stream's final chunk delivers (S3's include_usage earning its keep). Off-the-shelf: slowapi decorates routes with per-IP/per-user limits (source §9 walks it, including the storage_uri="redis://…" switch for multi-replica truth and exempting /healthz); it counts requests only — the token dimension you build, or buy at the gateway (S5).

Answer 429s well: include Retry-After and remaining-quota headers — clients that know when to return don't hammer. Key the right identity: per-IP alone is evadable (VPNs, CGNAT also makes it unfair — the material's point); per-key is primary, per-IP a backstop for unauthenticated surface. Streaming wrinkle: you can't debit exact tokens until generation ends — reserve the estimate, reconcile after; accept small overshoot rather than double-counting. Load shedding is the honest last line: your Modal config already sheds at the platform edge (max_inputs=64 per replica, max_containers=2) — when the gateway sees upstream queueing (rising TTFT, 429/503s), shedding early at the gauntlet with a clear 503 beats stacking doomed requests. Throttle streams? Rarely: pacing token delivery adds latency for everyone to slow abusers you should be limiting instead.

Three librarians, three caches. The photocopier clerk is an exact-match cache — store the response under a hash of the byte-identical request; same bytes in, same copy out. The reference librarian is a semantic cache — store responses under an embedding (a vector capturing meaning) of the prompt; a new question close enough in meaning — above a similarity threshold — gets the stored answer: she recognizes "how do I reset my password?" and "password reset — how?" as the same question. And the bookmark on the reading desk is vLLM's prefix cache, which you already know from the inference stage — it never skips the answer, it skips re-reading the shared beginning of the prompt (recomputing KV for a repeated prefix). Different layers, different wins: the first two save the whole generation; the third makes unavoidable generations start faster.

The failure mode that matters is the confident wrong answer. A semantic false hit doesn't error — it serves someone else's answer fluently. Defenses: threshold tuned against a labeled sample, cache keys segmented by anything that changes truth (model, system-prompt version, language, tier), and TTLs. Never cache personalized generations without the user in the key — the classic leak. Economics: the material cites a study where a semantic cache cut API calls ~69% across a large user base; your number is whatever your traffic's paraphrase-repetition rate is — measure hit rate before believing any vendor's. A hit costs ~10–50ms and $0 GPU vs seconds and real money; even 15% hit rate on a head-heavy FAQ workload pays for Redis many times over. Streaming: cache the assembled text (your relay's done path has it), replay as chunks on hit. Thinking mode: cache the answer, not the reasoning — or skip caching thinking requests entirely; they're the expensive, rare, freshness-sensitive tail.

Calling an upstream without protection is phoning with no discipline. The disciplines: a timeout — a deadline after which you abandon the wait — hang up after two minutes on hold, always; a retry with exponential backoff and jitter — re-attempt failed calls with growing, randomized delays — call back only if the line was busy, never if you already placed the order (you'd order twice), and not everyone redialing on the same beat; a circuit breaker — after N consecutive failures, stop calling for a cool-down (open), then probe with one test call (half-open) before resuming (closed) — stop calling the number that's been dead all morning, glance at it hourly; and a budget — a hard cap on spend per identity per period — the phone bill your teenager cannot exceed. Idempotency — an operation safe to perform twice with the same result — is the property deciding what's retryable at all.

Retries multiply load: 3 attempts at 100% failure = 3× traffic at the worst moment — cap attempts, jitter always, and let the breaker end the stampede. Timeout budgets compose: if your client gives you 10s and you give vLLM 10s, you've left zero for yourself — inner deadlines must be strictly smaller than outer ones. Breaker thresholds are judgment: too twitchy (trips on 2 blips) and you fail needlessly; too numb (50 failures) and it never helps — start ~5 consecutive failures / 30s cool-down and tune with S5's load tests. Budget math for your stack: an L4 at roughly $0.80/hr serving ~50 output tok/s ≈ $0.004 per 1k output tokens of compute time — so a 100k-token/day per-key budget bounds each key to pennies of GPU; the point isn't the pennies, it's that runaway loops and leaked keys hit a wall you chose.

You can't assert output == "Paris is the capital…" against a generative model — same input, different phrasing tomorrow. The escape is a clean split along your favorite line, the boundary. Everything amber is deterministic: given this request, does validation reject it? Is the payload to vLLM shaped right? Does the relay frame SSE correctly? Does the gauntlet 429? All exactly testable — if the model server is replaced by a stand-in. That stand-in is a mock (a fake dependency that returns scripted responses and records how it was called) or a stub (the simpler cousin: canned responses, no call recording). Everything cyan — answer quality — gets a different discipline the material frames well: treat model behavior statistically — run N samples, assert properties and rates ("≥95% contain a JSON block", "0/50 leak the system prompt"), not exact strings. That's the world of evals, a later stage; today you build the harness that makes the amber layer provably correct.

The testing pyramid, gateway edition — and the two FastAPI mechanisms that make its base cheap:

Mechanism one — dependency override: FastAPI's built-in switch, app.dependency_overrides[real_dep] = fake_dep, replacing any dependency for the duration of a test. Because S1 injected the upstream client as get_http, swapping the entire model server for a script is one line — the architectural payoff of DI. Mechanism two — fixtures (pytest functions that build reusable setup — a client, a fake upstream — injected into tests by argument name), with conftest.py sharing them across files and parametrize running one test over many cases (the material walks all three, plus setup/teardown via yield).

# the whole trick in eight lines (full harness in the lab)
def fake_vllm(request: httpx.Request) -> httpx.Response:      # scripted "model server"
    return httpx.Response(200, json=OPENAI_SHAPED_REPLY)

transport = httpx.MockTransport(fake_vllm)                    # httpx's built-in stub layer
app.dependency_overrides[get_http] = lambda: httpx.AsyncClient(
    transport=transport, base_url="http://fake")              # S1's DI seam, flipped
# every route now runs its FULL logic — auth, limits, cache, relay — against the script

Mocks drift. A mock encodes your belief about vLLM's responses; when vLLM upgrades (new fields, changed finish_reasons), green tests can lie. Antidotes: build mock payloads from captured real responses (re-record on upgrades), and keep the thin smoke layer against the live endpoint. Test the unhappy paths hardest: 422s for every contract rule, 401/403/429 from the gauntlet, upstream 503 → breaker behavior, malformed SSE lines mid-stream — production is mostly unhappy paths. Flaky ≠ broken: the material example is sobering — a study found GPT-4's accuracy on a prime-number task collapsing from 84% to 51% between two 2023 snapshots. Same API, same prompt, different quarter. Anything asserting on live model output belongs in statistical evals with thresholds, never in the commit-blocking suite; a single-sample "the model answered correctly" test is a coin you'll eventually lose.

A classic load tool reports "p95 request duration: 24s" — and for a streaming endpoint that's useless, maybe great news (long thoughtful answers) or a disaster (users staring at nothing). You already own the right vocabulary from the inference stage; now it becomes the load test's vocabulary: TTFT (time to first token — how long until the user sees anything), ITL (inter-token latency — the pace tokens arrive after that), and the roll-up 2026 practice has settled on, goodput — throughput counting only requests that met their latency targets (e.g. TTFT < 500ms AND ITL < 50ms), not merely completed. Raw throughput says the kitchen shipped 100 plates; goodput says how many arrived hot. A load test that measures request duration on a stream conflates the two numbers users actually feel.

Two more mechanics decide whether results mean anything. Open vs closed loop: a closed-loop test (N virtual users, each waiting for its response before sending again) self-throttles — as the system slows, arrival rate drops, hiding the very overload you're probing. An open-loop test (requests fired at a fixed rate, e.g. Poisson arrivals, regardless of completions) models real users, who don't coordinate — your queueing-theory stage predicted exactly this failure of closed-loop thinking. Tool support: the tool must parse SSE and expose per-event timestamps as custom metrics — plain "HTTP duration" tools measure the wrong thing out of the box.

Judgment calls: load test the gateway and the upstream separately before together — GuideLLM straight at Modal gives the model-server baseline; the gateway run on top isolates your layer's added TTFT (target: single-digit ms warm). Test through the gauntlet with real keys so limiters and caches participate (a 95% cache-hit load test flatters you; disable or vary prompts to probe the miss path). And test the failure modes you built: kill the Modal app mid-run and watch the breaker turn a hang into fast 503s.

§12's map of ways to run this thing — a VM you administer, serverless functions (code that runs per-request on managed infrastructure, billed per invocation), a managed PaaS, or containers — processes shipped with their entire filesystem (Python, dependencies, your code) as one immutable image, so "works on my machine" becomes "works anywhere the image runs". The shipping-container metaphor is literal: standardized box, any ship, any crane. For a gateway the industry answer is settled: containers, because every deploy target left standing — Cloud Run, ECS/Fargate, Kubernetes, even Modal — is fundamentally "give me an image." And note what the boundary buys you at this final step: your image is CPU-only, small, and boring. No CUDA, no weights, no GPU drivers — all of that lives across the boundary in the Modal image you already maintain. Two artifacts, two lifecycles, two bills.

A production image is built in stages — a multi-stage build (a Dockerfile with several FROM stages, where only chosen files from earlier stages are copied into the final image, leaving build tools behind). Walk the five moves; the lab has the full file:

Picking the target is mostly picking who manages what. Cloud Run (or Azure Container Apps): hand over the image, get autoscaling + HTTPS; mind two knobs from S3 — request timeout must exceed your longest stream, and scale-to-zero gives your gateway a cold start too (mild — CPU containers boot in seconds, but stack it mentally atop Modal's GPU cold boot). ECS/Fargate + ALB: more wiring, more control — remember the 60s idle timeout. Kubernetes: you already speak it from the serving stage; the gateway is a plain Deployment + Service + Ingress, no KServe ceremony needed, and its replicas autoscale on CPU while your GPU pool scales on very different signals — independent scaling being precisely the boundary's promise. Sizing: an async gateway is cheap — a 0.5-vCPU/512MB replica comfortably relays hundreds of concurrent streams; you'll scale for redundancy (≥2 replicas) before you scale for load. What stays out: Redis is a managed service or its own container, never inside the gateway image — replicas must share it (S4's whole premise).