Kubernetes & Infrastructure for LLMs

Where the model actually runs

The previous stage taught you what happens inside one GPU: prefill and decode, KV cache, continuous batching, quantization, prefix caching. This stage is about everything around that GPU — getting bytes to it, getting a GPU allocated to you in the first place, and keeping the whole thing alive under real traffic.

Kubernetes was designed to ship 200 MB of stateless code to a CPU. You are shipping 20 GB of stateful weights to a $600/month accelerator. Almost everything hard about this stage comes from that mismatch.

Sessions

All four sessions are in this file — one tab each. §1 was covered in a prior session; S1–S4 are the full stage.

What "done" looks like for this stage

Driving this workbook

Why this section exists

On Modal, you wrote modal.Volume.from_name("huggingface-cache") and forgot about it forever. The weights appeared. That single line was hiding an entire engineering discipline, and Kubernetes will not hide it for you.

Here is the problem in one sentence: a pod cannot serve a single token until every byte of the model is physically present on the node it landed on. Not in a bucket. Not in a registry. On that node's disk, then in its RAM, then in the GPU's memory. Until then you are paying for an idle accelerator and the user is watching a spinner.

A container image is 200 MB of code that changes daily. A model is 20 GB of numbers that change monthly. Kubernetes ships the first one beautifully. It was never asked about the second.

Three questions fall out of that picture, and they are the spine of this session:

1. What am I actually moving? — a model on disk is a directory of files, not a file. Which files, and which ones can you drop?
2. Where does it live before it moves? — Hugging Face, an S3 bucket, a registry, a database of pointers. These are not interchangeable.
3. How does it get onto the node? — five real mechanisms with genuinely different failure modes. This is the part you will be asked about in an interview.

Taught from zero

Four concepts, each in three passes: intuition, then mechanism, then the numbers and the edges where it breaks.

A · What a "model" actually is on disk

A model you download is a box of flat-pack furniture. Inside there are three kinds of thing:

• The planks — the weights. Ninety-something percent of the mass. Just numbers.
• The assembly instructions — how the layers wire together, how many attention heads, what the hidden size is. A file called config.json.
• The little Allen key — the tokenizer. Tiny, unglamorous, and without it the whole thing is useless because you cannot turn text into numbers.

Formats differ mainly in how many of those three they put in the box. A weight-only format ships planks alone and assumes the runtime already owns the manual. A "self-contained" format tries to ship all three.

The punchline of the section's format section: as of 2026 no widely-used format is truly self-contained. The honest label is mostly self-contained. Safetensors leaves out the tokenizer and the architecture. ONNX carries architecture and weights but has no idea what a tokenizer is. GGUF gets closest — weights, metadata and tokenizer in one file — and pays for it by being tied to a narrow set of runtimes. There is no Docker moment for models yet.

Here is what a real Hugging Face repository looks like on disk after download. Click any file.

Two mechanisms in that picture deserve naming, because they come back constantly.

Sharding and the index file. Large models are split into numbered .safetensors files with a companion model.safetensors.index.json that maps every tensor name to the shard holding it. Llama 3.1 405B ships as 30 shards. This exists partly for filesystem sanity and partly because shards can be downloaded in parallel — which turns weight transfer from a single-stream problem into a bandwidth problem you can actually throw money at.

Zero-copy loading. A .safetensors file starts with a JSON header giving every tensor's dtype, shape and byte offset. Because the offsets are explicit, the loader can mmap the file and hand the GPU driver pointers into it, rather than deserializing objects into Python and copying. Compare the format it replaced: PyTorch's .bin uses Python pickle, which is both slower and a remote-code-execution hazard.

B · The distance ladder — where bytes live before they reach the GPU

Stop thinking "download the model". Think: a relay race with five hand-offs, from a registry somewhere on the internet all the way into the GPU's memory. Every strategy in this session is simply a decision about which hand-offs you pay for, and how often you pay for them — once per cluster, once per node, or once per pod.

The costly insight is that last part. A hand-off you pay once per cluster is free at scale. A hand-off you pay once per pod is a tax on every autoscaling event you will ever have.

1. Time to first byte on the node — dominated by which tier you start from, and whether the transfer is parallel. Sequential single-stream registry pulls are the villain of this section.
2. Bytes duplicated across the fleet — node-local approaches store one copy per node. Ten nodes × 20 GB = 200 GB of disk you are paying for to hold one model.
3. Contention under simultaneous cold start — the killer. When an autoscaler adds ten replicas at once, they all reach for the same source at the same moment. A 2 GB/s NFS server serving ten concurrent readers gives each of them 200 MB/s. Your carefully measured single-pod boot time was a lie.

C · The five ways to put weights on a node

There are only three underlying ideas, and five named techniques that implement them:

• Fetch it when the pod starts — simple, always works, slow, repeated forever.
• Keep it somewhere shared and mount it — one copy for the fleet, but every read crosses a network.
• Package it like software — put the weights in a container image and let the container infrastructure you already run do distribution, caching, deduplication and signing for you.

The industry has spent three years moving from the first idea to the third. As of Kubernetes 1.36 the third one is finally a stable, boring, built-in feature.

Before the five techniques, the flow they all plug into. Step through it.

Every technique below is a different answer to step 3 and step 4. Everything else is identical. Expand each card — the YAML fragment shown is the field that is the mechanism, nothing more.

# the mechanism, in four lines
volumes:
- name: models
  emptyDir: {}     # node-local scratch, dies with the pod

# the mechanism: one volume, many simultaneous readers
accessModes:
- ReadOnlyMany      # ROX — many pods mount it read-only at once
persistentVolumeReclaimPolicy: Retain

# the mechanism: let containers see into each other
spec:
  shareProcessNamespace: true
  # sidecar then runs:
  #   ln -s /proc/$$/root/models /mnt/models && sleep infinity

# the mechanism: an image *is* a volume
volumes:
- name: model-volume
  image:
    reference: registry.example.com/gemma-e4b:v1
    pullPolicy: IfNotPresent
# mount with  subPath: models  for modelcar compatibility

D · Model registries — the difference between a pointer and a payload

Two things get called "model registry" and they do opposite jobs.

A catalogue stores metadata and a pointer: name, version, accuracy, lineage, who trained it, and a URL to where the bytes actually live. A content store holds the bytes themselves. MLflow and Kubeflow Model Registry are catalogues. An OCI registry is a content store. Hugging Face Hub is both, which is why it feels so convenient and why it becomes a production dependency you didn't mean to take.

The reason a catalogue earns its keep is one layer of indirection. Instead of pinning a bucket path into every deployment, you pin a logical name and version:

# KServe resolves this through the registry at deploy time
storageUri: model-registry://iris/v1

Move the bytes to a different bucket, a different region, a different storage class — nothing in the deployment changes. The registry translates. That indirection is registered with a ClusterStorageContainer, the same extension point that lets you add any URI scheme you like.

How should weights reach the pod?

Read top to bottom. Follow no ↓ until a yes → exits right. If nothing exits, the bottom-left box is your answer — and it is the right answer most of the time.

Do the arithmetic first

Everything above is a shape. Here is the shape with numbers in it, using your own deployment as the worked example.

Worked example · what it costs to move Gemma E4B onto a GKE node

Assumptions — state them, then verify them

• Model directory ≈ 10 GB (bf16 safetensors + tokenizer + config). Verify with hf download <repo> --dry-run or the file list on the repo page before you trust any number below.
• Target node: g2-standard-8 — 1× L4, 24 GB VRAM, 8 vCPU, us-central1.
• On-demand list price mid-2026: ≈ $0.85/hr (≈ $620/month running 24/7). The g2-standard-4 is ≈ $0.71/hr (≈ $514/month).

Step 1 · Time to get 10 GB onto the node

The whole calculation is seconds = bytes ÷ throughput. What changes is the throughput, and it changes by two orders of magnitude.

Add engine initialisation on top of every row — weight load into VRAM, CUDA graph capture unless --enforce-eager. You already know that cost from the inference stage; the point here is that byte movement can dwarf it by 10×, and byte movement is the part you control with a YAML field.

Step 2 · What the waiting costs

An idle GPU bills exactly like a busy one.

• $0.85/hr = $0.0142 per minute.
• A 4-minute cold start burns $0.057 of GPU doing nothing.
• 30 cold starts a day (a realistic scale-to-zero pattern) = 120 min/day = $1.70/day ≈ $51/month.
• That is ~8% of a $620/month node, spent watching a progress bar.
• Cut the cold start to 30 s and the same churn costs $6/month.

The $45/month is not the real prize. The real prize is that a 30-second cold start makes scale-to-zero viable, which is what lets you run a $620/month node for $80.

Step 3 · The storage bill, which is smaller than the material implies

The material ranks PersistentVolumes as having the highest storage efficiency — one copy, shared. True. But price it:

• Node-local duplication (image volumes, emptyDir): 10 GB × 10 nodes = 100 GB of attached disk. At roughly $0.10/GB-month for balanced persistent disk that is about $10/month. Rounding error.
• Shared file service: managed NFS bills on provisioned capacity, and the entry tiers start in the hundreds of GB to multiple TiB. Price your actual tier — a "one shared copy" of a 10 GB model routinely costs more per month than ten duplicated node-local copies.

Step 4 · The one formula to remember

# per-replica time to ready, under simultaneous scale-up
t_ready  =  (model_bytes / (source_bandwidth / n_replicas))  +  t_engine_init

# the cache tier exists to make the first term ≈ 0 for every
# replica after the first one on a given node.

What changed since the material went to press

Three production examples from the last few months

The same idea, three ways

Career-relevant framing: every capability in this session exists in an open-source form, an NVIDIA form, and a managed cloud form. Knowing the mapping is most of what "knows the ecosystem" means.

Your Gemma deployment, three ways

Now the manifests. Target: GKE, one L4, the same vLLM flags you are already running on Modal. GPU scheduling fields appear here because the pod needs them to run — S2 explains every one of them. Ignore them for today; focus on the volumes.

0 · The Secret — modal.Secret.from_name("huggingface-secret")

kubectl create namespace llm

kubectl create secret generic hf-token \
  --from-literal=HF_TOKEN="$HF_TOKEN" \
  -n llm

Gemma is gated, so any path that downloads from the Hub needs this. Note which of the three options below doesn't — that is a real security win, not a footnote.

Option A · Download at boot — the closest translation of what Modal does today

# deployment-a-download.yaml — init container + emptyDir
apiVersion: apps/v1
kind: Deployment
metadata: { name: gemma-e4b, namespace: llm }
spec:
  replicas: 1
  selector: { matchLabels: { app: gemma-e4b } }
  template:
    metadata: { labels: { app: gemma-e4b } }
    spec:
      nodeSelector: { cloud.google.com/gke-accelerator: nvidia-l4 }
      tolerations:
      - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }

      initContainers:
      - name: fetch-weights
        image: python:3.12-slim
        command: ["sh", "-c"]
        args:
        - pip install -q "huggingface_hub[hf_transfer]" &&
          HF_HUB_ENABLE_HF_TRANSFER=1
          hf download google/gemma-4-E4B-it --local-dir /mnt/models
        env:
        - name: HF_TOKEN
          valueFrom: { secretKeyRef: { name: hf-token, key: HF_TOKEN } }
        volumeMounts:
        - { name: model, mountPath: /mnt/models }

      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.11.0
        args:
        - --model=/mnt/models
        - --served-model-name=gemma-4-e4b
        - --max-model-len=10000
        - --gpu-memory-utilization=0.92
        - --max-num-seqs=256
        - --enable-prefix-caching
        - --async-scheduling
        - --quantization=fp8
        - --kv-cache-dtype=fp8
        - --tensor-parallel-size=1
        - --limit-mm-per-prompt={"image":0,"video":0,"audio":0}
        ports: [{ containerPort: 8000, name: http }]
        resources:
          limits: { nvidia.com/gpu: "1" }
        volumeMounts:
        - { name: model,      mountPath: /mnt/models }
        - { name: vllm-cache, mountPath: /root/.cache/vllm }
        - { name: shm,        mountPath: /dev/shm }
        startupProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 10
          failureThreshold: 60          # 10 minutes to become ready
        readinessProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 10

      volumes:
      - { name: model,      emptyDir: { sizeLimit: 30Gi } }
      - { name: vllm-cache, emptyDir: {} }
      - { name: shm,        emptyDir: { medium: Memory, sizeLimit: 2Gi } }

Two things to notice. The vLLM args lost their shell quoting — your Modal file wraps the --limit-mm-per-prompt JSON in single quotes because it goes through a shell; args: is passed directly to exec, so quotes would become part of the value. The startup probe is doing load-bearing work — without a generous failureThreshold, Kubernetes kills the pod mid-download and you get a crash loop that looks like a model bug.

Option B · PVC — modal.Volume("huggingface-cache"), literally

# pvc-and-hydrate.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: model-weights, namespace: llm }
spec:
  accessModes: [ ReadWriteMany ]   # what the Filestore CSI actually provisions
  storageClassName: standard-rwx
  resources: { requests: { storage: 1Ti } }   # check the tier minimum — it bites
---
apiVersion: batch/v1
kind: Job                              # run once, populate the share, exit
metadata: { name: hydrate-gemma, namespace: llm }
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: fetch
        image: python:3.12-slim
        command: ["sh", "-c"]
        args:
        - pip install -q "huggingface_hub[hf_transfer]" &&
          HF_HUB_ENABLE_HF_TRANSFER=1
          hf download google/gemma-4-E4B-it --local-dir /weights/gemma-4-e4b
        env:
        - name: HF_TOKEN
          valueFrom: { secretKeyRef: { name: hf-token, key: HF_TOKEN } }
        volumeMounts: [{ name: w, mountPath: /weights }]
      volumes:
      - name: w
        persistentVolumeClaim: { claimName: model-weights }

Then in the Deployment, swap the model volume and drop the init container entirely:

        volumeMounts:
        - name: model
          mountPath: /mnt/models
          subPath: gemma-4-e4b
          readOnly: true              # enforce ROX at the mount
      volumes:
      - name: model
        persistentVolumeClaim:
          claimName: model-weights
          readOnly: true

Option C · OCI image volume — the default from the decision tree

First build a model image. Weights go in /models so the same image also works as a modelcar later.

# Dockerfile.model
FROM python:3.12-slim AS fetch
RUN pip install --no-cache-dir "huggingface_hub[hf_transfer]"
ENV HF_HUB_ENABLE_HF_TRANSFER=1
ARG REPO=google/gemma-4-E4B-it
ARG REV=<pin-a-commit-sha>
RUN --mount=type=secret,id=hf \
    HF_TOKEN=$(cat /run/secrets/hf) \
    hf download $REPO --revision $REV --local-dir /models

FROM scratch                          # nothing executes; smallest possible image
COPY --from=fetch /models /models

export IMG=us-central1-docker.pkg.dev/$PROJECT/models/gemma-e4b:v1

docker build -f Dockerfile.model --secret id=hf,env=HF_TOKEN -t $IMG .
docker push $IMG
crane digest $IMG        # deploy by digest, not tag, if you care about reproducibility

Then the volume is three lines and the HF token disappears from the cluster entirely:

        volumeMounts:
        - name: model
          mountPath: /mnt/models
          subPath: models              # forward/backward compatible with modelcars
          readOnly: true
      volumes:
      - name: model
        image:
          reference: us-central1-docker.pkg.dev/PROJECT/models/gemma-e4b:v1
          pullPolicy: IfNotPresent    # pull once per node, then reuse the layer cache

Modal → Kubernetes, the model-data rows

The full mapping table is the S4 capstone. These are the rows this section owns.

Exercise

Why this section exists

Install Kubernetes on a machine with an H100 in it. Run kubectl describe node. Under Capacity you will see cpu, memory, pods, ephemeral-storage. You will not see a GPU. Not "0 GPUs" — the concept does not exist.

Kubernetes natively understands exactly two compute resources: CPU and memory. Everything else is a plugin somebody had to write. Before a pod can ask for an accelerator, four separate pieces of software have to cooperate to teach the cluster that accelerators are a thing, which ones this node has, and how to hand one to a container.

On Modal you wrote gpu="L4:1" and a GPU appeared. That string was a purchase order. In Kubernetes it is a claim against an inventory that does not exist until you build it.

Three questions structure this session:

1. How does a GPU become something the scheduler can allocate? — and what does that abstraction hide from you.
2. How do I get the right pod onto the right GPU? — labels, selectors, taints, and the new declarative model that is replacing all of it.
3. What if I need less than one GPU, or more than one? — MIG, time-slicing, MPS in one direction; tensor and pipeline parallelism in the other.

Taught from zero

A · How a GPU becomes schedulable

A contractor walks into the building on Monday. They are physically present and fully capable, and yet no work can be assigned to them, because four different systems don't know they exist:

• Facilities has to install their laptop and software so the machine actually works — that's the GPU driver and container toolkit.
• HR has to notice a person is in the building — that's Node Feature Discovery.
• Someone has to record what they're qualified to do — that's GPU Feature Discovery.
• The staffing system has to carry them as available headcount so work can be assigned — that's the device plugin.

The entire user-facing surface is this:

resources:
  limits:
    nvidia.com/gpu: 1   # an "extended resource" — a plain integer count

Step through what that triggers.

Three consequences of modelling a GPU as a number, all of which you will hit:

1. Extended resources must be whole integers. You cannot request nvidia.com/gpu: 0.5. Every sub-GPU technique later in this session is a workaround for that one restriction.
2. Limits always equal requests. Kubernetes forbids overcommitting extended resources. With CPU you can request 0.5 and burst to 2; with GPUs there is no burst, no oversubscription, no throttling. It is allocated or it isn't.
3. The scheduler sees a count, not a capability. A cluster with H100s and T4s reports nvidia.com/gpu on both. A pod asking for one GPU is equally happy with either — right up until your 20 GB model lands on a 16 GB T4 and OOMs at load time.

B · Steering pods — selectors, affinity, taints and tolerations

• nodeSelector — "I need a desk in the east wing." A hard requirement, expressed as exact label matches. Simple and blunt.
• Node affinity — the same idea with grammar: required or preferred, set operators, weights, multiple fallback tiers.
• Taints and tolerations — the inverse. The node pushes pods away, and only pods carrying the matching pass are allowed to stay.

Cloud providers automatically taint GPU node pools so that ordinary workloads don't squat on hundred-dollar-a-day hardware. GKE applies nvidia.com/gpu=present:NoSchedule. That means your GPU pod needs a toleration and a way to be steered — because the toleration alone only says "I am allowed on GPU nodes", not "put me on one".

# permission — get past the node pool's taint
tolerations:
- { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }

# steering — actually land on the right hardware
nodeSelector: { cloud.google.com/gke-accelerator: nvidia-l4 }

Everything above is a workaround for the device plugin's core limitation: it can only count. DRA replaces counting with describing.

With Dynamic Resource Allocation a pod stops saying "give me one of the things called nvidia.com/gpu" and starts saying "give me a device from the nvidia-gpu class whose memory attribute is greater than 40 GiB". The scheduler and a vendor-supplied driver resolve that at bind time, not at manifest-writing time.

# the mechanism: a CEL expression over device attributes
selectors:
- cel:
    expression: "device.attributes['gpu.nvidia.com'].memory.isGreaterThan(quantity('40Gi'))"

What this buys you beyond tidiness: allocation can consider things a static count cannot — current utilisation, NVLink topology between the specific cards being handed out, power state, MIG profile availability. And it makes sharing a first-class API concept rather than a device-plugin config hack.

C · Splitting one GPU four ways

• Whole GPU — you have the kitchen to yourself. Everything works, nothing is contended, and if you only ever boil one egg you have wasted a kitchen.
• Time-slicing — four cooks take turns at the one stove. Nobody has their own fridge shelf, so one cook can fill the fridge and starve everyone else. No memory isolation at all.
• MPS — the cooks work at the stove simultaneously under one shared plan, rather than swapping in and out. Better utilisation and lower latency than taking turns; still one fridge.
• MIG — the kitchen is physically walled into seven small kitchens, each with its own stove and its own fridge. Real isolation. Also: you can never cook anything that needs more than one small kitchen.

Time-slicing is configured on the device plugin, not in your workload. The whole feature is two fields:

sharing:
  timeSlicing:
    renameByDefault: true    # expose as nvidia.com/gpu.shared, not nvidia.com/gpu
    resources:
    - name: nvidia.com/gpu
      replicas: 8            # one physical card now advertises 8 schedulable units

Set renameByDefault. Without it, a node with one L4 reports nvidia.com/gpu: 8 and every existing manifest in your cluster silently starts landing eight pods on one card.

Blunt version: for LLM inference you almost always want the whole GPU. Three reasons, all downstream of how vLLM works.

1. vLLM claims the VRAM up front. Your own config sets --gpu-memory-utilization 0.92, which pre-allocates 92% of the card for weights plus KV cache. Two such pods time-slicing one L4 do not each get 46% — the first one to start takes 92% and the second fails to allocate. Time-slicing shares compute time, never memory.
2. The KV cache wants every spare byte. Throughput on a decode-bound workload is roughly proportional to how many sequences fit in the KV cache. Giving away half the VRAM does not halve throughput, it collapses your batch size and hurts more than half.
3. Requesting two time-sliced GPUs is a trap. nvidia.com/gpu: 2 in shared mode gets you a slice on two different physical cards, each shared with strangers. That is almost never what anyone means. Configure the plugin to reject it.

MIG is the exception that proves the rule: it is genuinely useful when you host many small models. Seven 5 GB models on one A100 each get a hardware-isolated mini-GPU with a guaranteed memory fence. But MIG cannot help a model that needs more than one slice, and the slice sizes are fixed profiles — there is no 6 GB option if NVIDIA didn't ship one.

Whole GPU, MIG, time-slicing or MPS?

Follow no ↓ until a yes → exits right. Bottom-left is the default, and for LLM serving it is the default by a wide margin.

D · Where GPU nodes come from

A GPU is welded to a VM shape. You cannot add a GPU to a running node, and you cannot rent 0.4 of one. The unit of capacity is the node pool, and adding one node means booting a whole machine, installing drivers, and pulling images before a single token gets served.

This is the part that surprises people migrating from a serverless platform. There are two independent control loops, and the slow one is invisible from your Deployment.

1. Pod-level — an HPA (or KEDA, or Knative) decides more replicas are needed and creates pods. Fast: seconds.
2. Node-level — those pods sit Pending because no node has a free GPU. A cluster autoscaler notices and provisions a machine. Slow: minutes, and it can fail outright if the region is out of L4s.

Spot / preemptible GPUs. Reclaimed with about 30 seconds' notice on GCP and two minutes on AWS. The discount varies enormously by SKU and region: scarce datacentre cards like A100 and H100 routinely run 60–70% below on-demand, while smaller inference cards sometimes show discounts in the 10–20% range because demand is steady. Check the live rate for your exact SKU and region rather than assuming a rule of thumb. For inference, spot is viable only when three things are true: cold start is short (Session 1's work), you keep an on-demand baseline for the floor, and your router can drain a node inside the notice window.

When one GPU isn't enough. The inverse problem, and the vocabulary is worth owning:

Do the arithmetic first

Worked example · what a GPU node pool actually costs you

Step 1 · The baseline bill

One g2-standard-8 (1× L4, 24 GB) in us-central1, mid-2026 list:

• On-demand ≈ $0.85/hr → ≈ $620/month running 24/7.
• Committed-use discounts on GPU machines are shallow — roughly 13% for 1 year, 19% for 3 years, versus 37–55% on general-purpose CPU shapes. Do not budget as if GPU CUDs behave like CPU CUDs.
• Add the control plane: GKE Standard charges a per-cluster fee of about $0.10/hr ≈ $73/month. Autopilot bills per-pod-request instead.

Step 2 · The question nobody asks — is MIG on a big card cheaper per GB?

Intuition says "one A100 split seven ways must beat seven small cards". Check it.

So MIG slicing an A100 costs about 3× more per GB-hour than simply renting L4s, and you also lose a little capacity to partition overhead. MIG's value is isolation and density on hardware you already own — not price per gigabyte. If the question is "cheapest way to serve seven small models", the answer is usually seven small cards, or one card with seven models loaded into one vLLM process.

Rates are illustrative list prices for the arithmetic; re-derive with your region's live numbers before quoting them.

Step 3 · Sizing the pool for Gemma

Your README's own L4 benchmark shape: batch 1 gives 41.5 tok/s, batch 32 gives 934 tok/s. Take the high end as the steady-state ceiling for one replica.

# requests one replica can absorb, at 100 output tokens each
934 tok/s ÷ 100 tok/req  ≈  9.3 req/s  ≈  800,000 requests/day

# your stated workload
10,000 req/day  →  0.116 req/s average,  ~0.6 req/s at a 5x peak

# headroom
9.3 / 0.6  ≈  15x

Conclusion: one replica covers your peak fifteen times over. You do not have a scaling problem, you have a utilisation problem — a $620/month node running at about 6% of capacity. That reframes the whole exercise: the money is in scale-to-zero and cold-start work, not in autoscaling. Session 3 picks this up.

What changed since the material went to press

Three things happening right now

The same idea, three ways

Give your Gemma pod a GPU

1 · Create the node pool — this is what gpu="L4:1" really costs

gcloud container node-pools create l4-pool \
  --cluster=llm-cluster --region=us-central1 \
  --machine-type=g2-standard-8 \
  --accelerator=type=nvidia-l4,count=1,gpu-driver-version=latest \
  --num-nodes=0 --enable-autoscaling --min-nodes=0 --max-nodes=3 \
  --node-labels=workload=llm \
  --spot                      # drop this for the on-demand baseline pool

# GKE auto-applies the taint  nvidia.com/gpu=present:NoSchedule
# and installs the driver DaemonSet because of gpu-driver-version=latest.

The EKS equivalent, as a Karpenter NodePool rather than a fixed group:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: gpu }
spec:
  template:
    spec:
      requirements:
      - { key: node.kubernetes.io/instance-type, operator: In, values: ["g6.2xlarge"] }
      - { key: karpenter.sh/capacity-type,       operator: In, values: ["spot", "on-demand"] }
      taints:
      - { key: nvidia.com/gpu, value: "present", effect: NoSchedule }
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: gpu }
  limits: { "nvidia.com/gpu": 8 }
  disruption: { consolidationPolicy: WhenEmptyOrUnderutilized }

2 · The Deployment, with Session 1's image volume and this session's GPU fields

apiVersion: apps/v1
kind: Deployment
metadata: { name: gemma-e4b, namespace: llm }
spec:
  replicas: 1
  selector: { matchLabels: { app: gemma-e4b } }
  template:
    metadata: { labels: { app: gemma-e4b } }
    spec:
      # --- steering: land on an L4, not "any GPU" -----------------
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-l4
      # --- permission: get past the node pool's taint --------------
      tolerations:
      - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }

      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.11.0
        args:
        - --model=/mnt/models
        - --served-model-name=gemma-4-e4b
        - --max-model-len=10000
        - --gpu-memory-utilization=0.92
        - --max-num-seqs=256
        - --enable-prefix-caching
        - --async-scheduling
        - --quantization=fp8
        - --kv-cache-dtype=fp8
        - --tensor-parallel-size=1     # one card. no TP. correct for E4B.
        - --limit-mm-per-prompt={"image":0,"video":0,"audio":0}
        ports: [{ containerPort: 8000, name: http }]
        # --- the allocation: integer, limits == requests -------------
        resources:
          limits:
            nvidia.com/gpu: "1"
            cpu: "6"
            memory: 24Gi
          requests:
            cpu: "4"
            memory: 16Gi
        volumeMounts:
        - { name: model, mountPath: /mnt/models, subPath: models, readOnly: true }
        - { name: vllm-cache, mountPath: /root/.cache/vllm }
        - { name: shm, mountPath: /dev/shm }
        startupProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 10
          failureThreshold: 60
        readinessProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 10
        # NOTE: no livenessProbe. A liveness probe that fires during a slow
        # model load will kill the pod and restart the load. Forever.
      volumes:
      - name: model
        image:
          reference: us-central1-docker.pkg.dev/PROJECT/models/gemma-e4b:v1
          pullPolicy: IfNotPresent
      - { name: vllm-cache, emptyDir: {} }
      - { name: shm, emptyDir: { medium: Memory, sizeLimit: 2Gi } }

3 · The DRA version — what this becomes next

# Instead of nodeSelector + integer count, describe the device you need.
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata: { name: l4-claim, namespace: llm }
spec:
  spec:
    devices:
      requests:
      - name: gpu
        deviceClassName: nvidia-gpu
        count: 1
        selectors:
        - cel:
            expression: "device.attributes['gpu.nvidia.com'].memory.isGreaterThan(quantity('20Gi'))"
---
# ...and in the pod spec, replace resources.limits with:
        resources:
          claims: [{ name: gpu }]
      resourceClaims:
      - { name: gpu, resourceClaimTemplateName: l4-claim }

Read the difference out loud: the first version says "an L4 node, and one of whatever nvidia.com/gpu means there". The second says "a GPU with more than 20 GiB". Only one of those survives the day you add H100s to the cluster.

4 · Diagnostics you will use constantly

# What does the node think it has?
kubectl get nodes -o custom-columns=\
NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu'

# Which GFD labels exist to select on?
kubectl get node <node> -o jsonpath='{.metadata.labels}' | tr ',' '\n' | grep nvidia

# Why is my pod Pending? (read the Events, not the Status)
kubectl describe pod <pod> | sed -n '/Events:/,$p'

# Talk to the actual hardware
kubectl exec -it deploy/gemma-e4b -- nvidia-smi

Modal → Kubernetes, the GPU rows

Exercise

Why this section exists

You now have one replica that boots fast and holds a GPU. Everything from here answers two questions a single replica never has to face: when do I add another one, and which one gets this request.

Kubernetes has confident, well-tested, twelve-year-old answers to both. Scale on CPU utilisation. Distribute round-robin. Both answers are correct for the workload Kubernetes was designed around, and both are wrong here — not subtly wrong, but wrong in the direction that makes your service slower the more machinery you add.

A microservice replica is interchangeable and its requests are uniform. An LLM replica is neither. It has state you want to reuse, and every request costs a different, unpredictable amount.

Taught from zero

A · Autoscaling — the signal matters more than the autoscaler

An autoscaler is a thermostat: measure something, compare it to a target, add or remove capacity. Almost all the discussion about which autoscaler to use is misplaced energy. The only decision that really matters is what you measure. Autoscaling an LLM on CPU utilisation is checking the room temperature with a barometer — the instrument works perfectly and tells you nothing.

vLLM exposes Prometheus metrics on /metrics without any extra work. Four of them are the candidates worth scaling on:

Now the autoscalers, in the order you should consider them:

annotations:
  serving.kserve.io/deploymentMode: Standard
  serving.kserve.io/autoscalerClass: "keda"

Thrashing. Queue depth is spiky by nature. Pair any aggressive metric with a stabilisation window longer than your pod start time — otherwise the autoscaler adds a replica, the spike passes, it removes the replica, and the replica was never ready in the first place. You have paid five minutes of GPU for nothing and done it repeatedly.

Scale-to-zero. The material's position is that it "remains impractical for most LLM deployments due to model loading times measured in minutes". That was the right conclusion given minutes-long loads — and it is exactly what Session 1 was for. The honest 2026 framing is a calculation, not a rule:

# scale-to-zero pays when idle time saved > cold-start time wasted
saved   = (24h − active_hours) × $/hr
wasted  = cold_starts_per_day × cold_start_seconds ÷ 3600 × $/hr

# at $0.85/hr, 6 active hours, 30 cold starts of 45 s:
saved  = 18 × 0.85         = $15.30/day
wasted = 30 × 45/3600 × .85 = $0.32/day

The interesting term is user-visible cold start, not the dollar cost. $0.32 is nothing; a 45-second wait for the first request of the morning may or may not be acceptable. That is a product decision, and now you can price both sides of it.

B · LLM-aware routing — the largest single win in this session

You ask a librarian a follow-up question about a source. They send you to a random desk. The librarian at desk 3 has the material open in front of them and has read the first two hundred pages; you get sent to desk 7, who starts from the cover.

That is round-robin in front of a fleet with prefix caching enabled. Every replica has a KV cache full of conversations it has already processed, and the load balancer — which knows nothing about any of that — systematically routes each turn of a conversation to a different one. You paid for prefix caching in the last stage and the load balancer is throwing it away.

An LLM-aware router considers things a Service cannot:

• Queue depth per replica — vllm:num_requests_waiting. Send work to the least-loaded engine, not the next one in the ring.
• KV cache / prefix state — which replica already holds the prefix for this conversation. Called prefix-aware or cache-aware routing.
• Prefill vs decode role — if the fleet is disaggregated, prefill-heavy requests belong in a different pool.
• Priority / SLO class — interactive traffic outranks batch, and under pressure the router can shed the batch rather than degrade everyone.
• LoRA adapter placement — with --enable-lora, one vLLM process serves a base model plus several adapters. The one-model-per-endpoint assumption breaks, and the router has to know which pod has which adapter loaded.

The standard way to build this is the Gateway API Inference Extension plus an Endpoint Picker. Two objects and one protocol:

kind: InferencePool
spec:
  selector: { app: gemma-e4b }
  targetPorts: [{ number: 8000 }]
  endpointPickerRef: { name: gemma-epp }   # the brain lives here

Walk the request path — this is the part worth being able to draw on a whiteboard.

What it buys. The llm-d project reports an order-of-magnitude TTFT reduction against a round-robin baseline on its benchmark topologies. That number is workload-dependent and comes from the project itself, so treat it as a direction rather than a promise — but the direction is not in doubt. Cache-aware routing turns prefix caching from a per-replica optimisation into a fleet-wide one, and the payoff scales with how much your prompts repeat. RAG, agents and multi-turn chat repeat enormously.

What it costs. An extra component in the request path that must be highly available and fast, plus metric-scraping traffic to every pod, plus a new failure mode: an EPP that returns a bad endpoint is worse than round-robin. It also only pays when you have several replicas. With one replica an Endpoint Picker is pure overhead.

C · Disaggregated serving — the appliance end of the spectrum

You already know that prefill is compute-bound and decode is memory-bandwidth-bound. Running both on the same GPU means the hardware is permanently the wrong shape for one of them: while a long prompt is prefilling, the memory bandwidth sits idle; while tokens stream out, the compute units do.

Disaggregation splits them into two pools that can be sized, tuned and scaled independently. A prefill-heavy workload — RAG with 8,000-token contexts — gets more prefill replicas without paying for decode capacity it will not use.

Splitting the phases creates one hard problem: prefill produces the KV cache, and decode needs it. Those blocks have to move between pods, fast, or you have made everything worse.

• LMCache — "Redis for LLMs". An external, shareable KV block store, so blocks outlive a single replica and can be reused across the fleet.
• NIXL (NVIDIA Inference Xfer Library) — a thin point-to-point transfer library that abstracts over GPU memory, CPU memory, files and object storage. Usable even without disaggregation, e.g. to spill KV cache into host RAM for a bigger effective cache.

And here is the gate that keeps most people out:

# the bandwidth arithmetic that decides whether you can do this at all
standard pod networking        10–20 Gbps
KV-block transfer requirement  500–600 Gbps      # ~30x more

# what actually clears the bar
NVLink / NVSwitch    up to ~900 GB/s intra-node
InfiniBand / RoCE    up to ~800 Gbps  (RDMA, bypasses the CPU)

Without dedicated high-bandwidth networking, disaggregated serving is slower than not doing it. This is not a tuning parameter; it is a hardware purchase.

How should I scale and route this?

Follow no ↓ until a yes → exits right. Bottom-left is the default, and most production LLM services should genuinely stop there.

Do the arithmetic first

Worked example · choosing a KEDA threshold instead of guessing one

Everyone picks a number out of the air here. Derive it instead.

Step 1 · Find the knee

Run a concurrency sweep against one replica and record p95 TTFT at each level. Using the shape from your own L4 benchmark (batch 1 → 987 ms; batch 32 → 99 ms per-token latency at 934 tok/s aggregate), a realistic curve looks like this:

Throughput stops improving after the knee but latency keeps degrading — the classic saturation shape. The knee is your scaling target, not your maximum.

Step 2 · Turn the knee into a threshold

# scale out when the queue starts growing, i.e. the engine is past the knee
target on  vllm:num_requests_waiting  =  4

# reasoning: max_num_seqs is 256, but useful concurrency saturates near 32.
# A persistent waiting queue of 4 means arrivals now exceed the knee,
# and there is enough lead time to boot a replica before p95 breaks.

# pair it with a stabilisation window LONGER than pod start time
cooldownPeriod: 300        # 5 min — do not thrash a 3-minute boot
pollingInterval: 15

Step 3 · Sanity-check against your actual traffic

From Session 2: your workload peaks near 0.6 req/s and one replica absorbs about 9 req/s. The queue will never reach 4. The autoscaler you just configured will correctly never fire — and that is the right outcome, not a wasted afternoon. You have a documented, defensible threshold that will start working the day traffic grows 15×.

The money at your scale is elsewhere:

• Always-on single replica: $620/month, running at roughly 6% utilisation.
• Scale-to-zero with a 5-minute idle window and bursty daytime traffic — say 6 active hours: ≈ $155/month.
• Cold-start penalty at 30 starts/day × 45 s: ≈ $10/month and a 45-second wait for whoever arrives first.

Net: about $455/month saved, entirely because Session 1 made the cold start short enough to make Guard 1 answerable.

What changed since the material went to press

Three things happening right now

The same idea, three ways

Scale and route the Gemma deployment

1 · Expose the metrics vLLM is already producing

apiVersion: v1
kind: Service
metadata:
  name: gemma-e4b
  namespace: llm
  labels: { app: gemma-e4b }
spec:
  selector: { app: gemma-e4b }
  ports: [{ name: http, port: 80, targetPort: 8000 }]
---
apiVersion: monitoring.googleapis.com/v1     # GKE managed Prometheus
kind: PodMonitoring
metadata: { name: gemma-e4b, namespace: llm }
spec:
  selector: { matchLabels: { app: gemma-e4b } }
  endpoints:
  - { port: 8000, path: /metrics, interval: 15s }

Confirm before going further — if this returns nothing, everything below is decoration:

kubectl exec -it deploy/gemma-e4b -n llm -- \
  curl -s localhost:8000/metrics | grep -E 'num_requests_(waiting|running)'

2 · KEDA on the queue depth

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: gemma-e4b, namespace: llm }
spec:
  scaleTargetRef: { name: gemma-e4b }
  minReplicaCount: 0          # Modal's min_containers=0
  maxReplicaCount: 2          # Modal's max_containers
  cooldownPeriod: 300         # Modal's scaledown_window (5 min)
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300   # longer than pod start. non-negotiable.
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc:9090
      metricName: vllm_num_requests_waiting
      query: sum(vllm:num_requests_waiting{service="gemma-e4b"})
      threshold: "4"           # derived above, not guessed

3 · The whole thing as one KServe resource

Everything above — Deployment, Service, autoscaler, and the routing layer — collapses into this:

apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata: { name: gemma-e4b, namespace: llm }
spec:
  model:
    uri: oci://us-central1-docker.pkg.dev/PROJECT/models/gemma-e4b:v1
    name: gemma-4-e4b
  replicas: 1
  router:
    scheduler: {}       # generates the InferencePool + Endpoint Picker
    gateway: {}         # attaches to your Gateway
  template:
    containers:
    - name: main
      resources:
        limits: { nvidia.com/gpu: "1" }
      args: ["--max-model-len=10000", "--enable-prefix-caching", "--quantization=fp8"]

Name what the controller just did for you — this is an exit criterion for the stage. It generated the Deployment, the Service, the InferencePool, the Endpoint Picker deployment and its RBAC, the HTTPRoute onto your Gateway, and the autoscaler wiring. Everything in steps 1 and 2, plus the routing layer you had not built yet. Check the API version against the KServe release you install — LLMInferenceService is moving quickly.

4 · Two production details that are easy to skip

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: gemma-e4b, namespace: llm }
spec:
  minAvailable: 1
  selector: { matchLabels: { app: gemma-e4b } }
# Without this, a node upgrade or a spot reclamation can take your
# only replica away and you eat a full cold start in front of users.
---
# And in the pod spec: give in-flight generations time to finish.
terminationGracePeriodSeconds: 120
# A streaming response can run 30+ seconds. The 30 s default cuts it off.

Modal → Kubernetes, the scaling rows

Exercise

Should you migrate at all?

Three sessions have taught you how to move a model onto Kubernetes. This one starts by asking whether you should — because the most valuable thing a platform engineer can produce is sometimes a well-argued no.

§1 of this material put it as a ladder, and the rule from that session still holds: Kubernetes is an exit you earn, not a default. Your Modal deployment works. It scales to zero. It cost you roughly one file. The question is not "is Kubernetes better" — it is "what specifically does Modal not give me, and is that worth the operational surface I am about to take on?"

Serverless GPU platforms sell you the elimination of a problem. Kubernetes sells you control over it. You should only buy control when you have a use for it.

Where should this model actually run?

Follow no ↓ until a yes → exits right. Bottom-left is the default — and for a model serving 10,000 requests a day, the default is genuinely the right answer.

gemma_modal.py, field by field

Every line of your Modal file and what it becomes. Sessions 1–3 covered these in pieces; this is the whole thing in one place.

The cost model, done properly

Three billing shapes, and the only honest way to compare them is to compute the crossover rather than quote three monthly numbers.

The three shapes

The crossover, in one formula

# let R_s = serverless rate per GPU-hour
#     R_n = node rate per hour (incl. amortised control plane)
#     D   = duty cycle, i.e. active hours ÷ 730

serverless_month  =  R_s × 730 × D
selfmanaged_month =  R_n × 730          # node runs regardless

# break-even duty cycle:
D* = R_n / R_s

# worked, with R_n = 0.85 + 0.10 control plane = 0.95
# and a serverless rate of, say, 1.60/GPU-hour:
D* = 0.95 / 1.60 = 0.59   →   about 59% duty cycle, ~430 h/month

Below 59% busy, serverless is cheaper. Above it, the node is. Substitute your own numbers — pull the serverless rate from your Modal usage dashboard rather than trusting the figure above, which is illustrative only. Then look at your actual duty cycle. At 10,000 requests a day on a replica that handles nine per second, you are nowhere near 59%.

What the formula leaves out — and shouldn't

Three phases, each independently valuable

Structured so you can stop after any phase and still have gained something. Modal stays up the whole time — this is a parallel build, not a cutover.

Assembling the manifests

Everything from the three labs, in apply order:

llm-platform/
├── 00-namespace.yaml            # namespace: llm
├── 01-secret.yaml               # S1 — only if NOT baking weights
├── 02-node-pool.sh              # S2 — gcloud / Karpenter NodePool
├── 03-deployment.yaml           # S1 image volume + S2 GPU fields
├── 04-service.yaml              # S3
├── 05-podmonitoring.yaml        # S3
├── 06-scaledobject.yaml         # S3 — KEDA
├── 07-pdb.yaml                  # S3
└── kserve/
    └── llminferenceservice.yaml # Phase 3 — replaces 03 through 07

# build the model image first (S1), then:
kubectl apply -f 00-namespace.yaml
bash 02-node-pool.sh
kubectl apply -f 03-deployment.yaml -f 04-service.yaml
kubectl rollout status deploy/gemma-e4b -n llm --timeout=15m
kubectl apply -f 05-podmonitoring.yaml -f 06-scaledobject.yaml -f 07-pdb.yaml

The parity test — run this before you trust anything

# same prompt, both endpoints, temperature 0, compare token-for-token
for URL in "$MODAL_URL" "$K8S_URL"; do
  curl -s "$URL/v1/chat/completions" \
    -H 'Content-Type: application/json' \
    -d '{"model":"gemma-4-e4b","temperature":0,
         "messages":[{"role":"user","content":"What is EBITDA?"}]}' \
  | jq -r '.choices[0].message.content'
done

# If they differ: check --revision, --quantization and --kv-cache-dtype.
# FP8 on-the-fly quantization is not bit-identical across vLLM versions.

The runbook you should write before going live

• Pod Pending → check taints, then GPU allocatable on nodes, then node pool quota. In that order.
• Pod CrashLoopBackOff during load → almost always the startup probe firing too early. Raise failureThreshold.
• OOM at model load → wrong GPU SKU (missing nodeSelector), or gpu-memory-utilization too high for the KV cache. You already learned 0.95 OOMs on an L4; write it down.
• Cold starts got slow again → the node layer cache is cold. Check whether the pod landed on a new node.
• Autoscaler thrashing → stabilisation window shorter than pod start time.
• Spot node vanished mid-generation → PodDisruptionBudget plus an on-demand baseline replica.

Exit criteria, checked

The six things this stage promised, and where each one was earned.