Compute — how much of the machine do you still want to touch?

Somebody sized a server for the worst Tuesday of the year, and then paid for that Tuesday 365 days running.

That is the oldest cost mistake in infrastructure and the cloud did not fix it — it just made it easier to commit at scale. Before you can pick between a VM, a container, and a function, you need to see the gap the cloud is actually selling you: the difference between capacity you provisioned and capacity you used.

The cloud's answer is a third option — make the capacity line follow the demand curve — and the whole rest of this stage is machinery for doing that. But the machinery only works if your application will tolerate being replicated and killed at will. So we start there.

1 · The ladder, properly

min_instances: 2        # the floor you pay for even at 3am
max_instances: 20       # the ceiling that stops a runaway bill
metric: cpu_utilization # the signal the controller watches
target: 60              # the number it steers toward

2 · Stateless applications — why this is the hinge

class ProductManager:
    self.cache = {}                     # ← instance variable: STATEFUL, breaks replication
    def get(name): return self.cache.setdefault(name, db.get(name))

class ProductManager:
    def get(name): return db.get(name)  # ← temporary only: STATELESS, replicates freely

3 · Replicable applications — running reliably on unreliable parts

4 · The autoscaling control loop

Every autoscaler is a thermostat. Walk the loop:

5 · Decision tree — VM, container, or serverless?

Note what is not in this tree: cost. Cost is a consequence of the shape, not an input to it — and once the shape is fixed, the purchasing model (on-demand vs spot vs committed) is where the real money is. That is a separate decision, taken per session below.

6 · The same ladder in three dialects

Worked example — the same API, three shapes

The workload. A stateless HTTP API. 50 million requests per month. Average 200 ms of work per request. Diurnal traffic: mean 19 req/s, peak 77 req/s, trough 5 req/s. One 2-vCPU instance safely serves about 20 req/s of this workload. Reference instance price $0.10/hour (a general-purpose 2-vCPU class, us-east-1, mid-2026), 730 hours in a month, one load balancer at about $22/month.

Read the result honestly. The always-on fleet is the clear loser and that is the lesson of the session. But the interesting finding is the near-tie between the other two. Serverless has no floor and a higher slope; the fleet has a $168/month floor and a lower slope. Below roughly 50M requests/month, serverless wins outright. Above it, the fleet pulls ahead — but only if you keep it busy. The break-even is not a request count, it is a utilisation number: at this configuration the fleet stops winning once its average utilisation drops below about two-thirds.

Three current sources worth reading in full

Where you have already done this without calling it this

• Your Modal deployment of Gemma via vLLM is the top of the ladder with a GPU bolted on. Scale-to-zero, per-second billing, no instances you name — that is the serverless-container rung. What you learned there transfers directly to Cloud Run GPU and Azure Container Apps serverless GPU; those are the same product shape from the hyperscalers. The thing to notice is what Modal hid from you: the cold-start cost of pulling an image and loading weights. In S4 we put a number on it.
• Your Vertex AI script — T4 GPUs, autoscaling 1 to 3, spot instances — is the managed-VM-group rung. "Min 1, max 3" is exactly the min_instances / max_instances pair from the config fragment above. The min 1 is the important choice: you decided this endpoint keeps a warm floor rather than scaling to zero, which is the correct instinct for inference serving and the wrong one for a training job.
• Spot for training, warm for serving is the workload-shape rule applied to GPUs, and it falls straight out of the statelessness discussion. A training job checkpoints — it tolerates being killed, so interruptible capacity is nearly free money. An inference endpoint holds no state either, but it holds a latency promise, and a two-minute eviction notice mid-request breaks that promise. Same statelessness, different constraint.
• KServe already taught you the hard half. Readiness probes, replica counts, resource requests — that is steps 2 through 5 of the control loop. S2–S4 mostly teach you what each cloud calls those knobs.

Optional lab — model the crossover before you touch a console

AWS sells six different ways to run the same container, and the difference between the cheapest and most expensive is roughly 5×.

AWS built its compute portfolio by accretion over twenty years, so the names encode history rather than logic. ECS and EKS are orchestrators; Fargate is not an orchestrator at all, it is a compute engine that runs underneath either one. Getting that one relationship wrong is the most expensive mistake teams make on AWS containers. This session gives you the map and, more importantly, the purchasing decision that sits on top of it.

1 · EC2 — reading the name is half the skill

2 · Purchasing options — where the money actually is

Same instance, same performance, up to a 90% price difference. This is the highest-leverage decision on the page.

3 · Auto Scaling Groups, end to end

Target tracking, the only policy most services need — this fragment is the mechanism:

TargetTrackingConfiguration:
  PredefinedMetricSpecification: { PredefinedMetricType: ASGAverageCPUUtilization }
  TargetValue: 60          # AWS creates and manages the CloudWatch alarms for you
EstimatedInstanceWarmup: 180   # seconds before a new instance counts toward the average

4 · Containers — the relationship that trips everyone up

Node scaling inside EKS is its own layer: the Horizontal Pod Autoscaler scales pods, the Vertical Pod Autoscaler resizes them, and underneath, Karpenter (now generally preferred over the older Cluster Autoscaler) provisions right-sized nodes in seconds and consolidates underutilised ones. KEDA adds event-driven scaling on queue depth. If you know KServe, you already know this layer — what is new is only that the nodes are EC2 instances in an ASG.

5 · Lambda — and its hard edges

6 · Decision tree — which AWS compute for this workload?

7 · Three-cloud comparison — AWS-anchored

Worked example — one L4 GPU, three ways to pay for it

Reference: g6.xlarge — 1× NVIDIA L4 24 GB, 4 vCPU, 16 GiB — at $0.8048/hour on-demand in us-east-1 (July 2026; regions and rates vary, verify before committing).

Why spot works for training and not for serving — the same fact, two conclusions

Both workloads are stateless in the Tab-1 sense. The difference is what they promise. A training job promises a result, eventually — so a two-minute eviction notice costs you the time since your last checkpoint and nothing else. An inference endpoint promises a latency percentile, continuously — and a correlated capacity reclaim can take several spot replicas at once, so "run two spot instances in two AZs" is weaker protection than it looks.

The production pattern that falls out: a committed or on-demand warm floor sized to your p50 traffic, with spot capacity layered on top for the peak. That is exactly the min/max pair from Tab 1, with a purchasing decision attached to each end.

Three current sources

Why your vLLM serving is an EC2 instance and never a Lambda

Four independent disqualifications, any one of which is fatal:

1. No GPU. Lambda has no GPU resource type, no CUDA drivers, and no billing dimension for one. This is not a limit you can raise.
2. Memory ceiling. 10,240 MB is host memory, and it is less than the weights of most models you'd bother serving — before you consider that there is no VRAM at all.
3. The 15-minute wall. Fine for a request; useless for a persistent server holding a KV cache and batching across requests. Continuous batching — the entire reason vLLM is fast — requires a long-lived process that sees many requests at once.
4. Cold-start economics. Weight loading would land in the init phase and re-run on every cold start. Python ML functions already hit 3,000–4,500 ms of cold start without a model; add weights and the number becomes unbearable.

The AWS-native shape for your workload is therefore: a GPU instance in an Auto Scaling Group behind an application load balancer, with min = 1 (your warm floor — the same choice you already made on Vertex), a target-tracking policy on a custom metric (queue depth or tokens-in-flight, not CPU — CPU is meaningless on a GPU server), a generous instance warmup covering model load time, and a warm pool so scale-out doesn't pay the weight-loading cost twice. If you want Kubernetes semantics, the same thing on EKS with GPU node groups and KServe — which is closest to what you already know.

The Modal comparison is instructive. Modal gave you scale-to-zero GPU serving, which AWS does not sell: Fargate has no GPU, Lambda has no GPU. On AWS the closest equivalents are SageMaker asynchronous or serverless inference endpoints, or building it yourself with an ASG that scales to zero and eating the cold start. This is a genuine capability gap, and it is why Cloud Run GPU (S4) and Container Apps serverless GPU (S3) will feel more familiar to you than anything in this tab.

Optional lab — build the mechanism on a CPU instance

Azure will happily host your web app in four different services, and three of them are defensible.

Azure's compute story is cleaner than AWS's at the VM layer and messier at the PaaS layer. There is exactly one VM service and one scale-set service — no ECS/EKS-style fork. But above that sit App Service, Functions, Container Apps, and Container Instances, all of which will run an HTTP workload, and choosing between them is the single most common Azure architecture question. This session teaches the VM machinery properly (the Scale Set autoscaling model is the best-documented of the three clouds) and then resolves the PaaS choice with a decision tree.

1 · Azure VMs and the SKU-name decoder

2 · Virtual Machine Scale Sets — the autoscaling model, properly

Orchestration mode — pick this correctly at creation, it is not changeable later

Why Flexible matters for GPU work: mixed VM types in one scale set means you can express "one on-demand instance as my warm floor, up to five spot instances on top" as a single resource — the exact serving pattern from S2, but native rather than assembled.

Scaling policies and the timing knobs

3 · Flapping — derive the rule instead of memorising it

scale_in_threshold  <  scale_out_threshold × (N − 1) / N

N = 4,  scale_out = 80%   →   scale_in must be < 60%     ← Azure's own recommendation, explained
N = 2,  scale_out = 80%   →   scale_in must be < 40%     ← small groups need enormous gaps
N = 10, scale_out = 80%   →   scale_in must be < 72%     ← large groups can be tuned tightly

4 · AKS, and the PaaS layer above it

Functions hosting plans — this is the actual decision

5 · Decision tree — hosting an application on Azure

6 · Three-cloud comparison — Azure-anchored

Worked example — sizing the lag budget for a 4× spike

Setup. A VMSS behind a load balancer. Scale-out at 80% CPU, scale-in at 60% (which the formula above tells us is correct for N ≥ 4). Cooldown 10 minutes. VM boot plus application initialisation: 4 minutes. Baseline 4 VMs at 50% CPU. At 09:00 traffic goes 4×.

The three real options, and there is no fourth: (1) raise the floor so the baseline absorbs the spike — reliable, and you pay for four idle VMs every night; (2) accept nine minutes of degradation and communicate it as an SLO; (3) use predictive autoscale so the scale-out starts at 08:52 instead of 09:05 — free, but only works if the spike is genuinely cyclical and you have seven days of history proving it. Note that option 3 is the only one that improves both cost and latency, which is why it is worth the setup cost for any traffic pattern with a daily rhythm.

Three current sources

Mapping your existing stack onto Azure

• Your Modal Gemma deployment → Azure Container Apps with serverless GPU. This is the closest thing Azure sells to what you already run: a container, an A100 or T4, per-second billing, scale to zero. The differences you'd feel immediately are the quota-by-support-case friction and the narrow regional footprint at GA. The architectural shape is identical — which means your Modal container image is close to portable, and the interesting work is in the scaling rules and the cold-start budget, not the model code.
• Your Vertex AI script (T4, autoscale 1–3, spot) → a Flexible-mode VMSS with mixed VM types. Flexible mode is genuinely better than the AWS equivalent for this: one scale set expressing "one on-demand NCasT4_v3 as the warm floor, up to two spot instances above it." On AWS you assemble that from a mixed-instances policy; on Azure it is the mode's native purpose. But note the eviction notice is 30 seconds, not 120. If your Vertex spot strategy assumed a two-minute drain window, it does not transfer.
• Your GKE/KServe knowledge → AKS. Almost everything transfers unchanged. What is new is the wrapper: node pools with GPU SKUs, the cluster autoscaler configuration, and the fact that you must request GPU quota before the first pod will ever schedule.
• The identity link back to C2: where AWS gives an instance an IAM instance profile and GCP gives it a service account, Azure gives it a managed identity. Same idea, same purpose, different noun. Don't re-learn it — just add the third word to the Rosetta Stone.

Optional lab — a scale set that flaps, then doesn't

This is the cloud you already run GPUs on — and the one whose pricing rules will surprise you most.

Google's compute ladder is the tidiest of the three: Compute Engine → Managed Instance Group → GKE → Cloud Run → Cloud Run functions, with no ECS/EKS-style fork and no four-way PaaS ambiguity. The complexity has been moved somewhere else — into the discount system. GCP is the only cloud with an automatic no-commitment discount, and it is also the cloud where that discount pointedly excludes the machine families you care about. Your Vertex AI T4 script sits exactly on that fault line.

1 · Compute Engine — machine families and the naming grammar

Three nested terms, and the material defines them precisely: a machine family is a curated set of hardware configurations for a workload class; a machine series is a generation within that family (N2 is newer than N1; M3 newer than M2); a machine type is a specific vCPU/memory point within a series — or a custom machine type, which some series support and which has no clean AWS or Azure equivalent.

g2 - standard - 4          a3 - highgpu - 8g
│    │          │           │    │          │
│    │          └ 4 vCPU    │    │          └ 8 GPUs
│    └ configuration        │    └ high GPU density
└ series (G2 = L4)          └ series (A3 = H100)

n1-standard-8 + 1× nvidia-tesla-t4   ← N1 is the exception: GPUs attach separately,
                                        so you tune the CPU-to-GPU ratio yourself

Sources: Compute Engine GPU machine types Accelerator-optimized machine family VM instance pricing. Rates move; re-check before budgeting.

The discount system — this is where GCP differs most

2 · Managed Instance Groups — and the two-health-check rule

3 · GKE — the wrapper, not Kubernetes

You know Kubernetes. Here is only what GKE adds on top.

The 2026 update that makes this less of a fork: you can now run Autopilot-mode workloads inside a Standard cluster using Autopilot ComputeClasses. GKE provisions and manages those nodes while the rest of the cluster stays under your control with your own node pools. You can even set an Autopilot ComputeClass as the cluster or namespace default. Requirements are real — Rapid release channel, GKE 1.33.1-gke.1107000 or later — but the architectural point stands: Autopilot vs Standard is now a per-workload decision, not a per-cluster one.

The three autoscalers, which you already know from KServe but under different names: HPA adds pods on CPU, memory, or custom metrics; VPA right-sizes each pod's requests; Cluster Autoscaler adds and removes nodes under them. Node auto-provisioning goes one step further and creates new node pools of shapes you never defined. For your work the important interaction is that HPA scales pods in seconds while the Cluster Autoscaler must boot a GPU node in minutes — so a GPU deployment's real scale-out latency is the node-provisioning time, not the pod-scheduling time.

4 · Cloud Run — serverless containers, and the GPU story

5 · Decision tree — which GCP compute?

6 · Master three-cloud comparison

Worked example — Cloud Run GPU vs a dedicated L4 VM

This is the exact decision behind your Modal-vs-Vertex instinct, with numbers on it. Workload: an L4-sized inference endpoint — a 7B-to-9B model at FP16, comfortably inside 24 GB of VRAM.

Break-even is a duty cycle, not a request count:

break_even_hours = VM_monthly ÷ serverless_hourly
                 = $511 ÷ $1.05
                 ≈ 487 active hours per month
                 = 487 / 730
                 ≈ 67% duty cycle

Read it as a rule you can state out loud: below roughly 1.3 sustained requests per second, serverless GPU is cheaper; above it, the dedicated VM wins. And crucially, the crossover is high — 67% is a busy endpoint. Most internal tools, demos, batch scoring jobs, and pre-launch products sit far below it, which is why scale-to-zero GPU serving is genuinely the right default for that whole class of work.

Three current sources

Your Vertex AI script, re-read as architecture

You wrote: T4 GPUs, autoscaling 1 to 3, spot instances. Four deliberate decisions, and the vocabulary from this stage now names all of them.

The refinement worth making: keep min = 1 on standard (non-spot) capacity so the floor is real, and allow instances 2 and 3 to be spot. That is the pattern from every session in this stage — a committed or on-demand floor, interruptible headroom — and on GCP the MIG will auto-heal a preempted spot VM for you, since preemption counts as a repairable failure.

Your other two workloads

• Modal's scale-to-zero GPU serving → Cloud Run GPU, almost exactly. Container in, per-second billing, scale to zero, ~5-second instance start with drivers ready, no quota request for your first three L4s. The one real difference is concurrency: Cloud Run gives you an explicit dial (default 80, max 1,000), and for LLM serving you want it low — a value like 4 or 8 matched to what your vLLM instance can batch without blowing the latency budget. Getting that number right is worth more than any instance-type decision.
• Your GKE footprint. Two things to go and check today. First, whether your GPU workloads are in Standard node pools that could move to an Autopilot ComputeClass — you would stop paying for empty node capacity. Second, whether your cluster is covered by a Compute Flexible CUD: since 2024 one commitment spans Compute Engine, GKE (including most Autopilot pods), and Cloud Run, and it is not region-specific. If you are running steady CPU-side workloads across all three, a single Flex CUD at 28% or 46% is the highest-leverage thing on this page — on the CPU side. It will not touch your GPU spend.

Optional lab — a MIG with two health checks