STAGE 7 · the lens for everything below
Everything fails. Design for it, see it, recover from it.
Reliability is not a feature you add at the end. It is three habits practised together: redundancy so a failure has somewhere to fall, detection so you learn about it before your users do, and recovery so you get back inside a time budget you agreed to in advance.
The villain of this stage is the single point of failure — the one component whose death takes everything with it. Its quieter cousin is the untested backup: a disaster-recovery plan nobody has ever rehearsed is a wish, not a plan. Both are cheap to create and expensive to discover.
You already do this for LLM serving. You watch p95 latency, you trace requests through Langfuse, you alarm when tokens/sec collapses. This stage takes that instinct and applies it to the whole infrastructure — and adds the part that inference dashboards never covered: what you do when the region itself goes away.
OUTCOMES · what you'll be able to do by the end
Six things, all of them testable on paper
01 · Arithmetic
Turn an availability target into allowed downtime per month, and say what redundancy that number demands.
02 · Design
Given an RTO and RPO plus a budget, pick the right DR tier and defend the choice.
03 · Topology
Build multi-AZ and multi-region resilience, and explain active-active versus active-passive.
04 · Measurement
Define SLIs and SLOs with error budgets the way an SRE would, including burn-rate alerts.
05 · Fluency
Stand up metrics, logs and traces on any of the three clouds, and map your Prometheus/Grafana/Langfuse stack onto each.
06 · Judgement
Reason about what DR and SLOs mean specifically for a stateless GPU inference service.
FRAMING 1 · the loop that never stops turning
Redundancy → detection → recovery
Each of the three clouds sells you building blocks. They are not interchangeable brands of the same thing — each one sits at a specific point on this loop. Knowing where a service sits is more useful than memorising its name.
FRAMING 2 · the contract, and the ladder it implies
RTO and RPO are the contract
Recovery Time Objective (RTO) is how fast you must be back. Recovery Point Objective (RPO) is how much data you can afford to lose. Both are business decisions expressed in units of time, and together they select your DR tier. Cost climbs every rung — so you buy exactly the RTO/RPO the business will pay for, and not one rung more.
FRAMING 3 · you cannot promise what you cannot measure
Observability is how you operate
Metrics, logs and traces are raw material. On their own they are a bill from your monitoring vendor. They become operations when they roll up into an SLI (a number that reflects user happiness), an SLO (a target for that number), and an error budget (the failure the SLO explicitly permits). That last one is the load-bearing idea of Site Reliability Engineering: a budget you are allowed to spend turns reliability from an argument into arithmetic.
This is the shortest bridge from what you already know. Your Prometheus histogram of TTFT is an SLI generator. Your Grafana panel is the dashboard. Langfuse is the tracing signal. What C7 adds is the layer underneath — the infrastructure telemetry — and the discipline on top: a target, a budget, and a rule about what happens when the budget runs out.
MAP · four sessions, each building on the last
Where to go next
The sessions are cumulative. S1 defines every term and does the arithmetic; S2–S4 assume it and spend their time on the dialect differences. If a service name in S3 confuses you, the concept behind it was named in S1.
TRIAGE · you've been handed a workload — where do you start?
The first question is never "which cloud"
Reliability work has a natural order, and skipping down the list is the most common way teams waste money. A team with beautiful multi-region architecture and no tested restore is worse off than a team with one region and a rehearsed backup.
GUARD CLAUSES · follow "no ↓" until a "yes" exits right
Fix these in order. Each one is cheaper than the one below it.
PRIOR STAGES · what this leans on, and will not re-teach
Six threads you already have
C1 · Foundations
Regions and availability zones are the raw material of redundancy. Everything in this stage is a decision about how many of them to use.
C2 · Identity
Least privilege limits blast radius. Azure's February 2026 outage began as a storage permission change — a security control caused an availability incident.
C3 · Compute
Autoscaling, health checks and warm-minimum replicas are already reliability mechanisms. Here they get names: detection and recovery.
C4 · Networking
Load-balancer health checks and DNS-based traffic management are the network side of failover. S2's Route 53 section is C4 with a stopwatch attached.
C5 · Storage & Data
Leaned on heavily. RPO is a storage property. Durability, versioning and cross-region replication are the data half of every DR tier above.
C6 · App Architecture
Async decoupling and graceful degradation are resilience patterns. Remember the distributed-monolith lesson: chained synchronous calls multiply their failure rates.
HOW TO READ THIS
Three conventions
Three passes
Every major concept appears three times: intuition in plain words, mechanism with a diagram, then trade-offs with numbers and failure modes. Read all three; the third one is where the exam questions and the outages live.
Concepts before consoles
Click-paths, CLI and IaC appear only in each session's final Apply it section. The one exception is when the config file is the mechanism — an alert rule, an SLO definition, a health check.
Source vs. reality
Amber source is stale boxes flag where your cert guides have aged out. Monitoring product names drift fast; every correction here is dated and linked to the vendor's own documentation.
Sources used throughout: AWS, Microsoft and Google product documentation and status histories; the AWS Well-Architected Reliability Pillar; Google's Site Reliability Engineering and The SRE Workbook; and the published post-incident reports for the October 2025, February 2026 and May 2026 hyperscaler outages. Verified July 2026 — prices and service names change, so treat every figure as a worked method rather than a current quote.
S1 · SECTION 1 · why this session exists10%
One box, one wire, one plan nobody rehearsed
Almost every large outage in the last two years reduces to the same two sentences: something that existed only once stopped working, and the thing that was supposed to take over had never been asked to.
On 19 October 2025 a race condition in DynamoDB's internal DNS automation left an endpoint record empty in us-east-1. DynamoDB itself was fine — the phone source entry pointing at it was gone. Because dozens of other AWS services, and thousands of customer applications, resolved that one name, a single automated cleanup job produced roughly fifteen hours of cascading failure. That is a single point of failure. Not a server: a name.
Seven months later, on 7 May 2026, cooling failed in one data hall in the same region. Servers shut themselves down to avoid damage. No software rollback could help — someone had to make the room cold again. Coinbase's matching engine lost quorum because three of its five nodes lived in that one availability zone. That is a single point of failure too, hidden inside something that looked redundant.
S1 · SECTION 2 · core concepts, vendor-neutral50%
The vocabulary, defined once
Everything in sessions 2–4 is one of these ideas wearing a vendor's badge. Read this table once now; you can come back to it.
- RTO recovery time objective
- Maximum acceptable time a service can be down before the business is materially harmed. A promise about speed.
- RPO recovery point objective
- Maximum acceptable amount of data loss, measured in time. An RPO of one hour means you may lose the last hour of writes. A promise about data.
- MTTR mean time to recovery
- What your recoveries actually average, measured after the fact. RTO is the target; MTTR is the score.
- Availability nines
- Shorthand for uptime percentage. "Four nines" is 99.99%. Each extra nine divides your allowed downtime by ten.
- SLI service level indicator
- A measured number that tracks user happiness — e.g. the proportion of requests served successfully in under 300 ms.
- SLO service level objective
- Your internal target for an SLI over a window — e.g. 99.9% of those requests, over 28 days. Chosen by you.
- SLA service level agreement
- A contractual promise to a customer, with financial penalties. Always set looser than your SLO so you notice trouble before your lawyer does.
- Error budget
- The failure your SLO permits: 100% − SLO. At 99.9% over 30 days that's 43.2 minutes, or 0.1% of requests. Spend it on releases; when it's gone, stop shipping and stabilise.
- Blast radius
- How much breaks when this one thing breaks. Reducing it is the core reliability move — cells, zones, accounts, subscriptions, projects.
- Availability zone (AZ)
- An isolated datacentre (or group) inside a region with independent power, cooling and networking. Milliseconds apart, so synchronous replication is viable.
- Region
- A geographic cluster of AZs. Tens to hundreds of milliseconds apart, so cross-region replication is usually asynchronous — which is exactly why multi-region implies a non-zero RPO.
- Multi-AZ vs multi-region
- Multi-AZ survives a datacentre; multi-region survives a region, a bad config push, and a regulator. Multi-AZ is nearly free and usually automatic. Multi-region is a project.
- Active-passive
- One site serves traffic; the other waits. Simple, cheaper, and the standby is only as good as your last test.
- Active-active
- All sites serve traffic simultaneously. Failover is trivial; keeping state consistent is the hard part you've now signed up for.
- Health check
- A periodic probe that decides whether a target should receive traffic. Defined by path, interval, timeout, and how many consecutive results flip the verdict.
- Failover
- Moving traffic from an unhealthy target to a healthy one. Can be automatic (load balancer, DNS) or deliberate (a human runs a plan).
- Backup
- An independent point-in-time copy you can restore from. Protects against deletion, corruption and ransomware — because you can go back.
- Snapshot
- A point-in-time image of a volume or database, usually incremental. A backup mechanism, generally tied to one provider and often one region unless you copy it.
- Replication
- Continuous copying of changes to another location. Gives a tiny RPO — and faithfully reproduces your mistakes within seconds. Replication is not backup.
- Graceful degradation
- Shedding non-essential functionality to keep the core working. Serving a cached recommendation instead of erroring the whole page.
Availability nines — what a number actually promises
PASS 1INTUITION
An availability figure is a downtime allowance written as a percentage. It's the same trick as a punctuality target for a train line: "99.9% on time" sounds like a quality statement, but it's really permission to be late a specific number of times.
The useful move is always to convert the percentage back into minutes, because minutes are something a human can argue about. "99.9%" ends discussions. "43 minutes a month" starts them — and 43 minutes is roughly one bad deploy plus the rollback.
PASS 2MECHANISM
Availability = uptime ÷ total time. Downtime allowed = (1 − availability) × period. That's the whole formula. What matters is how it composes across a system:
- Serial (dependency chain): multiply the availabilities. Three components at 99.9% in a chain give 99.7% — you got worse by adding parts.
- Parallel (redundant): multiply the unavailabilities. Two independent 99.9% components give 1 − (0.001 × 0.001) = 99.9999%.
Parallel only works if the components are genuinely independent. Two web servers in the same AZ share a power feed; two regions share a global DNS control plane. Shared dependencies quietly turn parallel back into serial — which is precisely what took Coinbase down in May 2026 with three of five quorum nodes in use1-az4.
PASS 3TRADE-OFFS & LIMITS
Three things the arithmetic hides:
- Nines are not linear in cost. Going 99.9% → 99.99% typically means a second AZ and automated failover. 99.99% → 99.999% means multi-region active-active, and the cost multiple is not 10× — it's the cost of a distributed-systems team.
- Your SLA is not your availability. A cloud SLA of 99.99% for a single VM only pays out a service credit; it does not make the VM available. AWS's standard EC2 credit for the May 2026 event was a fraction of monthly compute spend against many hours of lost trading.
- Measurement window changes everything. 99.9% measured yearly permits one 8-hour outage. Measured monthly it permits 43 minutes and that same outage blows it. Always state the window.
- Availability is the wrong SLI for inference. An endpoint returning 200s at 40 seconds per token is "available" and useless. This is the single most important adaptation for your workload, and S2 returns to it.
Redundancy: multi-AZ, multi-region, and who is awake
PASS 1INTUITION
Multi-AZ is like keeping a spare tyre in the boot of the same car. Multi-region is keeping a second car at your sister's house in another city. The spare tyre handles the overwhelmingly common failure and costs almost nothing. The second car handles the rare catastrophe — a flood, a fire, or someone pushing a bad config to every machine in the city — and costs like a second car.
Active-passive versus active-active is the difference between the second car sitting in a garage with a full tank, and you actually driving both cars on alternate days. The second one is more expensive and you will notice immediately if one won't start.
PASS 2MECHANISM
The distinction that drives all the engineering is latency between the copies, because latency decides whether replication can be synchronous.
- Within a region (AZ to AZ) round trips are typically about 1–2 ms. A database can wait for the second copy to acknowledge each write before confirming it. Synchronous replication ⇒ RPO = 0. This is why "Multi-AZ" database options are a checkbox and multi-region ones are an architecture.
- Between regions round trips are tens to hundreds of milliseconds. Waiting for the far copy would add that to every write. So cross-region replication is almost always asynchronous ⇒ RPO > 0, equal to your replication lag at the moment of failure.
Traffic steering follows the same split. Inside a region a load balancer moves traffic in seconds because it holds live connections. Across regions you usually steer with DNS, and DNS is cached — your effective failover time includes the record's TTL plus whatever resolvers and clients ignore it. A 60-second TTL is not a 60-second failover.
PASS 3TRADE-OFFS & LIMITS
- Multi-region does not protect you from yourself. Google's June 2025 outage was a policy record with blank fields that replicated globally within seconds and crashed the same binary in every region. Regional isolation only helps if the failure is regional; a bad global config push is a serial dependency wearing a parallel costume. The defences are feature flags, staged rollout, and a kill switch — not more regions.
- Control planes have geography. The API you use to create resources often lives somewhere specific. Some AWS global services have their control plane in
us-east-1; ARC's lives inus-west-2. Recovery procedures that call a control plane in the region that is on fire do not work. Prefer data-plane operations during failover. - Quorum systems need odd splits across three AZs. Anything using consensus — etcd, ZooKeeper, Kafka controllers, a 5-node matching engine — must be spread so that losing one AZ never costs you a majority. Three nodes across two AZs is a trap: lose the AZ with two, lose quorum.
- Active-active with a shared database is a lie. If both regions write to one primary, you have an active-active compute tier in front of a single-region database. Your true RTO is your database's.
Health checks and failover, step by step
This sequence is identical on all three clouds. Only the product names change.
Steady state — the probe is already running
A health checker polls a specific endpoint on each target at a fixed interval. The endpoint matters enormously: /healthz returning a hard-coded 200 OK proves only that the web server process is alive. A deep health check verifies the dependencies the request actually needs — for an inference server, that the model weights are loaded and the GPU is visible, not merely that the HTTP port answers.
Shallow checks cause the worst class of outage: a fleet reporting perfect health while every request fails.
Fault occurs — and nobody knows yet
The GPU throws an uncorrectable ECC error; the disk fills; the process OOMs; the AZ loses cooling. Between this moment and the next probe, requests are being routed into a black hole. This gap is pure, invisible downtime, and its size is your probe interval.
The first term in your RTO is detection latency, and you control it directly by choosing the interval.
Probe fails — once
One failure is not proof. Networks drop packets; a garbage collection pause can time out a probe on a perfectly healthy host. If you act on a single failure you will flap: healthy targets get pulled, load concentrates on the survivors, they slow down, their probes time out, and you have built a machine that removes your entire fleet.
So health checkers require N consecutive failures. This is a deliberate, tunable delay.
Threshold trips — the verdict flips
After the configured consecutive failures the target is marked unhealthy. Do the arithmetic on your own settings, because this is where RTO is actually decided:
Tightening costs probe traffic and raises flap risk. Loosening costs downtime. There is no correct answer, only a documented one.
Traffic shifts — and the mechanism decides the speed
Where the shift happens sets how fast it is:
- Load balancer, in-region: the target is dropped from the pool. Effectively immediate for new connections.
- DNS, cross-region: the record changes, but clients and resolvers hold the old answer until the TTL expires — and some ignore TTLs entirely. Add TTL to your RTO, then add a margin for badly behaved clients.
- Anycast / global load balancer: withdrawn at the edge, no DNS caching in the path. Fastest, and the reason global front doors exist.
If the target is a database, "shifting traffic" also means promoting a replica — and that is a one-way door. Promote too eagerly during a network partition and you get split brain: two primaries accepting conflicting writes.
Recovery and fail back — the half everyone forgets
Failing over is the rehearsed part. Coming home is not. Fail-back requires reversing replication (the standby now holds the newest data), re-synchronising, and choosing a moment to switch — usually a second planned outage.
Then the loop closes: a blameless postmortem asks not "who broke it" but "what made this failure possible, and what made it take so long to see?" Every good postmortem output is a change to redundancy, detection, or recovery.
RTO and RPO — the two arrows pointing away from the incident
PASS 1INTUITION
Picture the moment of failure as a line on a calendar. RPO points backwards: how far back in time is the last good copy of my data? Everything between that copy and the line is gone. RTO points forwards: how long until customers can use the service again?
They're independent. You can have an RPO of five seconds and an RTO of six hours (your data is safe, but rebuilding takes all afternoon). You can have an RPO of a day and an RTO of two minutes (you come back fast, missing yesterday). Businesses usually care about one far more than the other, and asking which is the fastest way to a right-sized design.
PASS 2MECHANISM
RPO is bought with replication frequency; RTO is bought with pre-provisioned capacity. Two different budgets, two different vendors' invoices.
- Nightly backups ⇒ RPO up to 24 h. Hourly snapshots ⇒ RPO up to 1 h. Continuous log shipping ⇒ RPO of seconds. Synchronous replication ⇒ RPO of zero, at the cost of write latency.
- Nothing running ⇒ RTO of hours (provision, restore, verify). Data warm, compute cold ⇒ tens of minutes. Everything warm and small ⇒ minutes. Everything hot ⇒ seconds.
RTO is a sum, and people habitually budget only the last term: detect + decide + execute + verify + cut traffic over. The "decide" term is often the largest and is entirely human. That's why mature shops pre-authorise failover against a written trigger, so nobody is convening a call to ask permission.
PASS 3TRADE-OFFS & LIMITS
- Stated RTO is fiction until measured. Restore time scales with data volume and is dominated by throughput you have probably never tested. Restoring 10 TB at 250 MB/s is eleven hours before you even validate it. A game day converts your RTO from a slide into a measurement.
- RPO ≠ 0 doesn't mean "lose 5 seconds of everything". It means lose whatever was in flight — often the most valuable records, since they're the newest. For payments the answer is usually not smaller RPO but idempotent replay from an append-only log.
- Different data deserves different targets. One RPO for the whole estate over-buys for logs and under-buys for the ledger. Tier your data.
- GPU / GenAI A stateless inference service barely has an RPO. Model weights are immutable artefacts in object storage; there is no "last hour of writes" to lose. Your RPO question collapses into "is the weights bucket replicated to the second region?" — usually yes, cheaply. Your RTO, meanwhile, is dominated by something no database has: GPU capacity availability in the failover region, plus the time to pull tens of gigabytes of weights and warm the KV cache. Prompt logs and eval data are your only genuinely stateful assets, and they tolerate a long RPO.
Backup vs snapshot vs replication — three words people use as one
Backup
An independent, restorable point-in-time copy, ideally in a different account or vault with its own permissions and immutability.
Protects against: deletion, corruption, ransomware, bad migrations, "oops".
Does not protect against: a long RPO — it's only as fresh as the last run.
Snapshot
A block- or database-level image, usually incremental (only changed blocks stored). A mechanism often used to implement backups.
Watch for: snapshots frequently live in the same region and same account as the source. Copy them elsewhere or they share the source's fate.
Replication
Continuous propagation of changes to another location, synchronous or asynchronous.
Protects against: infrastructure loss, with a tiny RPO.
Does not protect against: anything you did on purpose. A DROP TABLE replicates in milliseconds. Replication is not backup.
Real designs use all three: replication for RPO, snapshots for fast local rollback, and backups in a separate security boundary for the day something malicious or catastrophic gets into the primary account. The 2025–26 wave of ransomware incidents is why every provider now sells immutable, delete-protected backup vaults as a distinct product from ordinary snapshots.
SLI → SLO → error budget: the SRE spine
PASS 1INTUITION
You already know this shape from LLM serving: pick a number that tracks whether users are happy (p95 time-to-first-token), decide what value is acceptable, and watch it. SRE just adds the third move — name the failure you're allowing, and spend it deliberately.
The error budget is a monthly allowance of badness. Under budget, ship features fast; you've earned the risk. Over budget, feature work stops and reliability work starts. It converts the eternal dev-versus-ops argument into a shared number that nobody gets to opinion their way out of.
PASS 2MECHANISM
An SLI is nearly always a ratio of good events to valid events. Get that shape right and everything else follows:
Note the second one is not "p95 latency < 2s". A percentile is a statistic about a window; an SLI is a count of good events. Counting is what lets you subtract from a budget. Keep percentiles on your dashboard for diagnosis, and use the bucket-ratio form for the SLO.
Then the budget: with a 99.9% SLO over 28 days and 50 million requests, you are permitted 0.1% × 50M = 50,000 failed requests. Burn rate is how fast you're consuming it, normalised so 1.0 means "exactly on pace to exhaust the budget precisely at the end of the window".
PASS 3TRADE-OFFS & LIMITS
- Too many nines is a bug. If you're comfortably beating your SLO every month you are over-investing in stability and under-shipping. Google's guidance is blunt: an unspent error budget is wasted money. Some teams deliberately inject controlled failure to keep the number honest.
- Choose the window deliberately. Rolling 28-day windows give continuous feedback; calendar months align with business reporting but reset your sins on the first. Rolling is better for engineering, calendar is better for executives — many teams publish both.
- Alert on burn rate, not on threshold. "Error rate > 1%" pages you at 3 a.m. for a blip that costs 0.2% of the budget. "Burning budget 14× faster than sustainable, for an hour" pages you only when the month is genuinely at risk. S4 does the arithmetic — it's Google's home turf.
- An SLO with no consequence is a dashboard. The policy — what actually stops when the budget is exhausted — is the part that makes it work, and it's a management artefact, not a technical one.
- GPU / GenAI Inference needs at least two SLOs. An availability SLO (did we answer?) and a latency SLO on p95 TTFT and inter-token latency (did we answer usefully?). Quality is a third axis that no infrastructure SLO captures — a model serving fluent nonsense is 100% available. Keep your Langfuse eval scores as a separate quality indicator and resist the urge to fold them into the reliability SLO.
Metrics, logs and traces — three signals, three jobs
PASS 1INTUITION
A metric is a number over time: cheap, aggregated, perfect for "is it bad right now?" A log is a timestamped event with detail: expensive, searchable, perfect for "what exactly happened to this one request?" A trace is one request's journey stitched across services: perfect for "which of my eleven hops ate the 900 milliseconds?"
The rule of thumb: metrics tell you that, traces tell you where, logs tell you why. You need all three, and you need them in that order during an incident.
PASS 2MECHANISM
They differ in cardinality economics, which drives every design decision:
- Metrics are pre-aggregated time series identified by a name plus labels. Cost scales with the number of unique label combinations, not with request volume. Putting
user_idin a label is the classic way to detonate your bill — one series per user, forever. - Logs cost per gigabyte ingested and per gigabyte retained. Cost scales with volume and verbosity. A debug logger left on in production is a budget incident.
- Traces cost per span, so they're usually sampled. Head sampling decides at the start (cheap, may miss the interesting 0.1%); tail sampling decides after seeing the whole trace (keeps the slow and failed ones, needs a collector holding spans in memory).
The connective tissue is OpenTelemetry (OTel) — a vendor-neutral standard for producing and shipping all three. This is the single biggest change since the materials were written: all three clouds now accept OTLP natively, and all three are steering customers off their proprietary agents. Instrument with OTel and your telemetry becomes portable, which for a multi-cloud learner is the whole game.
PASS 3TRADE-OFFS & LIMITS
- Observability bills are a reliability risk in themselves. Teams routinely spend more on monitoring than on the workload. Industry reviews consistently find 20–40% of ingested log volume is never queried. Route verbose logs to cheap tiers, sample traces, and cap metric cardinality — this is a design activity, not a cleanup task.
- Your monitoring must not share fate with the thing it monitors. During Google's June 2025 incident, monitoring and communication tooling was itself impaired, which delayed customer updates. If your alerting pipeline runs in the region that just died, you will be the last to know.
- Logs are the compliance signal. Audit logs answer "who did what" and have retention requirements measured in years. Keep them architecturally separate from application debug logs so you can retain one cheaply and discard the other aggressively.
- GPU / GenAI Your workload adds a fourth axis the classic three don't cover: token economics. Tokens in/out, cost per request, cache hit rate and eval scores are metrics in shape but product data in meaning. Langfuse holds them today; on each cloud you'll decide whether they belong in the managed metrics store, the trace store, or a warehouse.
Decision tree · which DR tier does this workload need?
GUARD CLAUSES · follow "no ↓" until a "yes" exits right
Pick the cheapest tier that satisfies the contract
Decision tree · metric, log, or trace?
GUARD CLAUSES · follow "no ↓" until a "yes" exits right
Which signal answers the question I have right now?
The same ideas in three dialects
This is the Rosetta Stone for the whole stage. Sessions 2–4 expand each column; here they sit side by side so the pattern is visible before the details arrive.
| Concept | AWS | Azure | Google Cloud |
|---|---|---|---|
| Metrics store | Amazon CloudWatch MetricsNumeric time series from nearly every service, free for many defaults.Metrics do not aggregate across regions; you stream or query cross-account to unify. | Azure Monitor MetricsNear-real-time platform metrics, organised by namespace and dimension.Platform metrics and custom metrics have different retention and alerting behaviour. | Cloud MonitoringBuilt on Monarch, Google's own global time-series store.Scoping is by metrics scope, not per-project — one scope can span many projects. |
| Managed Prometheus | Amazon Managed Service for PrometheusPromQL-compatible, remote-write from your existing scrapers.Metrics only — no logs or traces. Priced per sample ingested, stored and queried. | Azure Monitor managed service for PrometheusStores into an Azure Monitor workspace; deep AKS integration.18-month retention with no storage charge; billed on ingestion and query. | Google Cloud Managed Service for PrometheusDrop-in for a self-run stack; managed collectors on GKE.24-month retention included; PromQL works against Cloud Monitoring metrics too. |
| Managed Grafana | Amazon Managed GrafanaGrafana 12.4 workspaces, in-place upgrade from 10.4.Per-active-user pricing, which surprises teams who share dashboards widely. | Azure Managed GrafanaPlus free in-portal "Azure Monitor dashboards with Grafana".Use the free in-portal version if you only need Azure data sources. | Self-hosted or Grafana CloudNo first-party managed Grafana; the Prometheus API is the integration point.Google's answer is that Cloud Monitoring dashboards speak PromQL natively. |
| Logs | CloudWatch LogsLog groups and streams; Logs Insights for queries; metric filters; subscriptions for real-time fan-out.Console log delivery lags minutes — use subscriptions when you need real time. | Azure Monitor Logs / Log AnalyticsA workspace queried with KQL; per-table plans.Table plan choice (Analytics / Basic / Auxiliary) changes cost by up to 45×. | Cloud LoggingEvery entry passes the Log Router; sinks fan out to buckets, BigQuery, Pub/Sub, GCS._Required bucket cannot be disabled or deleted; _Default retains 30 days. |
| Traces | AWS X-Ray → CloudWatch Application SignalsTransaction Search gives 100% span visibility.X-Ray SDKs and Daemon are in maintenance; OpenTelemetry is the supported path. | Application InsightsDistributed tracing, dependency maps, live metrics, all inside Azure Monitor.Adaptive sampling is on by default — good for cost, confusing during an incident. | Cloud TraceNative OTLP ingest via telemetry.googleapis.com.Trace sinks were deprecated in Feb 2026; use Observability Analytics instead. |
| Alerting | CloudWatch Alarms → SNS / EventBridgeComposite alarms combine conditions.Alarms are regional; a multi-region view needs deliberate aggregation. | Alert rules → Action groupsAn action group is a reusable bundle of notification and automation targets.Rule scope is explicit and easy to set too narrowly, silently missing resources. | Alerting policiesConditions on metrics, uptime checks, log matches, or SLO burn rate.Burn-rate conditions are first-class, not something you assemble yourself. |
| SLOs as objects | Application Signals SLOsRequest- and period-based SLOs with built-in error-budget tracking.Newer capability — SLO recommendations and performance reports arrived March 2026. | Assemble from KQL + alertsNo single first-class SLO object; teams build them from queries and workbooks.The weakest of the three for turnkey SLO practice. | Service Monitoring SLOsSLO and error budget are API resources with burn-rate alerting built in.Strongest native SLO story of the three — unsurprisingly. |
| Zone redundancy | Multi-AZ / ASG across AZsManaged services expose it as a deployment option.Multi-AZ RDS is for availability, not read scaling — read replicas are separate. | Availability zones + availability setsZones for datacentre isolation; sets only for rack isolation inside one datacentre.Availability sets are the legacy, weaker construct — zones are what you want. | Regional resourcesRegional MIGs and regional GKE clusters spread across zones by default."Regional" is the multi-zone word in GCP; "multi-region" means across regions. |
| DNS / traffic failover | Route 53 health checks + failover routingPlus Amazon ARC for orchestrated recovery.ARC's control plane lives in us-west-2, deliberately away from us-east-1. | Azure Traffic Manager / Front DoorDNS-based and anycast global routing respectively.Front Door is itself a global dependency — it caused the Oct 2025 outage. | Cloud DNS + Global External LBAnycast global load balancing with a single global IP.Global anycast avoids DNS TTL delays entirely — a genuine architectural edge. |
| Backup service | AWS BackupCentral policy, scheduling and monitoring across EBS, EC2, RDS, DynamoDB, EFS and more; cross-region copy.Vault Lock gives write-once immutability for ransomware defence. | Azure BackupRecovery Services and Backup vaults, soft delete, immutability, long-term retention.Managed through "Resiliency in Azure" — the renamed Business Continuity Center. | Backup and DR ServiceBackup vaults with immutable, indelible copies; integrated with Cloud Monitoring.Distinct from ordinary persistent-disk snapshots, which are not a backup product. |
| DR orchestration | Amazon ARC · Region switchDeclarative plans, practice mode, per-region data plane. Plus Elastic Disaster Recovery and Resilience Hub.Plans validate themselves every 30 minutes — untested-backup insurance. | Azure Site RecoveryReplication with recovery plans and non-disruptive test failover.Test failover into an isolated network is the feature to actually use. | Backup and DR + IaCNo single orchestrator equivalent; recovery is composed from Terraform, MIGs and DNS.Weakest turnkey DR orchestration; strongest primitives. |
| Chaos / DR testing | AWS Fault Injection ServiceControlled experiments including AZ power interruption.Pairs with ARC to prove a failover plan actually works before you need it. | Azure Chaos StudioFault injection as a managed experiment service.Zone-down drill now surfaced inside Resiliency in Azure. | No first-party chaos serviceTeams use open-source tooling on GKE.Google's cultural answer is DiRT exercises and game days, not a product. |
S1 · SECTION 3 · reality check25%
Do the arithmetic on paper first
WORKED EXAMPLE 1
"We need 99.95%" — what did you just promise?
The conclusion writes itself. 99.95% monthly is not achievable with humans in the loop. It requires automated in-region failover, which means multi-AZ, which means your database is Multi-AZ and your compute is spread across at least two zones. That's what the number means — one number on a slide has just specified an architecture.
And notice what 99.95% does not require: a second region. Region-level events are rarer than the monthly budget implies. Multi-region is bought for the annual catastrophe and for regulators, not to buy the third nine.
WORKED EXAMPLE 2
Blended availability of a two-region active-passive setup
The seductive version of this calculation gives you six nines. The honest version does not, and the gap is the whole lesson.
Two conclusions worth carrying around. First, the routing layer caps you: you cannot be more available than the serial component you route through, no matter how many parallel regions sit behind it. Adding a third region moves you from 99.9999% to 99.99999% on the parallel term, which changes the total by nothing at all.
Second, this is not theoretical. Azure's October 2025 outage was a configuration change in Azure Front Door — the global routing layer. Every service behind it was individually healthy and globally unreachable. The parallel term was perfect; the serial term was zero.
WORKED EXAMPLE 3
Sizing an error budget for an inference endpoint
That last line is the standard fast-burn alerting threshold, and S4 builds the full multi-window ladder on it. The practical point for you now: a single 90-minute incident at 2% error rate costs you the entire month's budget. Which means the release policy attached to the budget will trigger, which means the next sprint changes. That is the mechanism working correctly.
GPU / GenAI Note what happens if you'd written this SLO as pure availability instead. A vLLM replica thrashing on KV-cache evictions returns 200 OK at nine seconds per first token. Availability: 100%. Error budget: untouched. Users: gone. For inference the latency SLO is the real one, and uptime is the easy secondary.
Three things that actually happened
POSTMORTEM AWS, 19–20 October 2025 — a race condition deleted a DNS record, and 15 hours followed
Two instances of DynamoDB's internal DNS automation ran concurrently in us-east-1. A stale-plan check let an older plan overwrite a newer one; cleanup automation then deleted the record entirely, leaving the DynamoDB endpoint unresolvable. Existing connections survived; no new ones could be made.
DynamoDB's own DNS was restored in about three hours. The outage lasted roughly fifteen, because EC2's DropletWorkflow Manager depends on DynamoDB, entered congestive collapse on recovery, and then Network Load Balancer health checks began failing across the fleet. Downdetector logged millions of reports across a thousand-plus services.
What to take from it for this session: the SPOF was a name, not a machine — dependency graphs include DNS. Recovery was slowed by a thundering herd on restart, which is why backoff and jitter are reliability features, not politeness. And AWS's own remediation was to disable the automation globally and add test coverage for the recovery workflow: the untested path was the recovery path.
Source: AWS, Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region — aws.amazon.com/message/101925
POSTMORTEM Google Cloud, 12 June 2025 — an unflagged null pointer, replicated worldwide in seconds
On 29 May a new quota-policy check was deployed into Service Control, the component that authorises every Google Cloud API call. The code lacked error handling and was not behind a feature flag, so it shipped globally in an active state but stayed dormant, waiting for policy data of a shape that didn't exist yet.
On 12 June a policy update wrote blank fields into the regional Spanner tables Service Control reads. Spanner replicated that globally within seconds. Every regional Service Control binary hit the new code path, dereferenced a null, and crash-looped. External API requests across Google Cloud and Workspace returned 503s.
SRE triaged within two minutes and identified the cause within ten. A "red button" kill switch was rolled out globally in about forty minutes. Recovery in us-central1 took roughly two hours forty minutes longer, because every restarting instance hammered the same Spanner infrastructure simultaneously. Total: about seven and a half hours.
What to take from it: multi-region bought nothing here, because the failure propagated through the replication that makes multi-region work. Global config is a serial dependency. The defences are staged rollout, feature flags, and a pre-built kill switch — and note that Google had the kill switch, which is why this was hours rather than days.
Sources: Google Cloud Service Health incident report (12 June 2025); ThousandEyes outage analysis — thousandeyes.com
PRACTICE Google SRE — where the error-budget discipline is actually written down
The two canonical texts are free online. Site Reliability Engineering (2016) defines SLIs, SLOs and error budgets and argues the case for treating reliability as a budget. The SRE Workbook (2018) is the practical follow-up: §2 and 5 are the ones to read for implementing SLOs and for the multi-window, multi-burn-rate alerting ladder that S4 uses.
Two ideas from them worth internalising now, because they reframe what you're doing: toil should stay under 50% of an SRE's time — anything above that is a signal to automate, not to hire. And 100% is the wrong reliability target for everything: users cannot distinguish 100% from 99.99% because their own network is less reliable than that, so the last nine is spent on nobody.
Sources: sre.google/sre-source sre.google/workbook
S1 · SECTION 4 · apply it — your context + hands-on10%
You already built half of this
Your Prometheus/Grafana/Langfuse stack is not a warm-up for cloud observability — it is cloud observability, self-hosted. The mapping is almost one-to-one, and knowing that changes how you should read sessions 2–4: you're not learning monitoring, you're learning three managed backends for a stack you already run.
What you have → what it's called here
- Prometheus scrape of vLLM
/metrics→ the metrics signal. Every cloud has a managed Prometheus that accepts remote-write, so this survives the move intact. - Grafana dashboards → the visualisation layer. Two clouds offer managed Grafana; the third speaks PromQL natively.
- Langfuse traces → the traces signal. Instrument with OpenTelemetry and the same spans land in X-Ray/Application Signals, Application Insights, or Cloud Trace unchanged.
- p50/p95/p99 TTFT → your SLI raw material. Convert the histogram to a bucket ratio and it becomes an SLO.
- Cost per 1k tokens → a metric with no cloud-native equivalent. Keep owning this one.
What's genuinely new in this stage
- Infrastructure telemetry beneath the model — node health, disk, network, control-plane events. You've been watching the top of the stack.
- The error-budget policy — the organisational half. A number that stops releases.
- DR, which your local stack has no concept of. What happens when the region hosting your endpoint is gone?
- Audit logging — who changed the deployment, and when. A compliance signal, not a debugging one.
Your DR story is unusual, and unusually good
Write this down now, because it recurs in every remaining session. A stateless vLLM service has no meaningful RPO — the weights are immutable objects in a bucket, and buckets replicate cross-region cheaply. Compare that with a stateful transactional database, where every second of replication lag is money at risk. You are on easy mode for the data half of DR.
Your RTO, though, has a term nothing in the DR literature covers: GPU capacity availability in the failover region. Restoring a Postgres instance means asking for CPU that is always there. Recovering an inference service means asking for L4s or A100s in a region that may be sold out precisely because everyone else is failing over too. Your realistic recovery sequence is: provision GPU nodes (minutes, or never), pull 15–30 GB of weights from the replicated bucket (minutes), load into VRAM and warm up (a minute or two), pass a deep health check, take traffic.
Which makes the warm-minimum GPU replica the single most consequential reliability-versus-cost decision you will make. One always-on GPU replica in the second region is roughly the cost of a small warm standby and converts "hope there's capacity" into a guaranteed floor. S2 puts real numbers on it.
OPTIONAL HANDS-ON · S1 · runs entirely on your laptop, zero cloud spend
Turn your existing latency histogram into an SLO with a burn-rate alert
This is deliberately cloud-free — S1 is the concepts session, and the point is that the SLO discipline is portable. Everything here is open source and runs in Docker.
- Bring up Prometheus and Alertmanager with
docker compose, scraping either your vLLM container or theprometheus/node-exporterimage if you don't have a model running. - Define a recording rule that computes your SLI as a bucket ratio rather than a percentile — the good-events-over-valid-events shape from Pass 2 above.
- Add a multi-window burn-rate alert. This is the artefact that is the mechanism, so here it is in full:
# slo.rules.yml — 99.9% of requests with TTFT under 2s, 28-day window groups: - name: inference-slo rules: - record: slo:ttft_good_ratio:rate1h expr: sum(rate(ttft_seconds_bucket{le="2.0"}[1h])) / sum(rate(ttft_seconds_count[1h])) - alert: ErrorBudgetFastBurn # 14.4x burn over 1h = 2% of the 28-day budget consumed expr: (1 - slo:ttft_good_ratio:rate1h) > (14.4 * 0.001) for: 5m labels: {severity: page} - alert: ErrorBudgetSlowBurn # 6x burn over 6h = 5% consumed — ticket, don't page expr: (1 - slo:ttft_good_ratio:rate6h) > (6 * 0.001) for: 30m labels: {severity: ticket}
- Break it on purpose. Add artificial latency (or just lower the
lethreshold to something your service can't meet) and watch the fast-burn alert fire in Alertmanager. - Write the policy sentence — one line: "When the 28-day budget is exhausted, feature deploys pause until it recovers above 25%." Without this sentence the whole exercise is a dashboard.
TEARDOWN
docker compose down -v— the-vmatters; it removes the Prometheus TSDB volume, which grows quietly.docker image pruneif you won't reuse the images.- Nothing here bills. That changes from S2 onward — from the next session on, always set a budget alert before you create anything.
S2 takes the ladder to AWS. You'll see the four tiers as concrete services, watch Route 53 health checks and Amazon ARC do the failover you just diagrammed, and meet CloudWatch as the umbrella it really is.
The GenAI tie-in gets specific: what an inference-endpoint SLO looks like in Application Signals, and what a warm GPU replica actually costs per month.
S2 · SECTION 1 · why this session exists10%
The region everything depends on
AWS has thirty-odd regions. A startling amount of the internet runs in one of them. us-east-1 is the oldest, the largest, the default in every tutorial, and the home of several global control planes — which means it is simultaneously the most redundant region AWS operates and the most consequential single point of failure on the public internet.
Since October 2025 it has had four significant events. Two are worth studying because they fail in completely different ways and demand completely different defences.
20 Oct 2025 · a software SPOF
A DNS automation race condition emptied the DynamoDB endpoint record. Fifteen hours, cascading through EC2 and NLB. Multi-AZ was no defence — the failure was regional and logical. Only multi-region helped.
7–8 May 2026 · a physical SPOF
Chillers failed in one data hall; servers thermally shut down in use1-az4. Multi-AZ was a complete defence — and the companies that went down publicly were the ones whose "redundant" clusters had a majority of nodes in that single zone.
That contrast is the entire session in miniature. Multi-AZ is cheap, automatic, and handles the physical failure that happens most often. Multi-region is expensive, manual, and handles the logical failure that happens rarely and hurts more. You need to know which you're buying.
S2 · SECTION 2 · core concepts50%
Redundancy on AWS: zones, then regions
PASS 1INTUITION
On AWS, "multi-AZ" is usually a checkbox and "multi-region" is usually a quarter of engineering time. That asymmetry is deliberate: AWS built regions as hard isolation boundaries — separate control planes, separate failure domains, no automatic anything between them. They will not silently make your data cross a region boundary, which is excellent for compliance and inconvenient for DR.
So the default posture is: get multi-AZ right first, because it's nearly free and covers the common case. Reach for multi-region when you have a specific reason — a regulator, a latency requirement, or a downtime cost that justifies the second bill.
PASS 2MECHANISM
Three layers, each with its own multi-AZ story:
- Compute: an Auto Scaling group spanning subnets in three AZs, behind an Application or Network Load Balancer. The ALB has a node in each enabled AZ; when one dies, its node is withdrawn and the ASG replaces the lost capacity elsewhere. This is C3 and C4 machinery doing reliability work.
- Data: RDS Multi-AZ maintains a synchronous standby in another AZ and fails over by moving the DNS endpoint — typically 60–120 seconds, and the endpoint name doesn't change, so applications reconnect rather than reconfigure. Aurora goes further, with storage replicated six ways across three AZs. Multi-AZ is for availability, not read scaling — the standby serves no reads. Read replicas are a separate, asynchronous thing.
- Storage: S3 stores objects redundantly across multiple AZs by default in its standard classes. This is why "put the artefact in S3" is such a strong reliability move and why it matters so much for your model weights.
Going multi-region means doing each of these again, deliberately: S3 Cross-Region Replication for objects, cross-region read replicas or DynamoDB global tables for databases, AMIs and container images copied across, and CloudFormation or Terraform to rebuild the infrastructure itself.
PASS 3TRADE-OFFS & LIMITS
- Zone names are per-account. Your
us-east-1ais not myus-east-1a— AWS randomises the mapping to spread load. During the May 2026 event AWS named the physical zoneuse1-az4, an AZ ID, precisely because zone names are meaningless across accounts. Use AZ IDs when correlating with AWS communications or coordinating across accounts. - Global services have a home region. Several AWS global control planes are hosted in
us-east-1; Amazon ARC's is inus-west-2, deliberately. If your runbook calls a control-plane API in the burning region, your runbook doesn't work. Favour data-plane operations during recovery — this is why ARC's Region switch executes from the region being activated. - Cross-AZ data transfer costs money in both directions and adds ~1ms. Chatty microservices spread across three AZs can generate a surprising transfer bill; it's usually still worth it.
- Three AZs, not two, for anything quorum-based. Two AZs cannot survive one failure with a majority intact.
Decision tree · multi-AZ or multi-region for this workload?
GUARD CLAUSES · follow "no ↓" until a "yes" exits right
Buy the smallest blast-radius defence that meets the requirement
The four DR tiers as AWS services
§11 of the companion AWS reference names these four, and the names have been stable for a decade. What has changed is the tooling that orchestrates them.
Tier 1 · Backup and restore RTO hours–days · RPO hours
What runs in the DR region: nothing. You hold copies.
- AWS Backup is the central plane — one place to define policy, schedule, retention and cross-region copy for EBS, EC2, RDS, Aurora, DynamoDB, EFS and more. Backup Vault Lock makes copies write-once, which is the ransomware defence.
- EBS snapshots are incremental — only changed blocks are stored — and must be explicitly copied to another region to survive a regional event.
- S3 for objects, with versioning on and lifecycle rules ageing cold data into Glacier storage classes. Retrieval from deep archive takes minutes to hours; budget that into RTO if it's on your restore path.
- RDS automated backups give point-in-time recovery from transaction logs.
The part people get wrong: backing up data but not the infrastructure. Restoring a database into an empty region without VPCs, subnets, security groups or load balancers means building all of that under pressure. CloudFormation or Terraform in version control is a DR control, and its absence is usually what turns a 4-hour RTO into a 14-hour one.
Tier 2 · Pilot light RTO tens of minutes · RPO minutes
What runs in the DR region: the data layer, live. Compute exists as images and templates but is switched off.
- RDS cross-region read replica, promoted to primary during failover — a one-way operation, so the decision to promote is the real gate.
- DynamoDB global tables replicate continuously and are already multi-region active; for DynamoDB the pilot-light and active-active stories converge.
- S3 Cross-Region Replication keeps objects current. This is the mechanism that makes your model-weights bucket a solved problem.
- AMIs and ECR images copied to the DR region so launch is fast, plus the VPC scaffolding pre-built by IaC.
The material's guidance holds up well here: deploy infrastructure changes to both regions simultaneously via IaC, and consider a separate account per region for isolation. The failure mode of pilot light is drift — the DR region's templates quietly diverge from production over months, and you discover it during the disaster.
Tier 3 · Warm standby RTO minutes · RPO seconds–minutes
What runs in the DR region: everything, but small. A complete functional stack at perhaps 10–25% of production capacity.
- An ASG with a low minimum, behind a real ALB, in front of a promoted-on-demand replica. Auto Scaling grows it when traffic arrives.
- Route 53 health checks with failover routing can shift traffic automatically, or you can gate it behind a deliberate human decision.
The real advantage is not speed, it's confidence. A warm standby is continuously exercised: if a deployment breaks it, you find out on a Tuesday afternoon, not during the outage. This is the tier where the untested-backup problem stops existing, and that's worth more than the twenty minutes of RTO it saves over pilot light.
You can also send 1–5% of production traffic to the standby continuously. It costs almost nothing and converts "we believe it works" into "it served 40,000 requests yesterday".
Tier 4 · Multi-site active-active RTO near zero · RPO near zero
What runs in the DR region: production. There is no DR region; there are two production regions.
- Route 53 latency-based or weighted routing, or AWS Global Accelerator for anycast entry that sidesteps DNS caching entirely.
- DynamoDB global tables or Aurora Global Database for the data layer.
- Failover becomes "shift the weights", which is fast and reversible.
What you've actually bought: you have not removed the CAP trade-off, you've moved it into your data layer. Aurora Global Database replicates asynchronously with typically sub-second lag — near-zero RPO, not zero. DynamoDB global tables are last-writer-wins, so concurrent writes to the same item in two regions silently discard one. If your application cannot tolerate that, active-active is not available to you at any price, and warm standby is the honest ceiling.
Failover machinery: Route 53, and what replaced "Route 53 ARC"
PASS 1INTUITION
Route 53 is DNS with opinions. Beyond translating names to addresses, it can run health checks against your endpoints and change its answers based on the results. That's cross-region failover in its simplest form: two records, one health check, and DNS hands out the healthy one.
It works, and it's the right first answer. But DNS is cached, and during a real disaster you often need to coordinate a dozen steps — scale this, promote that, then move traffic — in a specific order, under stress. That coordination problem is what Amazon Application Recovery Controller exists to solve.
PASS 2MECHANISM
Route 53 health checks probe an endpoint from multiple AWS locations and require agreement from a threshold of them, which prevents one bad vantage point from triggering a failover. They can also watch a CloudWatch alarm instead of an endpoint — useful when "healthy" is a composite judgement rather than an HTTP 200.
Failover routing policy pairs a primary and a secondary record; the secondary is served only when the primary's health check fails. Keep the TTL low — 60 seconds is typical — and remember that your effective failover time is detection plus TTL plus badly-behaved resolvers.
Amazon ARC adds three things on top:
- Readiness checks continuously audit whether the standby could actually take the load — capacity, configuration, quotas, routing. This is automated defence against the untested backup.
- Routing controls are on/off switches with safety rules, so you flip traffic deliberately rather than trusting an automatic health check to make a region-level call.
- Region switch orchestrates the whole sequence from a declarative plan — scale EKS or ECS, run Lambda steps, orchestrate RDS, update Route 53 — with a practice mode and a data plane in every region, so the plan executes from the region being activated rather than the one that's failing.
PASS 3TRADE-OFFS & LIMITS
- Automatic region failover is usually a mistake. Zone failover should be automatic; region failover should be a decision. A false positive that moves all traffic to a cold region can cause a worse outage than the one it was reacting to — which is exactly why ARC separates health signals from routing controls.
- DNS TTL is a floor, not a guarantee. Corporate resolvers and JVMs that cache DNS forever will keep hitting the dead endpoint. Global Accelerator or CloudFront avoid this by never changing the address clients hold.
- Region switch plans self-validate every 30 minutes — configuration and IAM permissions. That cadence is the product's answer to drift, and it's the feature to sell internally.
- Practice mode is the point. A plan you have never executed is a wish with YAML syntax. Pair it with AWS Fault Injection Service, which can simulate an AZ power interruption, and you have an actual rehearsal rather than a tabletop.
⚠ BOOK IS STALE · DR orchestration renamed and expanded
Your guide (and most 2024-era material) says: Route 53 Application Recovery Controller, offering readiness checks and routing controls.
Current as of July 2026: the service is now Amazon Application Recovery Controller (ARC) — the "Route 53" prefix is gone, reflecting that it orchestrates far more than DNS. Region switch launched in 2025 as a full multi-region recovery orchestrator with declarative plans, practice mode and per-region data planes; GovCloud support arrived December 2025, and post-recovery workflows, native RDS execution blocks and Terraform provider support in February 2026.
Sources: aws.amazon.com/application-recovery-controller Region switch in ARC — AWS docs
CloudWatch: the umbrella, and what's underneath it
PASS 1INTUITION
CloudWatch is not one product. It's a brand covering a metrics database, a log store, an alarm engine, a dashboard tool, a synthetic-monitoring service and — since the X-Ray absorption — an APM product. Treat "CloudWatch" as a shelf rather than a thing, and ask which item on the shelf you actually need.
The mental model that helps: it is a metrics repository first. AWS services push numbers in; you graph them, alarm on them, and act on them. Logs and traces were bolted on later and still feel slightly separate.
PASS 2MECHANISM
- Metrics are identified by namespace, name and dimensions (instance ID, environment). Most services publish free default metrics; anything from inside the guest OS — memory, disk usage — requires the CloudWatch agent, because AWS can't see inside your VM. Metric data is retained for 15 months at decreasing resolution.
- Alarms evaluate a metric against a threshold for N periods and move between OK, ALARM and INSUFFICIENT_DATA. They trigger SNS, Auto Scaling, EC2 actions or EventBridge. Composite alarms combine several so you page on "the service is broken" rather than on nine correlated symptoms.
- Logs live in log groups (one per application) containing streams (one per source). Metric filters turn a text pattern into a countable metric — the log-based-metric bridge from S1. Logs Insights runs SQL-like queries across groups. Console delivery lags several minutes; when you need real time, use subscription filters to stream to Kinesis or Lambda.
- CloudTrail is not CloudWatch. CloudTrail records API calls — who did what, from where. It's the audit signal and answers different questions. Send it into CloudWatch Logs when you want to alarm on it.
- Metric Streams continuously export metrics to S3, Redshift, OpenSearch or third parties in near real time — the escape hatch when you want the data somewhere else.
PASS 3TRADE-OFFS & LIMITS
- Metrics don't aggregate across regions natively. Each region is its own repository. A global view needs cross-account/cross-region dashboards, Metric Streams into one place, or Managed Grafana pointed at several sources — a real design decision for multi-region workloads.
- Custom metrics are billed per metric per month, where a metric is a unique name-plus-dimension combination. Adding a high-cardinality dimension multiplies your bill by its cardinality. Emit request IDs as logs, never as dimensions.
- Alarms on averages hide everything. Average latency across a fleet stays flat while one instance times out every request. Alarm on percentiles, and alarm per target group where it matters.
- INSUFFICIENT_DATA is not OK. Decide explicitly how each alarm treats missing data. An alarm that goes quiet when the instance dies — because it stopped publishing — is worse than no alarm.
⚠ BOOK IS STALE · tracing has moved to OpenTelemetry
Your guide says: instrument with the X-Ray SDK, run the X-Ray daemon, view traces in the X-Ray console.
Current as of July 2026: the X-Ray SDKs and Daemon entered maintenance mode on 25 February 2026 and reach end-of-support on 25 February 2027. The supported path is OpenTelemetry — AWS Distro for OpenTelemetry (ADOT) or vanilla OTel SDKs — feeding CloudWatch Application Signals (auto-discovered services, dependency maps, SLOs with error budgets) and Transaction Search. Transaction Search is the significant one: it stores spans as structured logs in an aws/spans log group and gives 100% span visibility, versus X-Ray's default 5% sampling. Existing X-Ray instrumentation keeps working; new work should not use it.
Application Signals also gained SLO recommendations, service-level SLOs and SLO performance reports in March 2026 — this is now a credible native SLO story on AWS, which it was not when the material was written.
Sources: AWS Cloud Operations Blog — X-Ray end-of-support Transaction Search — AWS docs
⚠ WORTH KNOWING · the open-source path is first-class on AWS now
Amazon Managed Service for Prometheus (AMP) accepts remote-write from the Prometheus you already run and speaks PromQL. Amazon Managed Grafana (AMG) now creates Grafana 12.4 workspaces, with in-place upgrade from 10.4 available since May 2026 — that brings Scenes-powered dashboards and queryless Drilldown apps for Prometheus metrics, Loki logs and Tempo traces.
For your stack this is the shortest migration path on any cloud: point remote-write at AMP, import your existing dashboards into AMG, keep every PromQL expression. Watch the pricing model though — AMP bills separately for samples ingested, samples stored, queries processed and managed collectors, and AMG bills per active user.
Sources: AMG 12.4 in-place upgrade (May 2026) AMP document history
The three-cloud view, anchored on AWS
| Need | AWS | Azure | Google Cloud |
|---|---|---|---|
| Zone-level redundancy for VMs | ASG across 3 AZs + ALBCapacity is replaced automatically when a zone is lost.AZ names are randomised per account; use AZ IDs when talking to AWS. | VM Scale Set, zone-redundantInstances spread across zones in one region.Availability sets are the older, weaker rack-level construct — not a zone substitute. | Regional managed instance groupSpreads across zones by default."Regional" is GCP's word for multi-zone, which trips up AWS people constantly. |
| Synchronous DB standby | RDS Multi-AZ / AuroraFailover ~60–120 s; endpoint name unchanged.Standby serves no reads — that's what read replicas are for. | Zone-redundant Azure SQL / Flexible ServerZone-redundant HA option per tier.Zone redundancy availability varies by service tier and region. | Cloud SQL HA / AlloyDBRegional instance with a standby in a second zone.HA is an instance property set at creation and awkward to add later. |
| Object durability + cross-region copy | S3 + Cross-Region ReplicationMulti-AZ by default; CRR for the second region.CRR only replicates objects written after it's enabled — backfill separately. | Blob Storage, GRS / RA-GRSRedundancy chosen as an account-level setting.Read access to the secondary needs the RA- variant, chosen up front. | Cloud Storage dual-region / multi-regionRedundancy is a bucket location type.Turbo replication gives an RPO target; standard replication does not. |
| Central backup policy | AWS Backup + Vault LockOne policy plane across many services, cross-region copy, write-once immutability.Not every service is supported — check before assuming coverage. | Azure BackupRecovery Services / Backup vaults, soft delete, immutability.Managed from "Resiliency in Azure", formerly Business Continuity Center. | Backup and DR ServiceBackup vaults with immutable, indelible copies.Distinct from PD snapshots, which are a mechanism, not a backup product. |
| Cross-region traffic failover | Route 53 failover routing + health checksOr Global Accelerator for anycast, avoiding DNS caching.Effective RTO = detection + TTL + resolvers that ignore TTL. | Traffic Manager (DNS) / Front Door (anycast)Front Door adds WAF and caching.Front Door is a global dependency — it caused the 29 Oct 2025 outage. | Global External Application Load BalancerSingle global anycast IP, no DNS in the failover path.Genuinely the cleanest of the three for cross-region failover. |
| Orchestrated, rehearsable failover | Amazon ARC · Region switchDeclarative plans, practice mode, self-validating every 30 min.Executes from the region being activated, not the failing one. | Azure Site Recovery recovery plansOrdered steps with non-disruptive test failover.Test failover into an isolated VNet is the feature that earns its keep. | Compose it yourselfTerraform + MIGs + DNS, no single orchestrator.Best primitives, weakest turnkey story. |
| Alarm → action | CloudWatch Alarms → SNS / EventBridgeComposite alarms reduce noise.Set the missing-data behaviour explicitly or alarms go silent when hosts die. | Alert rule → action groupReusable notification and automation bundle.Scope is explicit; set it too narrow and resources are silently unmonitored. | Alerting policy → notification channelIncludes native SLO burn-rate conditions.Burn-rate alerting is built in rather than hand-assembled. |
| Distributed tracing | Application Signals + Transaction SearchOTel-based; 100% spans in a log group.X-Ray SDK/Daemon end-of-support 25 Feb 2027 — start on OTel. | Application InsightsDependency maps and live metrics.Adaptive sampling on by default — verify before trusting counts. | Cloud TraceNative OTLP at telemetry.googleapis.com.Trace sinks deprecated Feb 2026; Observability Analytics replaces them. |
| DR / chaos rehearsal | AWS Fault Injection Service + ARC practice modeCan simulate AZ power interruption.The only way to prove an RTO instead of asserting it. | Azure Chaos Studio + ASR test failoverZone-down drill in Resiliency in Azure.Test failover doesn't disrupt production replication — use it quarterly. | Open-source chaos tooling on GKENo first-party service.Google's answer is cultural: DiRT exercises and game days. |
S2 · SECTION 3 · reality check25%
What the rungs actually cost
WORKED EXAMPLE · ILLUSTRATIVE MODEL
Pilot light vs warm standby: the monthly delta for a conventional stack
A three-tier web application. Production runs 4 × m6i.large behind an ALB, with an db.r6g.large Multi-AZ database. What does each DR rung add in a second region?
How to use this number. Two thousand dollars a year buys forty minutes of RTO. Whether that's a bargain depends entirely on one figure the business owns: cost per minute of downtime. Common industry benchmarks put enterprise downtime somewhere between $5,600 and $9,000 per minute — at those rates the warm standby pays for itself in the first twenty seconds of the first incident. For an internal tool used by thirty people, it obviously doesn't. Ask for the number before you design the tier.
WORKED EXAMPLE
An inference-endpoint SLO, written the AWS way
Application Signals supports request-based SLOs — the fraction of good requests out of total — which is exactly the shape S1 argued for. For a vLLM endpoint behind an ALB:
GPU / GenAI Two things fall out of this that don't apply to a normal web service. First, autoscaling lag dominates your error budget, because spinning up a GPU node and loading weights takes minutes, not the seconds a stateless container takes. The mitigation is a warm minimum — which is a reliability purchase, not a performance one. Second, set the SLO where you can actually live: a 99.9% latency SLO on a spot-GPU deployment will be missed every single month, and an SLO that is always red is functionally the same as no SLO at all.
Three things that actually happened, on AWS
POSTMORTEM 7–8 May 2026 — one hot room, and who had really done multi-AZ
At about 17:25 PDT on 7 May, multiple chiller units failed in a single data hall in Northern Virginia. Temperatures rose past operating thresholds and servers executed emergency shutdowns to avoid physical damage. EC2 instances and EBS volumes in use1-az4 lost power. AWS shifted traffic away from the zone for most services, but recovery required physically restoring cooling and re-energising hardware in a controlled sequence — no rollback available. AWS was still reporting recovery in progress more than fifteen hours later.
The instructive part is the customer split. Workloads genuinely spread across three AZs absorbed it: the ASG replaced lost capacity elsewhere, RDS Multi-AZ failed over, and the day continued. Coinbase's matching engine lost quorum because three of its five nodes were in that one zone, and the exchange was inaccessible for roughly seven hours. FanDuel and CME Group were also disrupted. AWS's own guidance during the event was to restore from EBS snapshots or launch in unaffected zones.
What to take from it: "we're multi-AZ" is a claim about node placement per cluster, not about the account. Audit quorum-based systems specifically, using AZ IDs. And note the second-order failure: abruptly de-powered EBS volumes can come back in a stuck state, which is why cross-region snapshot hygiene matters even for a zone-level event.
Sources: AWS Health Dashboard incident updates, 7–8 May 2026; Network World
POSTMORTEM 19–20 October 2025 — the DynamoDB DNS race, read as an AWS architecture lesson
Covered in S1 for the SPOF principle; here's what it teaches about AWS specifically. AWS's own summary describes three distinct impact windows: DynamoDB API errors from 23:48 PDT on the 19th to 02:40 on the 20th, then Network Load Balancer connection errors from 05:30 to 14:09 caused by health-check failures across the NLB fleet, with full resolution at 14:20.
The middle window is the interesting one. NLB's own health checking depended on state that depended on DynamoDB. EC2's DropletWorkflow Manager likewise. This is a hidden serial dependency inside the provider's own control plane — you cannot see it in your architecture diagram, and no amount of multi-AZ design protects against it.
Practical takeaways: customers using DynamoDB global tables retained access to replicas in other regions, though with replication lag to and from us-east-1 — multi-region data paid off. Retry storms without jitter made recovery slower for everyone, so exponential backoff with jitter in your SDK config is a contribution to collective recovery, not just your own. And AWS's remediation list included adding test coverage for the DWFM recovery workflow — again, the untested path was the recovery path.
Source: AWS — Summary of the Amazon DynamoDB Service Disruption (US-EAST-1)
GUIDANCE The AWS Well-Architected Reliability Pillar, and the DR whitepaper the material cites
The four-tier ladder in your guide comes directly from Disaster Recovery of Workloads on AWS: Recovery in the Cloud, which remains the canonical reference and is still maintained. Read it alongside the Reliability Pillar of the Well-Architected Framework, which organises the same material around five design principles: automatically recover from failure, test recovery procedures, scale horizontally to increase aggregate availability, stop guessing capacity, and manage change through automation.
"Test recovery procedures" is the one to take personally. AWS's position — and the whole reason Fault Injection Service and ARC practice mode exist as products — is that an untested recovery procedure should be assumed broken. The May 2026 and October 2025 events both produced remediation items about testing recovery paths, at AWS's own scale.
Worth also knowing that AWS Resilience Hub lets you declare RTO and RPO targets for an application and then continuously assesses your actual architecture against them, producing a score and specific remediation. It's the closest thing to an automated grader for everything in this session.
Sources: DR of Workloads on AWS (whitepaper) AWS Cloud Resilience
S2 · SECTION 4 · apply it — your context + hands-on10%
Your stack, on AWS
Migration path · keep what you have
- Prometheus → AMP. Add a
remote_writeblock pointing at your AMP workspace with SigV4 auth. Your scrape configs, recording rules and alert rules move unchanged. This is the least disruptive managed-monitoring migration available on any cloud. - Grafana → Amazon Managed Grafana. Import your dashboard JSON, add AMP and CloudWatch as data sources. You now have GPU metrics from vLLM and instance metrics from CloudWatch on one panel.
- Langfuse → OTel → Application Signals. Instrument with ADOT and enable Transaction Search for full span visibility. Keep Langfuse for prompt/eval data — it does something Application Signals doesn't.
- Alertmanager → CloudWatch Alarms + SNS, or keep Alertmanager against AMP if you prefer PromQL alerting. Both are supported; AMP also integrates with PagerDuty as an alert receiver.
DR design for a stateless vLLM service
- Weights: S3 bucket with versioning and Cross-Region Replication to the DR region. This is your entire RPO story and it costs a few dollars.
- Images: ECR replication configured to the DR region. Pulling a 12 GB container across regions during an incident is not a plan.
- Infrastructure: the same Terraform module applied to both regions, differing only in variables.
- Capacity: the hard part. Either accept "we'll request GPUs and hope", or hold one warm replica, or hold an EC2 Capacity Reservation in the DR region — a reservation costs the instance price but guarantees the capacity exists.
- Traffic: Route 53 failover with a deep health check that verifies a real inference completes, not that the port is open.
Compare that against a stateful service and the asymmetry is stark: no replication lag to reason about, no promotion decision, no split-brain risk. Your DR is a deployment problem. The corollary — which is easy to miss — is that your DR plan is only as good as your deploy pipeline, so pipeline reliability is now a DR concern.
OPTIONAL HANDS-ON · S2 · free tier, smallest SKU, budget alert first
An inference-shaped SLO on CloudWatch, without a GPU
You don't need a GPU to learn this — you need a metric that behaves like one. This lab publishes a synthetic latency metric, alarms on it the right way, and shows you the log-based-metric bridge. Total cost: pennies, or zero within free tier.
- Budget alert before anything else. Billing → Budgets → create a monthly cost budget of $5 with an alert at 50% and 80%. Do this first, every time, on every account.
- Create a log group
/lab/inference. Publish a handful of JSON log lines from CloudShell (free) that look like inference records —{"ttft_ms": 1450, "status": 200}. - Create a metric filter on that group extracting
ttft_msas a metric. This is the log-to-metric bridge from S1, and the config is the mechanism:# metric filter pattern { $.ttft_ms = * } → metric name: TTFTMilliseconds → namespace: Lab/Inference → value: $.ttft_ms - Create an alarm on the p95 statistic of that metric, not the average — threshold 2000 ms, 2 out of 3 datapoints. Set Treat missing data as to
breachingand notice why that choice matters when the publisher stops. - Publish a burst of slow values and watch the alarm transition. Then look at the same data as an average and see how thoroughly it hides the problem.
- Optional, if you want the DR piece: create two Route 53 health checks against any two public endpoints and configure failover routing in a test hosted zone. Health checks bill per check per month — a few cents.
TEARDOWN — in this order
- Delete the log group
/lab/inference. Log groups retain forever by default and ingestion bills continuously — this is the number-one surprise charge on learning accounts. - Delete the CloudWatch alarm and any custom metrics you created (custom metrics bill per metric per month even with no data flowing).
- Delete Route 53 health checks — they bill monthly whether or not anything depends on them.
- Delete the test hosted zone (hosted zones bill monthly).
- If you created a Capacity Reservation while experimenting, cancel it — it charges the full instance rate whether or not an instance is running.
- Leave the budget alert in place. It costs nothing and it's the safety net.
S3 crosses to Azure, where the same ideas carry different names and one genuinely different shape: Azure separates metrics and logs into two stores with two query languages, and its DR story is a dedicated replication product rather than a set of primitives.
You'll also meet the outage that best illustrates blast radius — a security policy change that took down virtual machine provisioning worldwide.
S3 · SECTION 1 · why this session exists10%
A permission change that stopped the world's VMs
At 18:03 UTC on 2 February 2026, a policy intended to improve security by disabling anonymous access to storage accounts was applied — because of a data-synchronisation problem in its targeting logic — to a set of Microsoft-managed storage accounts that were deliberately configured for anonymous read. Those accounts host virtual machine extension packages.
VM agents download extension packages during provisioning. So every VM create, scale, start and configuration operation that touched an extension began failing across multiple regions. Scale sets couldn't add instances. AKS node pools couldn't scale out. Azure DevOps and GitHub Actions pipelines failed when hosted runners couldn't be built. Azure Backup, Container Apps, Batch, Firewall and Redis were all affected downstream. A region-by-region mitigation then overloaded the Managed Identity service in East US and West US, producing a second, distinct outage lasting nearly six hours. Total elapsed time before full resolution: over ten hours.
Two lessons, and they set up everything below. First, blast radius is a property of dependencies, not of geography — no zone or region choice would have helped, because the failure was in a shared platform artefact store. Second, a security control caused an availability incident, which is the C2 thread arriving here: least privilege limits blast radius, and privilege changes create it.
Source: Microsoft Azure status history, tracking ID FNJ8-VQZ — azure.status.microsoft
S3 · SECTION 2 · core concepts50%
Azure Monitor: one brand, two very different stores
PASS 1INTUITION
If CloudWatch is a shelf of loosely related products, Azure Monitor is a single umbrella with a hard seam down the middle. Everything on the left is Metrics: numbers, near-real-time, cheap, queried with a simple chart syntax. Everything on the right is Logs: structured records in a Log Analytics workspace, queried with a full analytics language called KQL.
Grasping that seam early saves a lot of confusion. When an Azure document says "logs", it often means any data in the workspace — including things you'd call metrics elsewhere, and including Application Insights traces. The workspace is a general-purpose analytics store that happens to be sold as a logging product.
PASS 2MECHANISM
Metrics are organised as namespace → metric → dimensions → time series. Platform metrics arrive automatically from Azure resources with no configuration. Guest-OS metrics (memory, disk) need the Azure Monitor Agent, for the same reason AWS needs the CloudWatch agent: the platform can't see inside your VM.
Logs land in a Log Analytics workspace, which is the unit of collection, retention, access control and billing. You choose which resources send diagnostic settings to which workspace, and that choice is an architecture decision — one central workspace is easy to query and hard to govern; one per team is the reverse.
KQL is the query language, and it's genuinely good. It reads left to right as a pipeline:
Application Insights is the APM layer: request rates, dependency maps, distributed traces, exceptions and live metrics. Modern workspace-based Application Insights writes into a Log Analytics workspace, which is why its data is queryable with the same KQL as everything else — a real advantage over having three separate query dialects.
Alerts come in three flavours — metric alerts (fast, cheap, on time series), log alerts (a KQL query on a schedule), and activity log alerts (control-plane events: someone deleted a resource, a service health event was published). All of them fire an action group.
PASS 3TRADE-OFFS & LIMITS
- The action group is the reusable unit, and that's a genuine design win. Notification targets and automation actions — email, SMS, webhook, Logic App, Azure Function, ITSM connector — are defined once and referenced by many rules. Change the on-call rota in one place. AWS achieves this with SNS topics but less formally.
- Alert scope is explicit and easy to get wrong. A rule scoped to a resource group silently ignores resources created in a different group later. Scope at subscription level with resource-type filters when you want "all VMs", not "these VMs".
- Log alerts have a floor on frequency and cost per evaluation. Don't reach for a log alert when a metric alert would do — it's slower and dearer.
- Table plans change your bill by up to 45×. This is the single biggest cost lever in Azure observability and it's per-table. Covered in the reality check below.
- Service Health alerts are underused. Azure publishes platform incidents, planned maintenance and health advisories into the activity log. An action group on Service Health events means you learn about the next FNJ8-VQZ from Azure rather than from your users. Set this up on day one of any subscription.
Zones, sets and region pairs — Azure's three redundancy words
PASS 1INTUITION
Azure has an extra concept AWS doesn't, and it causes real confusion. An availability set spreads VMs across racks and maintenance schedules inside one datacentre. An availability zone spreads them across physically separate datacentres. A set protects against a rack failure and a host reboot; a zone protects against the datacentre burning down.
They are not alternatives at the same level, despite being presented side by side in the portal. Sets are the older construct from before zones existed. If zones are available in your region — and they now are in most — use zones. Reach for a set only when zones aren't offered.
PASS 2MECHANISM
- Availability set = fault domains (distinct racks, power, network switch) × update domains (groups rebooted separately during platform maintenance). Typically 2–3 fault domains and up to 20 update domains.
- Zonal = you pin a resource to a specific zone. You control placement; you also own the redundancy. Useful for pinning a workload next to its data.
- Zone-redundant = the platform spreads the resource across zones for you and handles failover. This is what you almost always want, and it's a per-service capability — zone-redundant Azure SQL, zone-redundant Load Balancer, zone-redundant Storage (ZRS).
- Region pairs are Azure's distinctive idea: most regions are paired with another in the same geography, and the platform sequences updates so paired regions aren't updated simultaneously, and prioritises recovery. Geo-redundant storage (GRS) replicates to the pair automatically.
Storage redundancy names encode this directly and are worth memorising because they're an exam favourite and a real design choice: LRS (three copies, one datacentre) → ZRS (across zones in one region) → GRS (LRS locally + async copy to the paired region) → GZRS (ZRS locally + async to the pair). Add RA- for read access to the secondary — and you must choose that up front, because the read endpoint doesn't exist otherwise.
PASS 3TRADE-OFFS & LIMITS
- Zone redundancy availability varies by service, tier and region. Unlike AWS, where multi-AZ is nearly universal, on Azure you must check per service and per SKU. Cheaper tiers frequently don't offer it.
- Zone redundancy is not a guarantee against correlated failure. The root cause analysis for a 2025 East US 2 networking incident noted that services configured as zone-redundant and using VNet integration may have experienced impact across multiple zones, because the failure was in a control-plane component shared across zones. Zone redundancy defends against physical failure, not shared logical dependencies.
- Region pairs are a platform behaviour, not a DR plan. GRS gives you a copy in the pair, but with an RPO measured in minutes and — without RA- — no read access until Microsoft initiates a failover or you trigger a customer-managed one. Don't confuse "geo-redundant storage" with "we have DR".
- Paired-region failover for storage is not instant. Understand your account's failover semantics before relying on them in an RTO calculation.
Azure Backup and Site Recovery — two products, two jobs
PASS 1INTUITION
This is one place where Azure is clearer than its competitors, because the split matches the S1 vocabulary exactly. Azure Backup is backup: point-in-time copies you restore from, protecting against deletion, corruption and ransomware. Azure Site Recovery (ASR) is replication: continuous copying of running machines to a second region so you can fail over.
Backup answers "someone destroyed the data, take me back to yesterday". Site Recovery answers "the region is gone, run my machines somewhere else". You need both, and neither substitutes for the other — which is exactly the replication-is-not-backup point from S1, sold as two SKUs.
PASS 2MECHANISM
Azure Backup stores recovery points in a Recovery Services vault or a Backup vault, depending on workload. The features that matter for the S1 threat model: soft delete (deleted backups are retained for a grace period, so an attacker who deletes your backups hasn't), immutability (recovery points can't be modified or shortened), and long-term retention up to a decade for compliance.
Azure Site Recovery continuously replicates disk writes from source machines to the target region, maintaining crash-consistent and application-consistent recovery points. Two features carry most of the value:
- Recovery plans — an ordered sequence of groups with scripts and manual steps between them. Bring up the database tier, run a script, then the app tier. This is the runbook, executable.
- Test failover — spins the replicated machines up in an isolated virtual network without disrupting ongoing replication or touching production. This is the single most important feature in the product, because it directly kills the untested-backup problem. Run it quarterly and your RTO becomes a measurement.
ASR now surfaces default alerts through Azure Monitor for critical events — replication health degrading, failover failures, agent expiry — routable through the same action groups as everything else, so DR health lands in your normal alerting pipeline rather than a separate portal blade nobody opens.
PASS 3TRADE-OFFS & LIMITS
- ASR has a churn ceiling. The data-change rate per disk is capped; the High Churn option raises it substantially (to roughly 100 MB/s per VM) but write-heavy databases can still exceed it. Exceeding the limit silently degrades your RPO — which is the worst possible failure mode for a DR product, because it fails quietly.
- ASR replicates machines, not applications. It doesn't know your database needs quiescing, or that service B must start before service A. That's what recovery plans and scripts are for, and writing them is the actual work.
- For PaaS, ASR is usually the wrong tool. Azure SQL, Cosmos DB and App Service have their own geo-replication and failover-group mechanisms. ASR is an IaaS product.
- Test failover costs money while it runs — you're paying for real VMs in the target region. Budget for it, and clean up the test resources afterwards, which the product will prompt you to do.
⚠ BOOK IS STALE · three Azure names have moved
1. Business continuity management. Material from 2024–25 refers to Backup center, then Azure Business Continuity Center (ABCC). As of June 2026, ABCC has become "Resiliency in Azure" — a broader platform consolidating zone resiliency and high availability, backup and disaster recovery, and ransomware protection, with a zonal-resiliency preview that includes a zone-down drill and recovery orchestration. Backup center is no longer reachable from portal search; it's under the Resiliency Help menu.
2. Managed Grafana versions. Grafana 11 support retired on 15 June 2026 — workspaces still on 11 are auto-upgraded to Grafana 12. Separately, the Essential SKU retires on 30 March 2027; move to Standard or to Azure Monitor dashboards with Grafana.
3. Site Recovery classic experience. The classic experience required transition to the modernised experience by 30 March 2026. Any tutorial showing the old flow is describing something that no longer exists.
Sources: What is Resiliency? — Microsoft Learn Upgrade to Grafana 12 ASR feature updates
⚠ WORTH KNOWING · the open-source path on Azure, and a free Grafana you may have missed
Azure Monitor managed service for Prometheus stores metrics in an Azure Monitor workspace (a distinct resource from a Log Analytics workspace — the naming is unhelpful). It's fully PromQL-compatible, integrates natively with AKS including control-plane metrics, and retains data for 18 months with no separate storage charge.
For dashboards there are now two options. Azure Managed Grafana is the full product with plugins and non-Azure data sources. Azure Monitor dashboards with Grafana is a newer in-portal Grafana experience available at no cost and with no configuration, supporting Azure Monitor Metrics, managed Prometheus, Logs, Traces and Resource Graph. If your data sources are all Azure, start with the free one.
Sources: Visualize Azure Monitor data with Grafana Azure Monitor with Prometheus overview
Decision tree · alert, dashboard, or ignore?
Alert fatigue is a reliability problem, not a hygiene problem: a team that ignores pages will ignore the real one. Run every proposed signal through this before creating a rule.
GUARD CLAUSES · follow "no ↓" until a "yes" exits right
What should this signal actually do?
The three-cloud view, anchored on Azure
Azure's distinctive shapes — the two-store split, the action group, the paired region, the separate backup and replication products — each have counterparts elsewhere, but the seams fall in different places. This is the table to read when you already know Azure and need to translate.
| Azure shape | Azure | AWS | Google Cloud |
|---|---|---|---|
| Collection + billing boundary | Log Analytics workspaceOne resource that owns collection, retention, access control and cost.Also confusingly distinct from an Azure Monitor workspace, which holds Prometheus metrics. | Log group (per app), account+regionNo single equivalent object; scope is the account and region.Retention is set per log group and defaults to forever — the classic surprise bill. | Log bucket + metrics scopeBuckets hold logs; metrics scopes group projects for monitoring.Metrics scopes don't affect billing, so create them freely. |
| Query language | KQLA genuine analytics language over logs, metrics and traces in one store.Strongest query story of the three — and a real learning investment. | Logs Insights + PromQL + PPL/SQLDifferent dialects per store.CloudWatch Logs gained PPL and SQL support in 2026, easing the split. | Logging query language + PromQL + SQLObservability Analytics adds SQL over logs and traces.PromQL works across both Prometheus and native Cloud Monitoring metrics. |
| Reusable notification bundle | Action groupEmail, SMS, webhook, Logic App, Function, ITSM — defined once, referenced by many rules.The cleanest of the three. Change the rota in one place. | SNS topicAchieves the same thing less formally.No native ITSM connector concept; you wire it yourself. | Notification channelsAttached per alerting policy.Channel list is reusable, but there's no single named bundle object. |
| APM / distributed tracing | Application InsightsRequests, dependencies, exceptions, live metrics, all queryable with the same KQL as your logs.Adaptive sampling is on by default — check it before trusting counts mid-incident. | Application Signals + Transaction SearchOTel-based, 100% spans stored as structured logs.X-Ray SDK/Daemon end-of-support 25 Feb 2027. | Cloud TraceNative OTLP ingest, generous attribute limits.Trace sinks deprecated Feb 2026 in favour of Observability Analytics. |
| Cheap tier for high-volume logs | Table plans: Analytics / Basic / AuxiliaryChosen per table; roughly $2.30 / $0.50 / $0.05 per GB.Up to a 45× spread — the single biggest observability cost lever on Azure. | Log class: Standard / Infrequent AccessPlus S3 export for archive.Fewer tiers, so archival usually means leaving CloudWatch entirely. | Exclusion filters + sink to GCSRouter drops or redirects before storage.Excluded logs are gone, not cheaper — sample before you exclude. |
| Backup, as a distinct product | Azure BackupVaults with soft delete, immutability, long-term retention.Managed from "Resiliency in Azure", the renamed Business Continuity Center. | AWS Backup + Vault LockCross-service policy plane with write-once vaults.Coverage varies by service — verify rather than assume. | Backup and DR ServiceBackup vaults, immutable and indelible.Its Cloud Monitoring metrics are non-chargeable. |
| Replication for failover, as a distinct product | Azure Site RecoveryRecovery plans plus non-disruptive test failover into an isolated VNet.Churn ceiling per disk; exceeding it degrades RPO silently. | Elastic Disaster Recovery + ARC Region switchBlock replication plus declarative orchestration.Region switch plans self-validate every 30 minutes. | No direct equivalentComposed from replication primitives and Terraform.Best primitives, no orchestrator, no practice mode. |
| Platform incident notification | Service Health alertsIncidents, planned maintenance and advisories into the activity log, routed to an action group.Free, high value, and routinely not configured. Do it on day one. | AWS Health Dashboard + EventBridgeAccount-specific events can trigger automation.Personal Health Dashboard events are the ones that matter, not the public status page. | Service Health dashboard + Personalized Service HealthIncidents surfaced per project.Route to Pub/Sub if you want it in your own tooling. |
| Named cross-region relationship | Region pairsPlatform sequences updates and prioritises recovery; GRS replicates to the pair.A platform behaviour, not a DR plan you designed. | None — regions are independentYou choose the pairing yourself.More work, fewer surprises about where your data went. | Dual-region bucketsStorage-level pairing with optional turbo replication.Turbo replication gives a stated RPO; standard replication does not. |
S3 · SECTION 3 · reality check25%
Where the Azure bill and the Azure nines actually come from
WORKED EXAMPLE 1
Log Analytics table plans: the same data, 45× the price
Azure Monitor's cost is overwhelmingly log ingestion and retention. There are three ingestion plans, chosen per table, and the gap between them is enormous. Model a modest GPU inference cluster.
The judgement call. Basic Logs give up scheduled-alert capability and charge per GB scanned at query time. So the rule is: tables you alert on go to Analytics; tables you only read during an incident go to Basic; tables you keep for compliance go to Auxiliary. Industry reviews consistently find 20–40% of ingested volume is never queried at all — for that portion the right plan is "don't collect it", which a data collection rule can enforce at the agent.
This is also a reliability point, not just a cost one. Teams that get a shocking observability bill respond by turning off collection indiscriminately, and then fly blind through the next incident. Tiering deliberately is how you keep the visibility and lose the cost.
WORKED EXAMPLE 2
What zone redundancy is actually worth, in nines
Read that last block carefully, because it's counter-intuitive and it's the point. Adding a second region to a well-built zone-redundant deployment can reduce measured availability, because you've added a DNS steering layer in series and introduced failover events that themselves cause brief hard-down periods. The second region is not bought for the nines. It's bought for the tail event the nines don't model — a whole-region loss, a compliance requirement, or a February-2026-style platform incident where zones don't help.
Say this out loud in design reviews: "multi-region is catastrophe insurance, not an availability upgrade."
Three things that actually happened, on Azure
POSTMORTEM 2–3 February 2026 — FNJ8-VQZ, and the anatomy of a cascade
Microsoft's own status history describes it precisely: a data-synchronisation problem in the targeting logic of a policy meant that a change disabling anonymous read access was incorrectly applied to a subset of storage accounts that are intentionally configured to allow anonymous read for platform functionality — the VM extension package storage layer.
The cascade: VM agents couldn't download extension packages → VM create, update, scale, start and stop operations failed → VMSS couldn't scale → AKS node provisioning stalled → CI/CD pipelines on hosted runners failed, including GitHub Actions → Azure Backup, Container Apps, Batch, Firewall, Search and Redis all reported impact. Then the region-by-region mitigation drove load into Managed Identity, producing a second incident in East US and West US from 00:15 to 06:05 UTC affecting token acquisition.
Three things to take from it. One: your dependency graph includes platform artefacts you never chose. Two: the mitigation caused the second outage — a retry storm from the first — which is why staged mitigation matters as much as staged deployment. Three: for you specifically, note that AKS node scale-out was blocked. If your inference autoscaler had wanted a new GPU node during those ten hours, it would not have got one, and your error budget would have paid for it.
Sources: Azure status history — FNJ8-VQZ The Register, 3 Feb 2026
POSTMORTEM 29 October 2025 — Azure Front Door, and the cost of a global front door
An inadvertent configuration change in Azure Front Door — the global anycast routing, CDN and WAF layer — cascaded across more than a dozen services including Azure SQL, Virtual Desktop, Microsoft 365 apps and several security products. Peak Downdetector reports exceeded 18,000. Recovery required rolling back to a last-known-good configuration and reloading edge nodes carefully to avoid overloading them on return.
This is worked example 2 from S1 happening in public. Every service behind Front Door was individually healthy. The parallel redundancy term was excellent. The serial term — one global routing layer — was zero, and the total is dominated by the serial term. It came nine days after the AWS DynamoDB event, which is why late 2025 changed how a lot of architects talk about global front doors.
The design implication is uncomfortable: an anycast global front door is genuinely better than DNS failover for RTO, because it removes TTL caching from the failover path. It is also a new global dependency. The mitigation is not to avoid it but to know it's there — keep an origin-direct path you can publish, keep its DNS records pre-created with low TTL, and rehearse using it.
Source: Azure status history, 29 October 2025; contemporaneous incident reporting
GUIDANCE The Azure Well-Architected Reliability pillar, and the one Azure-specific habit worth stealing
Azure's Well-Architected Framework organises reliability around defining requirements from business need, designing for redundancy at every layer, and — the part Azure emphasises more than the others — designing a reliability testing strategy with Chaos Studio and ASR test failover as first-class practices.
The habit worth stealing regardless of which cloud you're on is Azure's health modelling: rather than monitoring resources individually, you define what "healthy" means for the application as a composite of its components, and build a single health signal from it. In practice this is a workbook or a composite alert that answers one question — is the service healthy, degraded, or down? — and it's what turns forty green tiles into an actionable page. AWS composite alarms and GCP's Service Monitoring do the same job under different names.
Practical starting checklist for any Azure subscription: enable diagnostic settings on everything into a workspace, set table plans deliberately, create one action group per on-call rota, add a Service Health alert, and schedule a quarterly ASR test failover with a calendar invite that has a named owner.
Sources: Azure Well-Architected — Reliability Azure Monitor Logs cost calculations
S3 · SECTION 4 · apply it — your context + hands-on10%
Your stack, on Azure
Migration path
- Prometheus → Azure Monitor managed Prometheus. Remote-write into an Azure Monitor workspace, or let the AKS add-on scrape your pods directly using standard
prometheus.ioannotations. Your vLLM/metricsendpoint needs no changes. - Grafana → Azure Monitor dashboards with Grafana first, since it's free and in-portal. Move to Azure Managed Grafana only when you need plugins or non-Azure data sources.
- Langfuse → Application Insights. Instrument with OpenTelemetry and point the exporter at App Insights; spans become
AppDependenciesandAppRequests, queryable with KQL alongside your logs. Watch adaptive sampling — turn it down before you trust request counts during an incident. - Alertmanager → alert rules + one action group. Build the action group first; it's the reusable piece.
The AKS GPU specifics
- Put GPU nodes in a separate node pool with taints, so system pods don't land on expensive hardware.
- Managed Prometheus has native AKS integration including control-plane metrics — genuinely useful, since API server latency is often the hidden cause of slow pod scheduling.
- The NVIDIA DCGM exporter publishes GPU utilisation, memory and temperature as Prometheus metrics; managed Prometheus scrapes it like anything else. This gives you the GPU layer your Grafana dashboards already expect.
- Set a deep readiness probe on the vLLM pod that runs a one-token generation, not a port check. This is the S1 shallow-health-check lesson applied where it bites hardest — a pod whose weights failed to load will pass a TCP probe forever.
GPU / GenAI What DR means here. Your weights sit in Blob Storage — choose GZRS so they're zone-redundant locally and geo-replicated to the paired region, which is the whole data half of your DR for a few dollars. Your container images go in Azure Container Registry with geo-replication enabled (a Premium SKU feature, and worth it). Then Site Recovery is not your tool: you're not replicating machines, you're redeploying a stateless workload. Your recovery is an AKS cluster in the second region — either pre-created and empty, or created from Terraform — plus a Front Door or Traffic Manager profile that can shift traffic. The binding constraint is the same as on AWS: GPU SKU quota and capacity in the failover region. Request the quota in advance, because quota requests are not instant and a quota denial during an incident is an unrecoverable RTO.
OPTIONAL HANDS-ON · S3 · free tier / B1s, budget alert first, full teardown
A workspace, a KQL alert, and the action group you'll reuse forever
- Budget alert first. Cost Management → Budgets → monthly budget of $5 with alerts at 50% and 90%. Then continue.
- Create a Log Analytics workspace in a region near you. Note the retention setting — leave it at the default 30 days; do not raise it in a lab.
- Create a B1s Linux VM (the smallest burstable SKU, free-tier eligible on new accounts) and enable diagnostic settings sending guest metrics and syslog into the workspace.
- Create an action group with your email as the only target. Name it something you'd actually reuse, like
oncall-primary. - Create a log alert rule using KQL. This is the artefact that is the mechanism:
// Alert when the VM stops reporting heartbeats — the // "missing data" case that silently kills naive alerts Heartbeat | where TimeGenerated > ago(10m) | summarize LastSeen = max(TimeGenerated) by Computer | where LastSeen < ago(5m) // evaluation frequency: 5 min · lookback: 10 min // action group: oncall-primary · severity: 2
- Stop the VM from the portal and wait. The alert should fire — you've just built a detector for the failure mode that a CPU-threshold alarm would never catch, because a dead machine reports no CPU at all.
- Add a Service Health alert on the subscription, notifying the same action group. It costs nothing and it's the single highest-value alert on any Azure subscription.
- Optional: set the syslog table's plan to Basic and observe the cost estimate change in the workspace's Usage and estimated costs blade.
TEARDOWN — in this order, and don't skip step 1
- Delete the Log Analytics workspace first. Ingestion bills continuously and a forgotten workspace collecting diagnostics is the most common Azure lab surprise. Note it enters soft-delete for 14 days — that's expected, and it stops billing.
- Delete the VM and its disk, NIC, public IP and NSG. Azure does not cascade-delete these; managed disks and public IPs bill on their own. Deleting the whole resource group is the reliable way.
- Delete the alert rules (log alerts bill per evaluation).
- Keep the action group and the budget — both are free and both are useful.
- If you experimented with Site Recovery: disable replication before deleting anything, or you'll leave orphaned replicated disks in the target region billing quietly. Also clean up any test-failover VMs.
S4 lands on Google Cloud, where SLOs and error budgets are not a pattern you assemble but API objects you create — this is the company that wrote the SRE source, and the product surface shows it.
You'll also do the burn-rate arithmetic properly, and read Google's own postmortem of the day their control plane crash-looped in every region at once.
S4 · SECTION 1 · why this session exists10%
The company that wrote the material, grading its own homework
Site Reliability Engineering is a Google invention. SLIs, SLOs, error budgets, toil budgets, blameless postmortems and burn-rate alerting all came out of running Search and Gmail at a scale where you cannot hire operators linearly with machines. That history shows up in the product: on Google Cloud an SLO is not a pattern you assemble from queries — it is an API resource with an error budget attached and burn-rate alerting built in.
It also shows up in how Google handles failure. On 12 June 2025 a policy record with blank fields propagated globally through Spanner in seconds and crash-looped the Service Control binary — the component that authorises every Google Cloud API call — in every region simultaneously. SRE triaged within two minutes, identified the cause within ten, and pushed a pre-built kill switch globally in about forty. Recovery in us-central1 took two hours forty minutes longer because every restarting instance stampeded the same infrastructure at once.
Read that timeline as a scorecard for this whole stage. Detection: excellent. Two minutes. Recovery mechanism: pre-built. The kill switch already existed, which is the difference between hours and days. Redundancy: irrelevant. Multi-region bought nothing, because the fault propagated through the very replication that makes multi-region work. And the thundering herd on restart is the same second-order failure that extended the AWS October 2025 outage — a pattern, not a coincidence.
S4 · SECTION 2 · core concepts50%
Cloud Monitoring, and the word the material gets wrong
⚠ BOOK IS STALE · start here, because the terminology changed
§14 of your GCP guide teaches "Monitoring Workspaces" as the central unit of organisation — one Workspace can monitor several projects, but only one Workspace can monitor a given project.
The construct is now called a metrics scope, and the constraint the material states has been removed: a project can belong to multiple metrics scopes. That change matters practically — it's what lets you build one org-wide infrastructure-health scope for SREs and separate per-team scopes over the same projects, which was impossible under Workspaces. Everything else the section says about dashboards, alerting policies, uptime checks and notification channels living in that scope is still correct.
The same section teaches Cloud Debugger in full, with feature lists and a worked scenario. Cloud Debugger was deprecated on 16 May 2022 and shut down on 31 May 2023. The open-source Snapshot Debugger that replaced it was itself archived on 7 September 2023 and receives no fixes or security patches. If a scenario in the material says "use Cloud Debugger", the modern answer is structured logging plus Cloud Trace, or Cloud Profiler for performance questions. There is no live production snapshot debugger on GCP today.
Also worth flagging: trace sinks were deprecated on 18 February 2026. Exporting trace data to BigQuery via sinks is replaced by the Observability Analytics page, which gives a SQL interface over both trace and log data.
Sources: Cloud Debugger deprecation — Google Cloud docs Google Cloud Observability release notes
PASS 1INTUITION
Google Cloud Observability (the suite formerly called Stackdriver) is the most opinionated of the three. Where AWS gives you a shelf of parts and Azure gives you two stores with a seam, Google gives you a fairly coherent pipeline: signals go in, a router decides where they land, and an SLO layer sits on top as a first-class thing.
The two ideas that are distinctively Google's, and worth learning even if you never deploy on GCP: the Log Router, which evaluates every single log entry against filters and sinks before anything is stored, and SLOs as objects, where the error budget and its burn rate are computed for you and alertable directly.
PASS 2MECHANISM
Cloud Monitoring collects metrics from Google Cloud services with no setup, from VMs via the Ops Agent (a single agent that handles both logs and metrics, replacing the older separate agents), and from anywhere via the API. Uptime checks probe endpoints from multiple global locations. Alerting policies fire on metric thresholds, uptime-check failures, log matches, or SLO burn rate.
Cloud Logging is where the Log Router lives, and its mechanics reward understanding because they're the cost lever:
Audit logs come in four types and the distinction is exam-relevant and operationally real: Admin Activity (configuration changes — always on, cannot be disabled), Data Access (reads and writes of user data — off by default for most services because the volume is enormous), System Event (Google-initiated actions like live migration — always on, doesn't count against ingestion quota), and Policy Denied (requests blocked by org policy or VPC Service Controls).
Log-based metrics turn a filter into a counter you can chart and alert on — the same bridge every cloud offers, and on GCP the standard answer to "alert me when this text appears".
PASS 3TRADE-OFFS & LIMITS
- Exclusion filters are the cost control, and they're destructive. An excluded log is not stored anywhere — it's gone. Exclude aggressively for volume, but route a sampled fraction to a cheap sink first so you retain the ability to investigate. Excluding 100% of a noisy category and then needing it during an incident is a common regret.
- Aggregated sinks at the organisation or folder level are the right pattern for security logging, and they're set up once. A central logging project receiving admin-activity logs from every project, with the security team having access there and nowhere else, is a genuinely clean design and one of GCP's better stories.
- Data Access logs will surprise your bill. They're off by default for good reason. Enable them selectively, per service, and expect a step change in volume when you do.
- Metrics scopes don't affect billing — a useful fact, because it means you can create scopes freely for organisational convenience without cost consequences.
SLOs as objects — the part Google does better than anyone
PASS 1INTUITION
On AWS and Azure you largely build SLOs out of parts. On GCP, Service Monitoring gives you a Service resource, and you attach ServiceLevelObjective resources to it. The platform then computes SLI compliance, remaining error budget and burn rate continuously, and you can alert on any of them.
Why this matters beyond convenience: when the error budget is a real object with a real API, it can appear in dashboards, in deployment gates, in release tooling. That's the difference between SLOs as a monitoring feature and SLOs as an operating model.
PASS 2MECHANISM
Two SLI shapes cover almost everything. Request-based SLIs are the good-events-over-valid-events ratio from S1. Windows-based SLIs ask "was this minute good?" and then measure the fraction of good minutes — useful when your service is a batch pipeline rather than a request handler.
Request-based SLIs come in two forms. A TimeSeriesRatio divides one counter by another. A distributionCut takes a latency histogram and counts what fell inside a range — which is exactly the bucket-ratio shape S1 argued for, expressed natively. Here's a real one against Prometheus metrics:
Note the metric type: prometheus.googleapis.com/.... Metrics your existing Prometheus scrapers produce become the raw material for a native GCP SLO with no translation layer. For your stack that's the shortest path from "I have a histogram" to "I have an error budget" on any cloud.
PASS 3TRADE-OFFS & LIMITS
- Rolling versus calendar windows behave very differently. A rolling 28-day window gives smooth continuous feedback; a calendar month forgives everything on the first. Rolling is better for engineering decisions, calendar is better for reporting to people who think in months. The API supports both — pick deliberately, and consider publishing both.
- Your SLI is only as good as your "valid events" denominator. Include health-check traffic and you dilute the signal until it means nothing. Include requests rejected for bad input and you're penalised for correctly rejecting garbage. Getting the denominator right is most of the work.
- The SLO object doesn't create the policy. Google's own guidance is that the error-budget policy — what stops when the budget is gone — is an agreement between teams, written down, with named signatories. No product ships that.
Burn-rate alerting, done properly
This is the technique S1 promised and the reason GCP's alerting is worth studying even if you deploy elsewhere. The problem it solves: a naive threshold alert either pages you constantly for blips, or notices a slow leak only after the budget is gone.
PASS 2MECHANISM
Burn rate is normalised so that 1.0 means "consuming budget at exactly the pace that exhausts it precisely at the end of the window". So a burn rate of 14.4 sustained for one hour consumes 1/14.4 of a 30-day window's worth of budget in that hour — which works out to 2%.
The two-window trick is the part people skip and then regret. Without the short window, an incident that ends at 09:00 keeps the alert firing until 10:00 because the one-hour average is still elevated. On-call engineers learn to ignore alerts that lie about whether the problem is current.
PASS 3TRADE-OFFS & LIMITS
- Low-traffic services break burn-rate alerting. At 100 requests an hour, three failures is a 3% error rate and a 30× burn rate. You'll page on noise. Options: lengthen the windows, set a minimum-traffic condition, or accept that a genuinely low-traffic service should have a looser SLO and be alerted on differently.
- Two tiers is usually enough. The canonical workbook ladder has three, but most teams find that fast-burn-pages plus slow-burn-tickets covers reality without an unmaintainable rule set.
- Burn-rate alerts replace threshold alerts; they don't supplement them. Running both means paging twice for the same incident, which is how alert fatigue starts.
Choosing the right observability tool — the material's scenarios, corrected
§14 closes with a useful seven-scenario table. Six of the seven still hold; one recommends a service that no longer exists. Here it is, current.
| The question you're actually asking | Reach for | Why, and what changed |
|---|---|---|
| Users say it's slow, but CPU and memory look fine | Cloud Trace | Only traces decompose one request's latency across services. Instrument with OpenTelemetry and send OTLP to telemetry.googleapis.com. |
| A bug that only reproduces in production, and logs lack detail | Structured logging + Cloud TraceThe material says Cloud Debugger — it was shut down on 31 May 2023. | Add structured log fields with the trace ID attached so logs and spans correlate, then redeploy. There is no live snapshot debugger on GCP any more. |
| A service burns more CPU than expected and keeps scaling up | Cloud Profiler | Continuous function-level CPU, heap, wall-time and contention profiling with a lightweight agent. Still current and still excellent. |
| Nobody notices unhandled exceptions until a customer complains | Error Reporting | Groups exceptions by stack trace, tracks frequency, and notifies on new or resurfacing errors. Extracts them from Cloud Logging automatically. |
| Regulations require seven years of logs, default retention is 30 days | Sink → Cloud Storage | Route via the Log Router to a bucket with lifecycle and retention policies. Cheapest durable option. A user-defined log bucket can also hold up to 3,650 days. |
| Alert whenever a specific message appears — "CUDA error", "payment failed" | Log-based metric + alerting policy | Define a counter from a log filter, then alert on it like any metric. The bridge between the log and metric signals. |
| Take automated action the moment a log event occurs | Sink → Pub/Sub → Cloud Run function | Real-time fan-out to your own code or to Slack/PagerDuty. The event-driven pattern from C6, applied to operations. |
| Is the service meeting its reliability promise, and can we ship this week? | Service Monitoring SLO + burn-rate alertNot in the material's table at all — the most important row. | The only tool here that answers a business question rather than a debugging one. |
DR on Google Cloud — strong primitives, assemble it yourself
PASS 2MECHANISM
GCP has no single DR orchestrator equivalent to ARC Region switch or Site Recovery. What it has instead is unusually clean primitives, plus one genuine architectural advantage.
- Backup and DR Service is the managed backup product, with backup vaults providing immutable, indelible copies that survive deletion of the source. It's integrated with Cloud Monitoring for metrics and Cloud Logging for job records, and — usefully — its Cloud Monitoring metrics are non-chargeable. This is distinct from ordinary persistent-disk snapshots, which are a mechanism rather than a backup product with policy and immutability.
- Regional resources by default. Regional MIGs and regional GKE clusters spread across zones without you asking. Remember the vocabulary trap: on GCP, regional means multi-zone; multi-region means across regions.
- Cloud Storage location types — regional, dual-region or multi-region — chosen at bucket creation. Dual-region with turbo replication gives an actual RPO target rather than best-effort.
- The architectural advantage: global anycast load balancing. A single global IP address fronts backends in many regions. Failover happens at the edge with no DNS record change and therefore no TTL caching in the path. This genuinely removes a term from the RTO arithmetic that both other clouds have to work around.
PASS 3TRADE-OFFS & LIMITS
- No orchestrator means no practice mode. ARC validates plans every thirty minutes; ASR offers non-disruptive test failover. On GCP the equivalent discipline is cultural — Google's own answer is DiRT exercises and game days. You must schedule them yourself, because no product will nag you.
- Global load balancing is also a global dependency. Same lesson as Azure Front Door. It is better than DNS failover and it is one more shared thing.
- Snapshots are not backups. Worth repeating on GCP specifically, because persistent-disk snapshots are so easy that teams stop there. They live in the same project under the same IAM as the thing they protect. A compromised project loses both.
⚠ WORTH KNOWING · the open-source path on GCP is the strongest of the three
Google Cloud Managed Service for Prometheus is built on Monarch, the same globally scalable time-series store Google uses to monitor itself. It's a drop-in replacement for a self-run Prometheus stack: managed collectors on GKE configured with lightweight custom resources, or self-deployed collection with remote-write. Two-year retention is included at no extra storage charge, and pricing is per sample ingested rather than per time series, which means horizontal pod autoscaling doesn't punish you with cardinality charges the way per-series billing does.
Crucially, PromQL queries work across both your Prometheus metrics and Cloud Monitoring's own system metrics — thousands of free GKE and infrastructure metrics queryable in the same expression as your vLLM histograms. Google has no first-party managed Grafana; the answer is that Cloud Monitoring dashboards speak PromQL natively, or you point your own Grafana at the Prometheus API.
Sources: Managed Service for Prometheus — Google Cloud docs Alerting on your burn rate
Decision tree · which burn-rate response does this signal deserve?
GUARD CLAUSES · follow "no ↓" until a "yes" exits right
You have an SLO and a burn rate. Now what?
The three-cloud view, anchored on Google Cloud
GCP's distinctive shapes are the SLO object, the Log Router, and global anycast entry. Read this table as "here is what Google gives you natively, and here is what the same thing costs you in effort elsewhere".
| GCP shape | Google Cloud | AWS | Azure |
|---|---|---|---|
| SLO as an API resource | Service Monitoring — Service + ServiceLevelObjectiveRequest-based or windows-based SLIs; error budget computed for you.The strongest native SLO story of the three, and it consumes Prometheus metrics directly. | CloudWatch Application Signals SLOsRequest- and period-based, with error-budget tracking; SLO recommendations added Mar 2026.Newer, and tied to Application Signals&#A93248; service discovery model. | Assemble from KQL + alert rulesNo single first-class SLO object.Workable, but the error budget is something you compute rather than something you query. |
| Burn-rate alerting | select_slo_burn_rate conditionMultiwindow burn-rate alerting is a built-in condition type on an alerting policy.Console will generate the fast/slow ladder for you. |
Alarms on the SLO&#A93248;s budget metricsApplication Signals exposes budget consumption you can alarm on.You still assemble the multiwindow logic yourself. | Scheduled KQL query alertsCompute the burn rate in the query.Most hand-rolled of the three; also the most flexible. |
| Managed Prometheus economics | Managed Service for PrometheusBuilt on Monarch; 24-month retention included; priced per sample ingested.Per-sample pricing means pod autoscaling doesn&#A93248;t punish you the way per-series billing does. | Amazon Managed Service for PrometheusRemote-write compatible, PromQL native.Bills four ways: samples ingested, stored, queried, plus collectors. | Azure Monitor managed PrometheusStores to an Azure Monitor workspace; deep AKS integration.18-month retention with no separate storage charge. |
| Log routing before storage | Log Router — sinks and exclusion filtersEvery entry evaluated before anything is stored; fan out to buckets, BigQuery, Pub/Sub, GCS.Excluded logs are discarded, not cheaper. Sample before excluding. | Subscription filters + Metric StreamsFan-out happens after ingestion into a log group.You pay to ingest first, then pay again to move it. | Data collection rules + table plansDCRs can filter and transform at the agent.Closest to GCP&#A93248;s model, and the right place to drop volume. |
| Multi-year log retention | User-defined bucket (up to 3,650 days) or sink B54B8; GCSLifecycle classes make archive nearly free.Admin Activity logs already live free in the immutable _Required bucket. |
Log group retention or export to S3 GlacierRetention set per log group.Default retention is never expire — the classic runaway bill. | Long-term retention on the table, or Auxiliary planUp to 12 years.Interactive vs long-term retention are priced very differently — ~$0.10 vs ~$0.02 per GB-month. |
| Synthetic / uptime probing | Uptime checksMulti-region probes, usable directly as an SLI source.Bills per check execution — delete them first when tearing down a lab. | CloudWatch Synthetics canariesScripted browser and API canaries.More capable, more expensive, and they run Lambda under the hood. | Standard tests in Application InsightsURL ping and multi-step availability tests.Classic availability tests were retired — use standard tests. |
| Continuous profiling | Cloud ProfilerFunction-level CPU, heap, wall-time and contention, always-on, low overhead.Genuinely differentiated — and the answer to the &#A93248;why is this scaling?&#A93248; question. | CodeGuru ProfilerSimilar idea, narrower language support.Overlaps awkwardly with Application Signals. | Application Insights ProfilerOn-demand rather than continuous for most tiers.Sampling-based; you enable it when you need it. |
| Exception grouping | Error ReportingGroups by stack trace, notifies on new or resurfacing errors, extracted from Cloud Logging automatically.No instrumentation needed if you log structured exceptions. | No first-party equivalentTeams use Sentry or build metric filters.A genuine gap in the AWS-native story. | Application Insights failures viewGroups exceptions by type and operation.Good, but tied to App Insights instrumentation. |
| Audit log model | Four types: Admin Activity, Data Access, System Event, Policy DeniedAdmin Activity and System Event always on and free.Data Access is off by default — enabling it is a real step-change in volume and cost. | CloudTrail management vs data eventsManagement events free for 90 days in Event history.Data events (S3 object-level, Lambda invokes) bill per event and add up fast. | Activity log + diagnostic settingsControl-plane events, 90 days free, exportable to a workspace.Activity log alerts are the mechanism for &#A93248;someone deleted a resource&#A93248;. |
| Cross-region traffic entry | Global External Application Load BalancerOne anycast IP, backends in many regions, failover at the edge.No DNS TTL in the failover path — a real architectural advantage. | Route 53 (DNS) or Global Accelerator (anycast)Two mechanisms, two failover profiles.DNS is cheaper; Global Accelerator is faster and removes TTL caching. | Traffic Manager (DNS) or Front Door (anycast)Front Door adds WAF and CDN.Front Door&#A93248;s Oct 2025 config incident is the cautionary tale for global front doors. |
S4 · SECTION 3 · reality check25%
Arithmetic, then the postmortem Google wrote about itself
WORKED EXAMPLE 1
A full burn-rate ladder for a Vertex AI inference endpoint
What this tells you to do. Ten bad rollouts a month exhausts the budget — so the budget is effectively a rollout-quality target. That's the error budget doing its job: it has converted "we should test models better before promoting them" from an opinion into a countable constraint with a deadline.
GPU / GenAI Notice also that the preemption number is small. One preemption costs about 1% of the month. That's the quantitative case for spot or preemptible GPUs in an inference fleet: if you can absorb roughly one preemption a week inside a 99.5% SLO — and this arithmetic says you can, with margin — then the cost saving is real and the reliability cost is budgeted rather than hoped-for. Change the SLO to 99.9% and the same preemptions consume 5% of the budget each, and spot stops being viable. The SLO decides your instance purchasing model. That's a genuinely useful thing to be able to say in a design review.
WORKED EXAMPLE 2
Log retention: what seven years of compliance actually costs
The pattern that generalises to all three clouds: route once, land in three places with three different economics. Hot and queryable for 30 days, cheap and searchable for a year, frozen and compliant for seven. Any design that keeps everything hot is overpaying by an order of magnitude; any design that freezes everything makes the next incident investigation impossible.
Three things that actually happened, and one source worth reading
POSTMORTEM 12 June 2025 — how a missing feature flag crashed every region at once
On 29 May, code adding extra quota-policy checks was deployed into Service Control. It had no error handling and, critically, no feature-flag protection — so it shipped globally in an active state, but stayed dormant because it needed policy data of a shape that didn't yet exist. Google's own report notes that had it been flag-protected, the issue would have been caught in staging.
On 12 June a policy change wrote unintended blank fields into the regional Spanner tables Service Control reads. Spanner did exactly what it's built to do and replicated the change globally within seconds. Every regional Service Control binary exercised the new code path, hit a null pointer, and entered a crash loop. External API requests across Google Cloud and Workspace returned 503s: Gmail, Drive, Meet, BigQuery, Vertex AI, plus Spotify, Discord, Snapchat and Cloudflare services that depend on Google infrastructure.
Response: triage within two minutes, root cause within ten, the pre-existing "red button" kill switch rolled out globally within about forty minutes. Then the long tail — us-central1 took roughly two hours forty minutes more, because every Service Control instance restarted simultaneously and overwhelmed the same Spanner infrastructure. Total roughly seven and a half hours from 10:51 to 18:18 PDT.
Five transferable lessons, and this is the single richest incident in the stage: (1) global config replication is a serial dependency wearing a parallel costume; (2) feature flags are a reliability control, not a product-management convenience; (3) a pre-built kill switch converts a multi-day incident into a forty-minute one; (4) restarts need backoff and jitter or recovery causes a second outage; (5) Google's own monitoring and communication tooling was impaired during the event, delaying customer updates — your observability must not share fate with what it observes.
Sources: Google Cloud Service Health incident report, 12 June 2025 ThousandEyes analysis The Register on the incident report
PRACTICE The SRE Workbook and 5 — the source for everything above
§2, Implementing SLOs, is the practical guide to choosing SLIs, setting targets and writing the error-budget policy. §5, Alerting on SLOs, is where the multiwindow multi-burn-rate ladder comes from — the 14.4× / 6× / 1× table above is lifted directly from it, and the section walks through why each simpler approach fails first.
Three ideas worth carrying into every design review, regardless of cloud:
- 100% is the wrong target for everything. Users cannot perceive the difference between 100% and 99.99% because their own network is less reliable than that. The last nine is spent on nobody.
- Toil should stay under 50% of an SRE's time. Above that, the answer is automation, not headcount — this is the economic argument that created the discipline.
- Postmortems must be blameless to be honest, and honest to be useful. The output of a good one is always a change to redundancy, detection or recovery — the loop from Tab 0.
Free online: sre.google/workbook/implementing-slos sre.google/workbook/alerting-on-slos
GUIDANCE Google's Well-Architected Framework, reliability pillar — and how it differs in emphasis
§15 of your GCP guide covers the five pillars: operational excellence, security, reliability, performance efficiency and cost optimisation. The framing is the same as AWS's and Azure's, but Google's reliability pillar leans harder on two things the others treat as secondary.
First, graceful degradation as a design requirement rather than a nice-to-have — explicitly designing what your service does when a dependency is unavailable, including load shedding and returning partial results. Second, observability as a reliability control rather than an operational afterthought: the framework treats your ability to detect and diagnose as part of the reliability budget itself, which is precisely the redundancy-detection-recovery loop from Tab 0 written as guidance.
The release-management material in the same section — canary, blue/green, rolling updates via Cloud Deploy, with rollback mechanisms — connects directly to error budgets. Progressive delivery is how you spend a budget deliberately rather than accidentally: a canary that fails costs you a fraction of a percent, where the same bad code shipped fleet-wide costs the month.
Source: Google Cloud Well-Architected Framework — Reliability pillar
S4 · SECTION 4 · apply it — your context + hands-on10%
Your stack, on GCP — the shortest migration of the three
You've already deployed to Vertex AI with T4s, autoscaling 1–3 and spot, and run a vLLM stack on an L4. This is the cloud where your existing observability transfers with the least translation, because the SLO layer consumes Prometheus metrics natively.
Migration path
- Prometheus → Managed Service for Prometheus. On GKE, deploy managed collection and annotate your vLLM pods; the collectors handle scraping, sharding and scaling. Your recording rules and alert rules port across as
Rulescustom resources. - Grafana → keep it, or use Cloud Monitoring. No first-party managed Grafana, but your existing Grafana points at the Prometheus API unchanged, and Cloud Monitoring dashboards accept PromQL directly.
- Langfuse → Cloud Trace via OTLP. Export OpenTelemetry spans to
telemetry.googleapis.com. Attribute limits are generous since the OTLP migration — 64 KiB values and 1,024 attributes per span, which matters when you're attaching prompts to spans. Keep Langfuse for evals and prompt management. - Your p95 dashboards → a real SLO. This is the step that's uniquely easy here: your existing TTFT histogram becomes a
distributionCutSLI in an SLO object, with burn-rate alerting on top. Nothing to rewrite.
DR for a stateless inference service on GCP
- Weights: a dual-region Cloud Storage bucket with turbo replication if you want a stated RPO. Cheap, and it's your entire data story.
- Images: Artifact Registry, with a repository in each region you might recover into.
- Compute: a regional GKE cluster is already multi-zone. For multi-region, the honest answer is a second cluster plus Terraform, because there's no orchestrator to lean on.
- Traffic: the global external Application Load Balancer with backends in both regions. One anycast IP, no DNS TTL in the failover path — the cleanest cross-region failover of the three clouds.
- Capacity: the binding constraint again. GPU quota is per-region and per-SKU. Request it in the DR region now; a quota request during an incident is an RTO you cannot meet.
GPU / GenAI One more GCP-specific note worth having. For Vertex AI endpoints the platform manages the serving infrastructure, so your DR conversation shifts: you're not recovering VMs, you're re-deploying a model to an endpoint in another region and shifting traffic. That makes RTO largely a function of model upload and endpoint provisioning time — measure it once, in a real region, and write the number down. It is almost always slower than people guess, and it is the single number your DR plan depends on.
OPTIONAL HANDS-ON · S4 · free tier / e2-micro, budget alert first, full teardown
An uptime check, an SLO object, and a burn-rate alert that pages you
This is the capstone lab for the stage: by the end you will have a real error budget burning in a real console, which is the thing all four sessions have been building toward.
- Budget alert first. Billing → Budgets & alerts → monthly budget of $5, thresholds at 50% and 90%. Non-negotiable.
- Create an e2-micro VM (free-tier eligible in several US regions) running any trivial HTTP server, with a firewall rule allowing port 80 from the health-check ranges only.
- Install the Ops Agent — one agent for both logs and metrics, and note it replaces the legacy separate agents the material may reference.
- Create an uptime check against the VM from multiple global regions. This is your SLI source and it costs a trivial amount per check.
- Create a Service in Service Monitoring, then attach an SLO: 99% availability on a rolling 7-day window. Use 7 days rather than 28 in a lab so you can actually see the budget move.
- Create a burn-rate alerting policy on that SLO. The console builds this for you, but understand what it's writing:
# burn-rate condition, conceptually select_slo_burn_rate( "projects/PROJECT/services/SVC/serviceLevelObjectives/SLO", lookBackDuration: "3600s" ) threshold: 14.4 # fast burn → page duration: "300s" # the short-window confirmation
- Break it. Stop the VM. Watch the uptime check fail, the SLI drop, the error budget start draining, and the burn-rate alert fire. Restart it and watch the budget stop draining but not refill — that asymmetry is the whole point of a budget.
- Optional: create a log-based metric counting a string you write into the log, then chart it. Five minutes, and it makes the log-to-metric bridge concrete.
TEARDOWN — the safest route is to delete the whole project
- Delete the uptime checks first — they bill per check execution and run continuously whether or not the target exists.
- Delete the alerting policy and the SLO, then the Service.
- Delete the VM, and check for an orphaned persistent disk and any static external IP — an unattached static IP bills more than an attached one.
- Delete any log sinks you created, and any user-defined log buckets (they retain and bill independently of
_Default). - Cleanest of all: IAM & Admin → Settings → Shut Down the whole project. This is the reliable way to guarantee nothing is left running, and it's what the official codelabs recommend.
- If you enabled Data Access audit logs while exploring, turn them off — the volume increase is significant and permanent until you do.
You can now do the six things on Tab 0. The test isn't recalling service names — it's being handed "we need 99.95% and an RPO of five minutes" and answering with a tier, a topology, an SLO, and roughly what it costs.
The one habit to carry forward: schedule the rehearsal. Every plan in this stage degrades into a wish the moment it stops being tested, and no product will remind you.