Automation: Infrastructure as Code & CI/CD

1Intuition

Imperative is turn-by-turn directions. Declarative is a destination address.

Directions — "left at the lights, third right, park behind the blue van" — work perfectly once, from one starting point. Change any assumption and they break silently: the van has moved, so you park somewhere wrong and never notice. Worse, if you are already halfway there, following the directions from the top takes you somewhere absurd.

An address works from anywhere. It survives roadworks. And critically — you can hand it to someone else and they will end up in the same place, which is exactly what "reproducible" means.

You already have both in your own toolkit. Your Vertex AI deploy script — build image, push image, create endpoint — is directions. Your Kubernetes and KServe manifests are addresses. You wrote replicas: 3, not "start two more pods"; the control loop worked out the rest. Terraform is that same instinct, applied to cloud resources instead of pods.

2Mechanism

A declarative tool runs a reconciliation loop: it reads what you asked for, reads what exists, computes the difference, and executes only the difference. The loop is the whole trick.

resource "google_container_node_pool" "gpu" {
  name       = "l4-inference"
  node_count = 2              # the WHAT. Never "add one node".
  node_config {
    machine_type = "g2-standard-8"
    guest_accelerator { type = "nvidia-l4", count = 1 }
  }
}

Nothing in that block says create. It says exists, with these properties. Run it against an empty project and it creates. Run it against a pool that already has two nodes and it does nothing. Change 2 to 4 and it adds two. Same file, three different behaviours, because the file describes an end state rather than an action.

3Trade-offs, limits, and where it breaks

• Declarative cannot express time. "Create the cluster, wait for the operator to become healthy, then apply the CRDs" is a sequence, not a state. Tools bolt on escape hatches — provisioner, local-exec, null_resource, CloudFormation custom resources — and every one of them is a small imperative script hiding inside your declarative config. They are not idempotent, they are not planned, and they are the usual cause of "it worked the first time."
• Ordering is inferred, not written. The tool builds a dependency graph from references. If resource B needs A but never references A, the tool may create them in parallel and B fails. You then reach for depends_on, which is a manual patch to an inferred graph — use it, but treat each one as a smell.
• Idempotency is a property of the provider, not a gift. The tool promises to compute the right delta; the underlying API has to behave when it applies it. Some APIs genuinely are one-shot — uploading a model artifact, running a database migration — and no declarative wrapper makes them replayable.
• Coverage lags the cloud. A service launched last week may have no resource type. You end up with a hybrid: mostly declarative, plus a thin imperative edge. That is acceptable. Pretending the edge does not exist is not.
• Replace is the sharp edge. Some properties are immutable in the API. Changing them means destroy-then-create, and the plan will tell you — in small text, next to a resource holding your data. This is why nobody applies without reading the plan.

1Intuition

State is the receipt. You told the shop what you wanted; the receipt records what you actually walked out with and its serial numbers. Without the receipt, you cannot return anything, because you cannot prove which of the identical items on the shelf is yours.

Concretely: you write "a bucket named acme-weights." The cloud creates it and hands back an ID. Next run, the tool needs to know whether that bucket is the one it made (so a name change means rename) or someone else's (so a name change means create a new one and leave theirs alone). Only the receipt can tell the difference.

2Mechanism

Once you have state, you have three pictures of your infrastructure, not two — and every operation in IaC is a comparison between two of them. Internalise this triangle and drift, plan, and import all stop being mysterious.

3Trade-offs, limits, and where it breaks

• State contains secrets in plaintext. A generated database password, a private key, a service account key — if the API returns it, it lands in state. State is therefore a secret. Encrypt the bucket, restrict read access as tightly as you restrict production, and never commit it to Git. OpenTofu additionally ships client-side state encryption (v1.7), which Terraform's open CLI does not have.
• State is a shared mutable file, so it needs locking. Two concurrent applies against one state file corrupt it. For years the S3 backend needed a separate DynamoDB table for locks. Terraform 1.10 added use_lockfile using S3 conditional writes, and 1.11 made it GA and deprecated the dynamodb_table argument. Most tutorials you will find still provision the lock table — you can delete it.
• State size is blast radius. One state file for the whole company means every apply risks everything and every plan takes ten minutes. Split by lifecycle and ownership: network in one, the GKE cluster in another, per-team workloads in their own. Cross-state reads via data sources or remote state outputs.
• Losing state is recoverable but grim. The resources still exist; the tool has simply forgotten it owns them. Recovery means importing each one back. Versioned buckets make this a rollback instead of an archaeology project — turn on versioning before you need it.
• Managed state is a genuine advantage of native tools. CloudFormation, ARM and Infrastructure Manager hold state for you: no bucket to bootstrap, no lock table, no chicken-and-egg. You trade control and portability for that. This is a real point in the native-vs-Terraform decision, not a detail.

1Intuition

Drift is an open window; the tool is a thermostat. You set 21°C — that is your config. The thermostat holds it. Someone opens a window. The room is now 17°C: reality has drifted from desired state. A good thermostat notices and acts. A bad one keeps insisting the setting is 21 while everyone freezes.

plan is asking the thermostat what it is about to do, before it does it. "I am going to run the heating for 40 minutes." You get to say "actually, close the window instead."

Idempotency is the promise that pressing the button twice does not make the room 42°C. Apply the same config ten times and you get one bucket, not ten.

2Mechanism

A plan is a diff with four verdicts per resource, and the symbols are worth memorising because every tool borrows them:

  + create            # resource is in config, not in reality
  ~ update in-place   # same resource, mutable field changed
-/+ replace           # immutable field changed → destroy THEN create
  - destroy           # in state, removed from config

Plan: 2 to add, 1 to change, 0 to destroy.

The line to fear is -/+ replace. It appears when you change a property the cloud API cannot mutate — a subnet CIDR, an instance boot disk image, a database engine version on some services. The plan is telling you, in the same neutral tone it uses for everything else, that it is about to delete a thing and build a new one. On a stateless inference pod that is fine. On the bucket holding your model weights it is a Tuesday you will remember.

3Trade-offs, limits, and where it breaks

• Not all drift is bad. If a cluster autoscaler changed node_count from 2 to 7 under load, reverting that is actively harmful — your next apply would scale production down mid-traffic. The fix is to tell the tool that field is not yours: lifecycle { ignore_changes = [node_count] }. Deciding which fields the platform owns and which you own is a real design decision, not a config detail.
• Detection is not continuous by default. CloudFormation drift detection is an on-demand or scheduled check, not a live watch. Terraform only sees drift when someone runs a plan. If nobody plans for six weeks, you have six weeks of invisible divergence. Mature teams run a scheduled plan on a cron and alert on any non-empty diff.
• Remediation is a choice with two right answers. Re-apply to revert the change (the config was right, the human was wrong), or update the config to adopt it (the human was right, the config was stale). Choosing "revert" reflexively is how you undo somebody's 3 a.m. incident fix at 9 a.m. on a Monday.
• Prevention beats detection. The durable fix is IAM: production write permissions belong to the pipeline's identity, and humans get read-only plus a break-glass role that fires an alert when assumed. This is C2 material doing operational work.
• Plans go stale. A plan computed at 09:00 and applied at 14:00 describes a world that may no longer exist. Good pipelines save the plan as an artifact and apply that exact plan, refusing if reality moved. Ignore this and you get the AWS outage in the Reality Check below — literally a stale plan overwriting a newer one.

1Intuition

You are replacing the engines on a plane that is currently flying with passengers on board.

• Rolling — swap one engine at a time. Cheap, no second plane needed, but for a while you are flying on a mix of old and new engines, and going back means swapping them all again, slowly.
• Blue-green — build an entire second plane, move everyone across at once. Expensive (two planes) but reverting is instant: move everyone back.
• Canary — move ten passengers to the new plane and watch them for twenty minutes before moving anyone else. Slowest, and the only one that finds problems that only appear with real passengers.

2Mechanism

trafficSplit:
  - version: gemma-2b-v3   # incumbent
    weight: 90
  - version: gemma-2b-v4   # canary
    weight: 10
  abortOn: { sli: p95_latency_ms, threshold: 850, window: 15m }

3Trade-offs, limits, and where it breaks

• Canary without a threshold is theatre. If nobody has agreed the abort condition in advance, the rollout becomes a group discussion at 2 a.m. and the change goes out anyway.
• Canary can be blind by construction. A 1% slice is too small to move a global error rate. You must measure the canary's own SLI, not the blended one. S3 has a calculator that makes this uncomfortably concrete.
• Blue-green doubles cost, and on GPU that is not a rounding error. Two full L4 fleets for an hour is a real invoice. Canary is usually the right answer for accelerated workloads for exactly this reason.

1Intuition

ARM JSON is the machine format. Bicep is the human format. The relationship is TypeScript to JavaScript: you write the pleasant one, a compiler emits the one the runtime actually consumes, and you can always read the output when you need to.

The difference matters more than it sounds, because ARM JSON is genuinely hostile to write by hand. Expressions live inside strings — "[concat(parameters('prefix'), '-vm')]" — so you are writing a programming language inside a data format with no syntax highlighting to help you. A modest VM deployment runs to a few hundred lines of JSON with quoting rules that punish typos.

Bicep removes the quoting, adds real types, and produces a file you can review in a pull request without losing the will to live. Same deployment, roughly a third of the lines.

2Mechanism

The template has the same skeleton in both syntaxes: parameters (inputs, typed, optionally with allowed values), variables (computed locals), resources (the declarations), outputs (values other templates or pipelines consume), plus modules in Bicep for composition.

// Bicep — dependency inferred from the symbolic reference
param location string = resourceGroup.location
resource plan 'Microsoft.Web/serverfarms@2023-01-01' = {
  name: 'asp-inference'
  location: location
  sku: { name: 'F1' }        // free tier
}
resource site 'Microsoft.Web/sites@2023-01-01' = {
  name: 'app-inference-01'
  location: location
  properties: { serverFarmId: plan.id }   // ← implicit dependsOn
}

Note plan.id. In Bicep, referring to another resource by its symbolic name creates the dependency automatically — the same graph inference Terraform does. The ARM JSON equivalent requires an explicit "dependsOn" array and a resourceId string expression, which is exactly the kind of hand-maintained bookkeeping that rots.

Dry run: what-if

Azure's equivalent of plan and change sets. It reports Create, Modify, Delete, Deploy, Ignore, NoChange per resource, with a property-level diff for modifications. It is available on the CLI (az deployment group what-if), in the azure/bicep-deploy GitHub Action, and in the BicepDeploy@0 Azure Pipelines task.

Resource and property changes are indicated with these symbols:
  + Create
  ~ Modify
  - Delete

~ Microsoft.Web/sites/app-inference-01
  ~ properties.siteConfig.alwaysOn: false => true

Resource changes: 1 to modify.

Post this into the pull request. That single habit — reviewer approves the what-if output, not the Bicep diff — is the highest-value practice in this session, and it is what Microsoft's own reference implementation does.

3Trade-offs, limits, and where it breaks

• what-if is less reliable than terraform plan, and you must know this. It works by asking the ARM API to evaluate the change, and coverage varies by resource provider. Some properties report as changing when they will not (noisy false positives on defaults the provider fills in); occasionally something genuinely changes and is not reported. Treat it as a strong signal, not a contract. Terraform's plan, backed by a state file it fully controls, is more precise here — a real point for Terraform on Azure.
• Classic deployments never delete. Default mode is Incremental: resources in the template are created or updated, and anything not mentioned is left alone. Complete mode does delete unmentioned resources in the target resource group — which is a blunt and frankly dangerous instrument, because it acts on the whole group rather than on what your template owns. Deployment Stacks exist precisely to give you the sharp version of this.
• API versions are pinned per resource and they age. @2023-01-01 in the snippet above is part of the resource type. Templates written three years ago pin old API versions and silently miss newer properties. There is no global upgrade; you bump them resource by resource.
• Bicep is Azure-only, by design. That is not a criticism — it is the trade. You get day-one coverage of every new Azure resource because Bicep targets the same ARM schema, and you get nothing at all for AWS or GCP.
• Loops and conditions exist but are constrained. Bicep has for loops, if conditions, and functions, but it is not a general-purpose language. Coming from CDK this feels limiting; coming from ARM JSON it feels like liberation. Both reactions are correct.

1Intuition

A classic ARM deployment is a shopping trip: you hand over a list, you get those items, and nobody remembers the trip afterwards. Cross something off next week's list and last week's purchase does not vanish from your kitchen.

A deployment stack is a subscription box. The provider knows exactly what is in your box. Remove an item from your preferences and next month's box does not contain it — and, if you have asked them to, they will come and take the old one away. They will also stop you rummaging in the box and swapping things yourself.

That is state's job — knowing what you own — delivered as a platform feature instead of a file you have to store and protect.

2Mechanism

az stack group create \
  --name inference-lab --resource-group rg-inference \
  --template-file main.bicep --parameters main.bicepparam \
  --action-on-unmanage deleteAll \
  --deny-settings-mode denyWriteAndDelete

3Trade-offs, limits, and where it breaks

• Deny settings can lock out things you did not expect. denyWriteAndDelete blocks writes from everyone, including automation that legitimately needs to touch a resource — an autoscaler adjusting capacity, a certificate renewal job, an operator writing status. There is an exclusion list for principals and actions; you will need it, and you will discover which entries you need by breaking something first. Start with denyDelete, which is safer and still prevents the worst outcome.
• detach is the humane default while adopting stacks. Pointing a stack with deleteAll at an existing production resource group and getting the template subtly wrong is a bad afternoon. Adopt with detach, verify the stack's managed-resources list matches your expectation, then tighten.
• Stacks are scoped. Resource group, subscription, or management group. Cross-scope architectures need more than one stack and a deliberate decision about which owns what — the same "how do I split state" question Terraform users argue about, wearing a different hat.
• Still no plan-quality diff. Stacks fix ownership; they do not fix what-if fidelity. You still get a good-but-imperfect preview.
• Tooling has caught up recently and unevenly. The azure/bicep-deploy action supports stacks first-class, as does BicepDeploy@0 in Azure Pipelines. Older material — including the material — predates all of it, so most tutorials you find will show plain deployments.

1Intuition

Think of it as three ways to run the same engine, differing only in who holds the keys and how often it runs.

• Terraform yourself — you own the binary, the state bucket, the CI wiring. Maximum control and maximum portability, because nothing about it is GCP-specific.
• Infra Manager — Google runs the same binary and holds the state. You give up some control and delete a whole category of chores: no bucket to bootstrap, no locking to configure, no state to encrypt and back up.
• Config Connector — a fundamentally different rhythm. Not "run a tool when I say" but "a controller watches, continuously, forever." Cloud resources become objects your existing GitOps agent reconciles alongside your workloads.

This third one should feel familiar. You already write declarative manifests that a controller reconciles — that is what your KServe deployments are. Config Connector extends that loop past the cluster boundary so a Cloud Storage bucket and an inference service are managed by the same agent, in the same repository, with the same review process.

2Mechanism

The difference that matters is reconciliation cadence, and it changes what drift even means.

apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket            # a GCS bucket, as a K8s CRD
metadata:
  name: gemma-weights-prod
spec:
  location: US-CENTRAL1
  uniformBucketLevelAccess: true
  # kubectl apply this. A controller reconciles it. Forever.

Same file shape as your KServe InferenceService. Same kubectl apply. Same controller pattern. The only new thing is that the object on the other end lives outside the cluster.

3Trade-offs, limits, and where it breaks

• Infra Manager is Terraform, so it inherits Terraform's licence question. You are consuming it as a managed service rather than distributing it, so BUSL is not a practical constraint here — but note the asymmetry: if you later move that configuration off GCP to run yourself, the licence question arrives with it. This is one of the few places where "which binary" and "which cloud" are genuinely coupled.
• Config Connector fights anything else that writes. Two controllers with opinions about one resource is a loop that never settles. Pick one owner per resource and be strict about it. This is also why "Config Connector for everything" is usually wrong — the cluster itself has to be created by something else.
• Continuous reconciliation reverts your incident fix. The S1 drift story ends differently here: your 3 a.m. console change is undone within minutes, possibly mid-incident, by a controller nobody thought about. That is either the best or the worst property of this model depending on whether you knew it was running.
• Terraform on GCP has the same state chores as anywhere. A GCS backend with versioning and object-level locking, split by lifecycle, encrypted, access-restricted. Infra Manager exists to delete that list.
• Migration off Deployment Manager was a one-way door. Google shipped DM Convert to translate configurations to Terraform, with terraform import for adopting resources without redeploying. If you meet an estate still on DM in 2026, it is already broken, not about to break.

1Intuition

You know the C7 definitions already: an SLI is a measured signal, an SLO is the target for it, an SLA is the contractual version with penalties, and the error budget is the unreliability the SLO permits — 99.9% availability over 30 days allows roughly 43 minutes of failure.

What this session adds is one move: stop treating the error budget as a report and start treating it as a spending account with a card attached.

Budget remaining? Ship. That is what it is for — an unspent error budget means you are over-investing in stability and under-investing in features. Budget exhausted? The pipeline stops accepting risky changes until reliability work restores it. No negotiation, no escalation, no judgement call at 4 p.m. on a Friday. It is the same rule for everyone, and it converts an argument about "are we moving too fast" into arithmetic.

Google's other two principles support this. Toil — manual, repetitive, automatable work with no lasting value — is capped at under 50% of an SRE's time, which is a budget for building automation rather than a hope. Blameless postmortems mean the goal is learning, not fault, because a team that fears blame hides the near-misses you most need to see.

2Mechanism

Here is the wiring. Every box is a service the GCP section already describes — the contribution of this stage is connecting them into a loop that closes without a human in it.

3Trade-offs, limits, and where it breaks

• The blended-SLI trap is the big one, and it is counter-intuitive. A small canary cannot move a global error rate enough to trip a global alert. You can run a completely broken model version at 1% and watch the SLO dashboard stay green for the entire bake window. The calculator below makes this concrete; internalise it before designing your first rollout.
• Bake windows fight statistics. A 5% canary on a low-traffic service may not see enough requests in fifteen minutes to distinguish a real regression from noise. Either lengthen the window, widen the slice, or accept that you are not measuring anything. Compute the request count before choosing the numbers.
• Automatic rollback needs a clean rollback path. Stateless services roll back trivially. Anything that has written a schema change, migrated data, or emitted events consumed by others does not. Expand/contract migrations are the prerequisite, not an optimisation.
• Quality regressions do not show up in latency or errors. This is the GenAI-specific failure. A new adapter that responds fast, returns HTTP 200 every time, and produces worse answers passes every conventional SLI. You need a quality SLI — an eval score on sampled traffic, a refusal rate, a response-length distribution — or the canary is checking that the server is up, not that the model is good.
• Error-budget policy is organisational, not technical. The hardest part is not the query; it is the prior agreement that an exhausted budget genuinely stops feature work. Without that agreement it is a dashboard everyone glances at and overrides.