A smoke detector is not a smoke measurement device. It is a chain: a sensor that produces evidence (particles per m³), a known uncertainty (it goes off when you toast bread), a threshold committed to in advance, and an action that follows automatically (a very loud noise). Nobody wants a smoke detector with a nice display and no buzzer. That is what a dashboard without thresholds is.

Evaluation is the buzzer, not the display.

Four objects, chained. Each has a distinct owner and a distinct failure mode.

The operating model, in one line: primitives feed evaluators → evaluators produce metrics → metrics are compared to thresholds → thresholds trigger actions.

Thresholds are product policy, not statistics. You set them from historical baselines, stakeholder risk tolerance, business SLOs, and legal constraints — then you revise them as evidence arrives. Getting them perfect on day one is not the goal; making the decision boundary explicit, reviewable and adjustable is.

Separate load-bearing from diagnostic metrics. Load-bearing metrics directly cause a ship/block/rollback. Diagnostic metrics explain why a load-bearing metric moved. A useful rule: two or three load-bearing metrics per production axis, everything else is debugging. When every metric is on the dashboard and none is tied to an action, the dashboard becomes noise and the team stops looking.

Where this breaks: thresholds set on aggregates hide concentrated failures. Overall violation rate 0.08% looks great until you notice all of it lives in the compliance-flag slice, which is 2% of traffic and 100% of your regulatory exposure. Gates must be slice-scoped or they are decorative.

Classical supervised evaluation assumes a stable map: one input, one correct output, compare, score. That works for a spam classifier. It breaks for a system whose job is to write a paragraph, because a paragraph has no single right answer, and it breaks again because the same input can produce different paragraphs on Tuesday.

Versioning gives you attribution hypotheses, not causal proof. Knowing the retrieval config changed on the same day accuracy dropped is a lead, not a verdict — you still need a targeted rerun or ablation to confirm it. Teams that skip this step spend weeks “fixing” prompts to compensate for an index regression.

A restaurant that serves brilliant food in ninety minutes at £400 a head, occasionally with a stone in it, is not “working” because the food is brilliant. Quality is one axis of four.

Then map product goals onto measurable signals. “Don't claim a refund is issued unless verified” becomes unsupported_action_claim_rate with scope slice=refund and threshold = 0.

Trade-offs are not the problem; implicit trade-offs are. Longer prompts and deeper retrieval buy quality with cost and latency. Tighter safety policy buys risk reduction with over-refusal on benign requests. Restricting tool calls buys latency with correctness. Decide the exchange rate before the release meeting, or the release meeting becomes an argument about vibes.

Risk tiers change the shape of the threshold. Two kinds:

A flight recorder doesn't store the pilot's thoughts. It stores the observable event sequence, precisely enough to reconstruct what happened. That's the standard here: log enough to replay and attribute, not enough to violate anyone's privacy.

Two layers. System primitives are the parts of the product that get received, used, produced or recorded. Evaluation constructs turn those into decisions.

The minimum instrumentation boundary is the floor below which you can still make charts but can no longer run an evaluation programme: you will not be able to reproduce a failure, compare fairly, localise a regression, or turn an incident into a test.

Evaluator versioning is the one people forget. If the judge model, judge prompt, rubric or scoring code changes, scores move even when the system didn't. A helpfulness score can rise because the assistant improved, or because the judge got lenient. Without evaluator versions in the run record you cannot tell those apart — and the trend line looks continuous either way.

Privacy shapes the log. Store the observable event sequence, not hidden model reasoning. To evaluate the refund workflow you need: was the tool called, with what arguments, did it succeed, did the reply make a verified claim. That's enough. Chain-of-thought storage buys little and costs a lot in retention and exposure.

A driving test isn't a random sample of roads. It deliberately contains a hill start, a three-point turn and an emergency stop, because those are where people fail. A good eval suite is designed the same way: not representative on average, but complete across the situations that matter.

Then split suites by purpose, because a suite you iterate against cannot also be the suite you gate on:

Synthetic data's self-reinforcing trap. If the same model family generates your cases and judges the responses, the generator produces cases the evaluator finds natural, and the system looks better than it is. Guardrails: keep a real-data core, sample synthetic cases from real traffic patterns, label synthetic cases separately, and report real and synthetic subsets as separate numbers.

Gold sets leak. Iterate against the same reference cases for three months and your prompt engineering has quietly optimised against those examples without improving general behaviour. This is the same phenomenon as benchmark contamination, applied to your own suite — S2 goes deeper.

Ownership or rot. A suite with no owner goes stale in a quarter: deprecated policies stay in, new intents never arrive, and the gate slowly stops measuring the product.

Four families of measuring instrument, in increasing order of cost and decreasing order of trustworthiness-per-dollar. The rule is simple: use the cheapest evaluator that reliably detects this failure mode, and escalate only when it can't.

Weigh yourself on a bathroom scale five times and you get five numbers. Nobody concludes they gained 300 g between readings. But teams do exactly that with eval scores — 8.10 → 8.18 becomes “the new prompt is better” and ships.

A regression is not “the metric went down”. It is four questions, all of which must be checked: is the change bigger than run-to-run variation? does it exceed the declared MMC? does it concentrate in a critical slice? does it cross a gate threshold?

Variance you can't remove, measure separately. Different sources need different fixes. Reply quality moving because the model sampled differently → decoding policy or a stronger prompt contract. Quality moving because a tool timed out → a reliability problem wearing a quality costume. Scores moving because the judge prompt changed → evaluator drift. Track tool failure rate, judge-score variance, fallback rate and timeout rate as their own signals, or all three failures look identical on the quality chart.

Controlling drift costs discipline, not money. Keep a small calibration set. When the judge changes, run old and new judge on the same calibration set before treating the new scores as comparable. For humans, run periodic calibration rounds and double-score a subset.

Offline evaluation is the pre-flight checklist. Canary gates are the first ten minutes after takeoff, watching the instruments. Online evaluation is the whole rest of the flight. All three run on the same aircraft, and findings from the last one rewrite the first one's checklist.

Gates come in three strengths. Each combines a metric, a scope, a threshold and an action:

Offline can't see production, and that's structural. Real users invent new intents, traffic mixes shift, tools degrade, policies change, indexes refresh, external services time out. Online evaluation has to catch two very different shapes: acute incidents (a visible spike — tool outage, violation jump) and slow-burn regressions (helpfulness down 2%/week; latency creeping as a backend degrades; refusals drifting up after a policy edit). Slow burns evade simple threshold alerts because no single day looks bad. You need trend monitoring over longer windows to see them at all.

The drift usually isn't the model. A new product tier creates a slice that fails badly while overall accuracy holds. A policy update changes what the right answer is. A tool schema change produces failures that look exactly like hallucinations. In all three cases the correct response is to update the suite and the slices — not to patch the prompt.

Imagine reading a sentence aloud with one word covered, and betting on what's underneath. Perplexity is the average number of equally-plausible words you were choosing between at each step. Perplexity 2 means you were basically flipping a coin. Perplexity 50 means you were as lost as if picking from a 50-sided die.

Now the crucial part: this measures how surprised the model was by text. It says nothing about whether the model will follow your JSON schema, refuse an unsafe request, or refrain from inventing a revenue figure. A model can be beautifully unsurprised by financial prose and still confidently fabricate numbers in it.

Training loss for a language model is cross-entropy: the average negative log-probability the model assigned to the tokens that actually came next. Perplexity is just that number exponentiated, which puts it back on a “number of choices” scale humans can reason about.

Two consequences fall straight out of that formula, and both bite people:

• Perplexity is tokenizer-dependent. It is per-token, so two models with different tokenizers are not comparable on it. A tokenizer that splits words more finely gets a flattering number for free.
• Perplexity is corpus-dependent. “Perplexity 8” is meaningless without naming the held-out set. On your own domain corpus it is a genuinely useful relative signal; across papers it is nearly noise.

So the training-time practice is: treat loss monitoring as necessary but incomplete, and add cheap behavioural smoke tests the moment the model can produce coherent text — a handful of routing-style classification prompts, a structured-output adherence check, a couple of refusal-correctness probes. Those catch things loss curves are structurally blind to.

Checkpoint selection is where this gets expensive. The lowest-validation-loss checkpoint is frequently not the best product checkpoint. Loss keeps improving while instruction-following degrades, or while the model gets more verbose (which costs you money at inference forever). Select on a small behavioural scorecard, with validation loss as a sanity-check rather than the objective.

What loss is good for: detecting divergence, detecting a data-mix shift mid-run, and confirming the tokenizer and formatting assumptions are what you think they are. A sudden loss discontinuity almost always means a data problem, not a learning-rate problem. The right action is to pause the run and inspect the mix, not to lower the LR and hope.

Where it breaks entirely: loss on data the model has memorised is meaninglessly low. Contamination and perplexity interact badly — which is the next concept, and it is the one that invalidates evidence rather than merely being unhelpful.

You cannot inspect a cake for flour quality. Everything you failed to check about the ingredients is now distributed irreversibly through the result. Data gates run before mixing because afterwards the only remedy is to start over.

Coverage is slice completeness, not volume. More tokens do not fix a missing slice. The artifact worth building is a slice matrix — intent × risk tier × locale — with counts in the cells, owned and maintained. Then define an edge-case budget: an explicit tolerance for failures in rare slices, set by severity. Low-severity long-tail failures can be tolerated at some rate. High-severity ones in your compliance slice get near-zero tolerance, and that decision is what stops “average helpfulness improved” from being an acceptable answer when unsafe compliance also rose.

A student who has seen the exam paper scores brilliantly. The score is real; the inference you want to draw from it is not. Contamination doesn't make your model worse — it makes your measurement worthless, silently, while every number improves.

Leakage is a spectrum, not an event. Three levels, each needing a different detector and a different response.

The gating shape, as pseudocode — this is the artifact, not the algorithm:

Treat leakage as a property of a slice, not of a dataset. Small overlap in a low-risk slice during early development is survivable. Overlap in your compliance slice is release-blocking, because that is the slice where you were relying on the measurement most.

Your internal suites are benchmarks too. Benchmark contamination isn't only about public leaderboards. If a training or synthetic-generation dataset ever includes prior versions of your regression suite, your gates weaken every cycle. The model memorises the test and you gradually lose the ability to detect real regressions — while every dashboard says things are improving. Contamination control is not research hygiene; it is a release-readiness requirement.

The output is a versioned artifact. The point isn't which similarity method you pick. It's that the process produces a contamination report, attached to a release, that can block progression. Without the artifact, detection is a one-off analysis someone did once and nobody can find.

“Did it follow instructions?” is like asking whether a contractor did a good job. Did they build the right thing, did they follow building regulations, and did they hand over the paperwork in the required format? Three questions, three failure modes, three different fixes — and they fail independently.

Measured separately, a regression tells you which layer to repair. Merged into one “quality” number, it tells you to rewrite everything. A starter micro-suite of 10–15 cases across your core intents is enough to expose brittle instruction-following before anyone runs a large review cycle.

Structured output deserves its own axis because it predicts operational reliability. Track parse-success rate, required-field-presence rate, field-type validity, allowed enum values, and constraint-violation flags — all deterministic, all cheap, all before any judge runs. This is the single highest-leverage set of checks in the whole training-time programme.

Not every constraint reduces to a rule. For the ones that don't, use a short anchored rubric — pass / borderline / fail — plus a severity field, rather than a numeric score. “Borderline” is not indecision; it is a genuine category meaning “met the intent and the format but ambiguous on a constraint”, and it routes to human review instead of being rounded into a mean.

Optimise a customer service team purely on “no complaints” and you will eventually get a team that refuses to do anything interesting, at length, very politely. Preference tuning shifts the whole output distribution, and the three things it shifts by accident are refusal behaviour, diversity, and length.

Over-refusal is a slice problem, not a global one. A global refusal rate hides the signal, because what you care about is correctness of refusal. The taxonomy that makes it measurable:

Gate the stage on all four together, or improving one silently pays for it with another.

Extra tokens need a justification. If verbosity rises without improving constrained helpfulness, safety recovery, or high-risk correctness, it is drift, not quality — and it will show up on your infrastructure bill and your p99 latency chart months later, at which point nobody will connect it to a preference-tuning run.

A factory doesn't inspect finished products and write reports. It puts gauges, interlocks and shutoffs in the line, so the expensive failure is prevented by design rather than discovered afterwards. At inference time you are not evaluating a model call — you are evaluating a pipeline, and failures live in the seams between steps.

Split the request into two coupled threads. The user-facing flow runs from input to rendered answer. The execution flow records retrieval hits, tool calls, failures, retries and verification flags. The rule that prevents the whole class of confident-lie failures is: the user-facing reply may only assert what the execution thread proved.

Prompts are the highest-frequency lever and the least governed. A small edit can change routing, refusal behaviour, verbosity and tool calling. Three things make prompt changes safe: log PROMPT_ID, PROMPT_VERSION and a template hash on every request; categorise changes by risk so you know what regression scope they demand; keep diffs reviewable and reversible. The dangerous edits are not the ones that change wording — they're the ones that change constraints, output contracts or tool instructions, and those should automatically trigger full high-risk-slice regression.

Every control costs something. Contract validation costs microseconds. Bounded repair costs a retry. A guardrail model costs a second full inference on the whole conversation, because safety assessment can't be done token by token. Low-temperature decoding on high-risk slices costs you response diversity users sometimes like. The design job is picking which controls earn their price per slice, not applying all of them everywhere.

A hospital has a vitals monitor (continuous numbers, cheap, no story), a patient chart tracing one person through admission → theatre → recovery (expensive, complete, one person), a nurse's notes (unstructured, searchable, verbose), and a diagnosis (a judgement about whether the outcome was good). Metrics, traces, logs, quality. Asking a vitals monitor which surgeon was slow is a category error.

The pull/push distinction is not trivia. Metrics are pulled, which means the scraper controls the rate and a dead target is visible as a failed scrape. Traces are pushed, which means the application controls the rate and a dead application is simply silent. That difference determines what you can build an alert on — and it is exactly the property that a scale-to-zero serverless endpoint breaks. Hold that thought for the lab.

Kubernetes gives you neither metrics nor tracing natively; both are conventions layered on top. For metrics the de-facto standard is Prometheus with the OpenMetrics exposition format — a CNCF project that standardised and extended the original Prometheus text format while keeping backward compatibility. For tracing it's OpenTelemetry. Logs in Kubernetes go to stdout/stderr and land in a node file, which makes kubectl logs easy and long-term retention someone else's problem — you add Loki or equivalent, or you lose them on pod churn.

A car has an odometer (only goes up, resets when you replace it), a speedometer (goes up and down, reading now), and — if you wanted p95 speed — you'd need to have been recording how long you spent in each speed band. That last one is the histogram, and it is the one people get wrong.

Histogram vs summary is the decision, and there is a right answer for LLM serving.

With two Modal replicas, a summary gives you two p95s and no legal way to combine them. That alone settles it: histograms for anything you will alert on. vLLM agrees — TTFT, time-per-output-token and end-to-end request latency are all exported as histograms.

The query, and the trap inside it:

Bucket boundaries are a design decision you make once. If your TTFT buckets top out at 2.5 s and reality is 9 s, everything above 2.5 s lands in +Inf and histogram_quantile extrapolates — your p99 becomes fiction that looks like data. Pick buckets that straddle your SLO threshold with resolution on both sides.

Cardinality is the thing that takes your monitoring down. Every unique combination of label values is a separate time series, stored and indexed independently. The multiplication is silent until it isn't:

The rule: labels are for values from a small, bounded, slowly-changing set. Anything unbounded — user ID, request ID, prompt text, document ID, error message — belongs in a trace or a log, never a metric label. If you need to jump from a metric spike to the individual requests behind it, that's what exemplars are for: a trace ID attached to a sampled histogram observation, giving you one click from the chart into a trace, without putting the ID in the label set.

Two phases, two completely different bottlenecks, and users feel them differently. Prefill reads the whole prompt and produces the first token — that's the pause before anything appears, and it is compute-bound. Decode produces every subsequent token one at a time — that's the speed of the text scrolling, and it is bound by memory bandwidth. A single “latency” number averages these two unrelated things into a number that describes neither.

The metric names, in both vocabularies. You will see both in the wild, so know the mapping:

Note the shape of that last row: vLLM V1 moved from exporting a prefix-cache hit-rate gauge to exporting queries and hits counters. That is strictly better — you can compute a rate over any window, instead of being stuck with whatever window the server chose.

Verify names against your build. Metric names and semantics change between vLLM releases; you are on 0.21.0. Before copying any query into production, curl your own /metrics and grep. This is a thirty-second check that prevents a dashboard that renders beautifully and shows nothing.

Prometheus is a meter reader that walks a fixed round every fifteen seconds. It needs a stable address and a target that's home. Both of those assumptions get interesting in Kubernetes, and one of them is simply false on serverless.

In Kubernetes the wiring is declarative. A ServiceMonitor (or PodMonitoring on Google's managed service) selects targets by label and sets the scrape interval; the container just exposes /metrics. For KServe you don't write the annotations by hand — the controller does it, from annotations on the ServingRuntime and InferenceService:

Managed collection follows the same shape with the operational burden removed. Google Cloud Managed Service for Prometheus runs collectors as a DaemonSet that scrapes only co-located targets and pushes to Google's backend — you keep PromQL and your existing instrumentation, and you stop running Prometheus servers. There are even predefined dashboards for the LLM serving path: Google documents managed-collection exporters for GKE Inference Gateway, vLLM and llm-d.

A motorway can be “100% occupied” with cars crawling at 8 km/h. Occupancy and throughput are different measurements, and the first one is the one that's easy to collect. DCGM_FI_DEV_GPU_UTIL tells you the GPU was busy during the sample window. It does not tell you it was doing useful work efficiently, and during decode it is routinely near 100% while the compute units are mostly idle, waiting on memory.

Every vendor uses the same architecture: a management component collects from the device, an exporter presents it on /metrics in Prometheus format. NVIDIA's is DCGM with dcgm-exporter, deployed as a DaemonSet on GPU nodes (port 9400), or provisioned automatically by the GPU Operator. AMD and Intel ship equivalents; there is no common naming convention across vendors, which matters if you ever run mixed fleets.

The metrics that matter, grouped by the question they answer:

The diagnostic combination, and the reason you collect more than one: GPU util high + memory-copy util high + SM activity low = memory-bound decode, which is the expected steady state and means you should be looking at batch size and KV cache, not at the model. GPU util high + SM activity high = compute-bound, which for an LLM means you are prefill-dominated and should look at prompt lengths and chunked prefill.

On Modal you do not run a DaemonSet, so DCGM is not available to you today — GPU telemetry is one of the concrete things you gain by moving to KServe on GKE, alongside the GPU Operator provisioning the exporter for you. Design the dashboard so the GPU row degrades gracefully: build the panels against DCGM metric names now, accept that they're empty on Modal, and they light up the day you move. A dashboard you have to rebuild at migration time is a dashboard nobody migrates.

A trace is a delivery tracking number for one request. Each handoff — gateway, retrieval, model, tool — stamps it with a start time, an end time and some attributes. The stamps only assemble into one journey because everyone agreed to pass the same identifier along. That agreement is context propagation, and it is the entire trick.

vLLM speaks OpenTelemetry natively — one flag on the server:

A GenAI span carries a standard attribute set. Names you'll actually see:

Prompt and completion content is opt-in and off by default — a deliberate PII decision. Capturing it means setting an explicit environment variable, and the content attributes carry warnings about sensitive data. Decide that per-slice, not globally.

A generic APM tool sees an HTTPS call to a hostname taking 4.2 seconds. An LLM-native tool sees which prompt version was used, what came back, how many tokens it cost, which retrieved chunks were in context, and what a judge scored it. Same underlying trace, radically different question-answering power.

Choosing: both are open-source, both self-host, both do tracing + evaluation + datasets + experiments. Langfuse leans toward prompt management, human annotation queues and CI-gating workflows; Phoenix leans toward OTel-purity and span-level analysis. For your stack the tiebreaker is that Langfuse's score model maps cleanly onto the S1 evaluator/metric vocabulary, and you already have Langfuse examples running.

Do not run two tracing systems on the same request path. Pick one collection path, and if you need the data in two places, fan out at the collector — the OTel Collector exists precisely so that instrumentation is decided once and destinations are decided later. Double-instrumenting produces two incomplete trees and a lot of confident wrong conclusions.

A monthly reliability target isn't a wish, it's a budget. 99.9% over 30 days means you are allowed 43 minutes of failure. Spend it slowly and nobody needs to wake up. Spend a third of it in an hour and someone does. That framing — rate of spend, not absolute level — is what stops alert fatigue.

Multi-window multi-burn-rate alerting is the technique that makes this usable. One alert per severity, each requiring a fast window and a slow window to agree — the slow window prevents a 30-second blip from paging, the fast window makes recovery detection quick:

Both alerts are on the same SLO. What differs is urgency, and therefore the destination: one pages, one files a ticket. Everything else goes on a dashboard and pages nobody.

Quality SLOs need longer windows than latency SLOs, for a statistical reason. If you sample 2% of 10 000 daily requests for judging, that's 200 scores/day — about 8 per hour. A one-hour window on a quality metric is nearly pure noise; the confidence interval on 8 samples is enormous. Quality burn-rate alerts want 6h/24h windows, or a much higher sampling rate on the specific slice you're gating. This is the S1 sample-size arithmetic showing up again, now as an alerting design constraint.

Every alert needs a runbook URL. the source material's example includes one and it's the most underrated line in the section. An alert without a documented response procedure is a notification that someone should feel bad.

You want to know whether marketing copy will sell. You can't run the sale (too slow, too confounded), and you can't hire a focus group per draft. So you write down what good copy looks like, and hand that checklist to a competent reader. LLM-as-Judge is the checklist plus a reader that never gets tired and costs a fraction of a cent.

Three escalating options. Most teams start at 1 and only move when they have data.

Fourth design axis, orthogonal to all three: what shape is the output? A single piece scored absolutely (pointwise), a head-to-head against a reference (pairwise), or a ranked list. Choose based on how the score gets used: if a decision is made from the output, binary is best, because it ties the score directly to the correctness of the decision.

The counter-intuitive caveat about explanations. Asking the judge to justify each score is standard advice, and it buys real interpretability. But there is evidence that requiring self-explanation degrades evaluation performance and bias mitigation — plausibly because the act of generating a justification entrenches a bias rather than exposing it. If you genuinely don't need explanations, an alternative is to take the probability-weighted mean over the score distribution rather than the single most-likely score. Be careful combining that with chain-of-thought prompting; the two interact oddly.

Practical read for you: keep explanations on. You need them for calibration and for debugging your rubric, and you are nowhere near the regime where squeezing the last few points of judge accuracy matters. Just know the trade-off is real and revisit it once your calibration numbers plateau.

Ask ten people to rate a film out of 100 and you get chaos. Ask them “would you recommend it — yes or no” and you get a signal. Every lever below is a variation on that: reduce the number of ways the instrument can wobble.

A fifth option when stakes justify the cost: LLM-as-jury — several models evaluate, each optionally adopting a different stakeholder's perspective. Combine it with binary questions and you get nuanced scoring out of individually-reliable judgements; that combination is sometimes called polling.

Make the criteria a proxy for the outcome or the KPI. Measuring outcomes is best but usually confounded; the next-best thing is judging against business-aligned indicators. Two warnings come with that: watch for adversarial actors gaming the metric, and remember that a criterion you can't connect to any downstream consequence is a criterion you're measuring because it was easy to write.

Every measuring instrument has systematic error. A cheap kitchen scale reads 3 g heavy — that's fine if you know it, because you subtract 3. The danger is an unknown bias, and the specific danger with judges is that several of their biases push in the same direction as the thing you're trying to measure.

The three that will actually bite you, with what the literature says and what to do:

Leniency, and the three ways out. Judges behave like a professor who gives every student an A and B. Combined with non-determinism, that means you should not compare two absolute scores to establish a ranking. Instead: (1) compare the two pieces directly in one pairwise call; (2) normalise within a group — generate several responses, score them together, and express each score relative to the group average (the group-relative approach used in GRPO-style training); (3) lower your expectations about what the judge is for — use it to identify problems rather than to rank near-equivalents. A technical RAG failure like total context loss scores near zero while normal operation scores 0.95, and that gap is detectable even by a lenient instrument.

The uncomfortable 2026 finding about ensembling. The standard defences — ensembling across judge models, reporting inter-judge agreement, reversing presentation order — reduce variance within the judge population. They do not remove systematic bias shared across it. Recent work identifies exactly this: shared biases that survive all three defences. The practical implication is blunt — agreement between judges is not evidence of correctness. Only agreement with humans is, which is the next concept.

When to stop using a frontier judge. Given the cost and bias profile, fine-tuned small judge models are worth considering for specific dimensions, and specialised evaluator models exist for particular tasks and industry-specific metrics. For your pipeline the relevant version of this is: your own Gemma E4B, running on hardware you already pay for, is a legitimate judge candidate for the deterministic-adjacent dimensions — once calibrated.

Before a new thermometer goes in a hospital, someone puts it in ice water and boiling water and checks it reads 0 and 100. Nobody argues about whether that's necessary. Calibrating a judge is the same procedure: a small set of cases where you already know the answer, and a number describing how well the instrument reproduces it.

Treat the judge as a classifier against a human reference, and report what you'd report for any classifier.

The loop: label 30–200 real examples with domain experts (domain knowledge matters here, and the gap between experts and generalist annotators is large) → baseline the full metric ensemble → rewrite the judge prompt against the worst-performing examples → repeat. In practice alignment plateaus in five to ten iterations.

Three legitimate places to stop calibrating:

1. Alignment above ~0.80 with iterations yielding under ~0.02 each — you are now fitting variance in the golden set, not real failure modes.
2. Human inter-rater agreement below ~0.6 κ on the same set — the task is genuinely ambiguous and further calibration optimises noise. Redefine the criterion instead.
3. A product decision: “catches 6 of 9 hallucinations” may be fine for a developer tool and unacceptable for a regulatory flag. Define that threshold before you start.

Report per-dimension, never as one average. An aggregate alignment of 0.80 can hide a faithfulness alignment of 0.55 — close to chance on a binary task. And two specific pathologies to watch for: high accuracy with low κ means the judge is right by chance and would score similarly on a shuffled dataset; high precision with low recall means it's conservative — right when it flags, but missing most real failures. Neither is trustworthy.

In a chat UI, when the answer is wrong you say so and it fixes it. API calls are stateless, so that loop doesn't exist unless you build it. Reflection is that loop, automated: generate, critique, apply the critique, regenerate.

Instead of one call, you make two or more. The output goes to an evaluator — which may be an LLM judge, an external tool, or a human — and crucially the evaluator returns a critique explaining how the response falls short, not just a score. That critique becomes part of a modified prompt, and the cycle repeats until the response clears a bar.

The one-round special case is the pragmatic default, and for a specific reason: with exactly one retry, you never have to define what “good enough” means. No threshold, no infinite-loop guard, no leniency problem. Given how hard it is to pick a threshold on a lenient judge's scores, that is a bigger simplification than it looks.

Reflection multiplies calls, so it multiplies tail latency. That's the whole trade. Where the cost of being wrong is high relative to an extra inference — code generation against a compiler, product listings that must be complete and correct — multiple reflection stages are clearly worth it, and the evaluator can be an external tool rather than a model. Where latency is the product — chatbots, real-time interfaces — reflection can push tail latency past acceptable limits, especially since you often already need a retry strategy just to get the first call to succeed.

Get the evaluator right or the loop is worse than nothing. The evaluation stage is the important part: a bad critique produces confident, expensive wrongness. And use a different model for critique than for generation — the self-preference bias above is exactly what you're defending against.

A middle path worth knowing: generate several drafts up front, critique all of them, prune the weak candidates, and continue — a beam-search shape rather than a linear loop. Better quality per round, higher cost per round. Develop a heuristic that sets reflection depth from problem characteristics, available time, and the business impact of the expected quality gain, rather than picking one depth for everything.

A clinical trial does not report “the drug worked”. It reports what dose, of what compound, in whom, against what comparator. Change any of those and the result no longer transfers. The system tuple is your dose sheet. It is the exhaustive list of things that, if changed, could change your results — so that when a number moves, the set of candidate explanations is finite and written down.

Without it, every regression investigation starts with an argument about what changed.

Your tuple is not abstract — it's the concrete inventory of your Gemma stack. Every row below gets a version string, and every evaluation run and every production trace carries all of them.

Two rows deserve the italics. The retrieval index changes on a cron, not on a deploy — so a quality drop can have no corresponding commit. And evaluator versions change the measuring instrument, not the thing measured: if the judge rubric changes, your whole history recalibrates and the trend line lies.

Versioning is cheap; the discipline is not. The tuple only pays off if it's emitted automatically. A tuple you have to remember to fill in is a tuple that is wrong within two weeks. Compute it once at gateway startup, attach it to every span as resource attributes and to every eval run as a frozen JSON blob.

Correlation is not attribution. You still need a targeted rerun — pin the old index snapshot, rerun the affected slice, compare. The tuple's job is to make that rerun possible, not to prove anything by itself.

Where it breaks: serverless. On Modal with min_containers=0 and max_containers=2, two replicas can be running different container images during a rolling deploy. If the tuple is captured gateway-side only, you will attribute a mixed-version window to a single version. Have the gateway read the engine's reported version from the response and record it per request, not per deploy.

Think of it less as a test set and more as an institutional memory of what has gone wrong. Every case earns its place by representing either a behaviour you must guarantee or a failure you have already survived. A set assembled that way grows in a specific direction — toward the things that actually break — instead of growing toward whatever was easy to write.

Your pipeline produces four output types plus a routing decision. Each needs a different evaluator, so each becomes its own slice with its own sizing and its own gate.

The compliance-flag slice is the one to design first, because it is the only one where the correct evaluator is free. A compliance flag must cite a span from the source document. Whether that span appears verbatim in the source is a string operation, not a judgement. That converts your highest-risk output into a deterministic check — the single biggest win available in this whole eval set.

def citation_is_bound(flag, source_text):
    # the model claims a compliance issue and cites a span
    if not flag["raised"]: return True
    span = norm(flag.get("cited_span", ""))
    return len(span) >= 24 and span in norm(source_text)

Three more structural rules, straight from the source material: keep a development set you iterate against, a release-gating set that stays stable enough for fair version-to-version comparison, and a hold-out set you look at rarely — because a set you tune against stops measuring generalisation and starts measuring how hard you tuned.

Sizing is per-slice, not global. The n column above is not decoration — a 1,030-case suite that is 80% sentiment tells you almost nothing about compliance flags, because the slice that matters has 40 cases and a confidence interval ±15 points wide. Size each slice for the delta you need to detect on that slice. The maths is in the reality check below.

Synthetic data expands, it does not replace. You can generate 500 paraphrases of a risk-score prompt in an afternoon, and it will make the suite look robust while measuring nothing new. the source material's warning is sharper than the usual one: if you use the same model family to generate cases and to judge them, the generator produces cases the judge finds natural, and the system self-congratulates. Keep a real-data core, label synthetic cases as synthetic, and report the two subsets separately.

Every case needs an owner and a reason. An unowned suite goes stale in a quarter — policies change, the taxonomy drifts, and nobody knows whether a failing case is a real regression or an out-of-date expectation. One line of provenance per case (owner, added_because) is what keeps the suite alive.

Airport security has layers with deliberately different sensitivities: a metal detector that everyone passes through, a swab test for a sample, a full search for the flagged. Nobody argues that the metal detector should catch everything — it's tuned to be fast and cheap, and the later layers exist precisely because it isn't sufficient.

Your gates work the same way. Each layer is cheaper and blunter than the one after it, and each has a different job. The mistake is trying to make one layer do everything — usually by putting the expensive judge in CI on every commit, which makes CI slow enough that people start skipping it.

Five layers, each with its own trigger, cost and action.

Gate sensitivity has a failure mode in both directions, and they're asymmetric. Too loose and you ship regressions — bad, but visible and correctable. Too tight and you get rollback fatigue: the gate fires on noise, people learn to override it, and within a month the gate is decorative while still appearing green in your compliance story. That second failure is worse because it's invisible. Review every gate after each rollout and ask whether it produced signal or noise; retire gates that only ever produce noise.

Hard blockers must be slice-scoped or they don't work. A global unsupported-claim rate of 0.05% looks excellent and can still hide a 4% rate inside the compliance slice, because the compliance slice is 6% of traffic. the source material's rule holds: zero-tolerance gates are evaluated within the high-risk slice, never on the aggregate.

Judge budget is a gate design parameter. Putting the judge at L2 at 100% coverage would cost roughly 5× more per PR and add 20 minutes. The subsample at L2 and the full run at L3 is a deliberate purchase of speed at the cost of sensitivity — L2 will miss small quality deltas, and that is fine, because L3 exists.

A factory does not weigh every biscuit. It weighs a sample — but not a random one. It deliberately over-samples the line that has been drifting, and the shift right after a machine change. Stratified sampling buys you statistical power exactly where you need it, which is never uniformly distributed.

Uniform random sampling of your traffic would spend most of its judge budget on the sentiment slice, because that's most of your volume — and sentiment is the slice you already measure for free with a reference metric. That's the whole argument for stratification in one sentence.

Three properties make online judging safe: it is asynchronous (never on the user's critical path), stratified (sample rate varies by slice), and budgeted (a hard daily ceiling on judge spend, so a traffic spike can't produce a cost spike).

The sample rates in the sampler box are not arbitrary — they're derived from how many judged samples each slice needs per day to make its weekly trend readable. That derivation is in the reality check.

Sampling bias is the quiet killer. If you sample only requests that completed, you systematically exclude timeouts, truncations and fallbacks — which is to say you exclude your worst outputs, and your quality metric improves precisely when your system degrades. Sample at the point the request is accepted, not the point it succeeds, and record an outcome label so failures stay in the denominator.

Judge scores need their own health metrics. A judge is a service: it has an error rate, a latency, and a refusal rate of its own. Track judge_call_failures_total separately, and never let a judge failure silently count as a passing score. Default a failed judge call to “unscored”, not to “pass” — the difference decides whether an outage looks like perfect quality or like missing data.

Retention and privacy. Traces of earnings and SEC analysis carry customer-identifying context. Store what you need to replay the evaluation — document IDs, retrieval ranks, tool outcomes, verification flags — and redact free text you don't need. the source material's phrasing is a good default: capture the observable event sequence, not the model's private reasoning.