Making a model prefer good answers

Your last stage ended on a bridge: “the model follows instructions — making it prefer good answers is next.” Here is what that sentence actually costs, mechanically. Instruction tuning showed your Gemma what a good analyst output looks like. It never showed it which of two plausible outputs is better. That comparison — and everything built on top of it — is this stage.

Every technique in this stage is one of three verbs. Not three eras — three things you can teach a model to do. Keep this ladder in your head; every method below gets placed on a rung.

This is the diagram the whole stage hangs off. The top path is classical RLHF as InstructGPT defined it. The arc underneath is DPO: the discovery that you can solve the same objective, on the same data, without ever building the reward model or running the RL loop.

There is one failure mode underneath every technique here, and it has a name. Charles Goodhart's original phrasing was that any observed statistical regularity collapses once pressure is put on it for control purposes; the popular compression is “when a measure becomes a target, it ceases to be a good measure.”

In post-training the measure is your reward — a reward model, an LLM judge, or a verifier — and the target is your policy. Optimise hard enough and the policy will find whatever the measure rewards that the measure did not mean to reward.

the source material is a leading alignment researcher's the RLHF literature (living edition). the section numbering below is the canonical map:

Your SFT'd Gemma will answer. It will answer in JSON, with the right keys, because you taught it the shape. What it will not do is reliably pick the better of two answers it could equally well produce — because during training it never saw a worse one.

That gap is not academic. Consider a prompt your fintech pipeline actually sees:

The vocabulary, before anything else

Every term below is used constantly for the rest of the stage. Nothing later assumes you already know one of these.

Post-training is three optimisation families, not one technique

the literature's framing, and the one worth internalising: post-training is a multi-stage processis the three-verb ladder from the stage map, stated in the material's own vocabulary.

The elicitation interpretation

A useful mental model for why so much can be gained on top of a fixed base model: post-training is mostly extracting capabilities the base model already has, not installing new ones. The base model can already write a calibrated risk assessment somewhere in its distribution; post-training raises the probability that it does so reliably, on the first sample.

This is why the honest answer to “can post-training fix X?” is usually a question about whether X exists in the base distribution at all.

Three passes on the RLHF problem formulation

The bridge from your fine-tuning stage — only what the literature adds

You have already run QLoRA SFT with correctly-rendered Gemma chat templates and TRL's SFTTrainer. §4 is a skim. Here are the four things in it you probably have not internalised, because they only start to matter once a second training stage exists.

Canonical recipes — how the field's answer changed three times

Step through the three recipes that define the eras. Notice what stays constant: SFT is always first, and evaluation always gates the end.

What post-training can and cannot fix

Worked example — the price of your preference data

Before any GPU cost, the decision that dominates your budget is who labels the pairs. the literature's current figures (as of 2026): a single piece of human preference data costs on the order of $1 or more per comparison, and can exceed $10 per prompt for specialist domains; AI feedback from a frontier model costs under $0.01.

Worked example — reading a KL number

You will see kl logged in every RL run and implicitly controlled by β in every DPO run. Here is what the number physically means, on three tokens.

Three things that are true this year and were not when the section was drafted

No training code yet — this session's lab is an inventory. You already own more of this pipeline than you probably realise.

Optional exercise

Pull 200 prompts from your pipeline's last week of traffic. For each, sample two completions from your SFT adapter at temperature=0.8. Do not label them yet. Just read twenty pairs yourself and write down, in plain English, the rule you were using to decide which was better.

If you can write that rule as code, you are on rung 3 and should be reading S6. If you cannot — and for a fintech analyst output you almost certainly cannot write it completely — you have just proven to yourself that you need rung 2. That is the whole point of the exercise.

Write down, as a differentiable function of tokens, what makes a compliance flag useful. You can't. Nobody can. That impossibility is the founding fact of RLHF, and the reward model is the workaround.

The trick is a swap that sounds too easy to work: it is much harder for a person to write a good answer than to recognise one, and harder still to write a rule than to point at the better of two options. So we stop asking for rules and start collecting comparisons — then fit a model that turns comparisons into a number.

The atom: one preference pair

Everything in sessions 2 through 5 is built on this record. It is worth staring at, because its shape constrains everything downstream.

{"prompt": "Flag compliance risk in this excerpt: …",
 "chosen":   "{\"risk_score\":0.78,\"evidence\":\"we expect margins…\"}",
 "rejected": "{\"risk_score\":0.5,\"evidence\":\"\"}"}

Three fields. Note what is not there: no score, no reason, no confidence. The magnitude of the preference has been thrown away — the standard practice is to binarise along the preference direction, reducing a rich rating into a bare chosen/rejected relation. Section 5.4 of the material covers attempts to keep that magnitude; none has become standard practice.

Bradley-Terry in three passes

rewards_chosen   = model(**inputs_chosen)     # (batch,)
rewards_rejected = model(**inputs_rejected)   # (batch,)
loss = -F.logsigmoid(rewards_chosen - rewards_rejected).mean

Four things called “reward model” that are not the same thing

This is the single most confusing terminology in post-training, partly because the term ORM is used inconsistently in the literature. Step through them.

separator_ids = tokenizer.encode(step_separator, add_special_tokens=False)
completions_ids = [c + separator_ids for c in completions_ids]
labels = [[-100] * (len(c) - 1) + [l] for c, l in zip(completions_ids, labels)]

Where the proxy cracks

This is the Goodhart section for reward models. Each of these is a real, documented way a reward model scores something other than quality.

Worked example — two toy reward scores

Worked example — how many comparisons does a reward model need?

A rough but useful sanity check before committing budget. InstructGPT used ~100K pairwise prompts to train its RM on top of ~10K SFT examples; Tülu 3 used ~1M preference pairs.

Three current findings the section predates or only gestures at

The decision tree above already gave the recommendation. Here is the argument in your specific terms, and then the code for both routes so you can see what you are declining.

Route you will take: a composite scorer that emits pairs

import json, jsonschema

ANALYST_SCHEMA = {...}  # your existing production schema

def hard_checks(text: str) -> dict:
    """Deterministic, unbiased, free. Returns a dict of booleans + a tier."""
    out = {"parses": False, "schema_ok": False,
           "evidence_present": False, "score_in_range": False,
           "flag_has_citation": False}
    try:
        obj = json.loads(text)
    except json.JSONDecodeError:
        return out
    out["parses"] = True
    try:
        jsonschema.validate(obj, ANALYST_SCHEMA); out["schema_ok"] = True
    except jsonschema.ValidationError:
        pass
    out["evidence_present"]  = bool(obj.get("evidence", "").strip)
    out["score_in_range"]    = isinstance(obj.get("risk_score"), (int, float)) \
                                and 0.0 <= obj["risk_score"] <= 1.0
    out["flag_has_citation"] = (not obj.get("compliance_flags")) or out["evidence_present"]
    return out

def tier(checks: dict) -> int:
    # lexicographic quality tiers; higher is strictly better
    if not checks["parses"]:      return 0
    if not checks["schema_ok"]:   return 1
    if not checks["score_in_range"]: return 2
    if not checks["flag_has_citation"]: return 3
    return 4

def make_pair(prompt, cand_a, cand_b, judge):
    ta, tb = tier(hard_checks(cand_a)), tier(hard_checks(cand_b))
    if ta != tb:                      # settled for free, zero bias
        hi, lo = (cand_a, cand_b) if ta > tb else (cand_b, cand_a)
        return {"prompt": prompt, "chosen": hi, "rejected": lo,
                "source": "programmatic", "margin": abs(ta - tb)}
    return judge.compare(prompt, cand_a, cand_b)   # biased; measure it

Note the source and margin fields. Keep them. In S5 you will want to be able to slice your training curves by whether a pair was settled programmatically or by the judge — if the judge-labelled slice behaves differently, you have found a bias.

Route you are declining: a real RewardTrainer config

For completeness, so you recognise it in someone else's repo. TRL's RewardTrainer implements exactly the one-line loss above.

from trl import RewardTrainer, RewardConfig
from peft import LoraConfig

cfg = RewardConfig(
    output_dir="gemma-fintech-rm",
    num_train_epochs=1,           # ONE. reward models overfit fast.
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=1e-5,
    max_length=1024,
    bf16=True,
    gradient_checkpointing=True,
)
trainer = RewardTrainer(
    model="google/gemma-4-E4B-it",     # + AutoModelForSequenceClassification head
    args=cfg,
    train_dataset=pairs,                # prompt / chosen / rejected
    peft_config=LoraConfig(r=16, lora_alpha=32, task_type="SEQ_CLS"),
)
trainer.train

If you ever do need this: group all pairs from the same prompt into one batch (the InstructGPT balancing point above), and evaluate the result on held-out pairs as a plain accuracy — “how often does the RM order the pair the way the judge did.” Anything under ~65% and the RM is not usable as an optimisation target.

Optional exercise

Take 100 pairs from your scorer. For 50 of them, swap the presentation order of the two candidates and re-run the judge. Count how often the verdict flips. That flip rate is your judge's position bias, measured directly, in about ten minutes.

Anything above ~5% and you must either always run both orders and take the agreement (doubling judge cost), or drop pairs where the two orders disagree (cheaper, and it also removes the genuinely ambiguous pairs — which, per the saturation analysis above, were low-information anyway). You will use this number again in S4 and S7.

Your reward model says this completion is worth 0.8. Now what? There is no target string to regress towards — the whole point was that you could not write one. You have a number attached to a sequence you sampled, and you need it to become a change in weights.

Policy-gradient reinforcement learning is the machinery for that. It is also the part of post-training that most people never run and most people misunderstand, so this session has an unusual job: give you complete mechanical understanding of something you will read about far more often than you execute.

The core idea in three passes

pg_loss = -advantages * ratio   # ratio = π_θ(token) / π_old(token)

The PPO four-model dance

This is the diagram people mean when they say PPO is heavy. Four copies of a language model, three of them just to make one of them learn safely.

PPO's clipping — what the min/max actually does

The min looks arbitrary until you enumerate the four interesting cases. Work through them:

GRPO — delete the critic, use the group

Group Relative Policy Optimisation, from DeepSeekMath and used through DeepSeek-V3 and R1. The idea is almost embarrassingly simple: if you want to know whether a completion was better than expected, generate several completions to the same prompt and compare them to each other.

The KL leash, properly

logits     = model.forward(gen[:, :-1]).logits
ref_logits = ref_model.forward(gen[:, :-1]).logits
logprobs     = F.log_softmax(logits,     dim=-1)
ref_logprobs = F.log_softmax(ref_logits, dim=-1)
tok     = logprobs.gather(-1, gen[:, 1:].unsqueeze(-1)).squeeze(-1)
ref_tok = ref_logprobs.gather(-1, gen[:, 1:].unsqueeze(-1)).squeeze(-1)
kl_approx = tok.sum(-1) - ref_tok.sum(-1)

Loss aggregation — a “detail” that silently biases length

You have per-token losses. How do you reduce them to one scalar? Three answers, all in production use, and they do not agree.

Where the proxy cracks — over-optimisation in the RL loop

Everything in S2 about biased reward models becomes actively dangerous here, because RL is a strong optimiser pointed directly at those biases.

People say PPO is “too heavy for a single GPU” without ever showing the sum. Here it is, for your actual hardware and your actual model.

Three things happening in RL tooling right now

The reward function you would write for your JSON task

This is the most transferable artefact in this tab. Notice it is a verifier, not a reward model — which means if you ever do run RLVR (S6), this function is already most of the work.

import json, jsonschema
from trl import GRPOConfig, GRPOTrainer

def json_parse_reward(completions, **kw):
    """1.0 if it parses at all. The floor."""
    out = []
    for c in completions:
        try: json.loads(c); out.append(1.0)
        except Exception: out.append(0.0)
    return out

def schema_reward(completions, **kw):
    """1.0 if it validates against the production analyst schema."""
    out = []
    for c in completions:
        try:
            jsonschema.validate(json.loads(c), ANALYST_SCHEMA); out.append(1.0)
        except Exception: out.append(0.0)
    return out

def evidence_grounding_reward(completions, source_text, **kw):
    """1.0 only if the evidence span is VERBATIM in the source. Anti-hallucination."""
    out = []
    for c, src in zip(completions, source_text):
        try:
            ev = json.loads(c).get("evidence", "").strip
        except Exception:
            out.append(0.0); continue
        out.append(1.0 if ev and ev in src else 0.0)
    return out

def brevity_penalty(completions, **kw):
    """Explicit length control. Without this, RL discovers padding."""
    return [-max(0.0, (len(c) - 900) / 900) for c in completions]

cfg = GRPOConfig(
    output_dir="gemma-fintech-grpo-demo",
    num_generations=8,            # the GROUP. this is what replaces the critic.
    reward_weights=[1.0, 2.0, 3.0, 0.5],
    beta=0.04,                    # KL coefficient. verifier reward → keep it small.
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    max_completion_length=1024,
    use_vllm=True,               # colocate mode by default since TRL v1.0
    log_completions=True,        # you WILL need to read what it generated
    bf16=True, gradient_checkpointing=True,
)

trainer = GRPOTrainer(
    model="google/gemma-4-E4B-it",
    reward_funcs=[json_parse_reward, schema_reward,
                  evidence_grounding_reward, brevity_penalty],
    args=cfg, train_dataset=prompts, peft_config=lora_cfg,
)

Two things to notice. Reward weights are a policy decision, not a hyperparameter — setting grounding to 3.0 and brevity to 0.5 is you stating that a hallucinated citation is six times worse than a verbose answer. Write that down somewhere a human can review it. And the brevity penalty is not optional: length bias is not a reward-model-only problem, and an unpenalised RL loop with a per-sequence-normalised loss will find padding.

Execution: Modal or Kubernetes?

Argued choice for this session: Modal. The GRPO run here is a learning exercise with a known short duration and no queue contention, so the operational machinery of Kueue buys nothing. S5 makes the opposite argument for the real DPO run, where checkpoint-and-resume on spot capacity does pay for itself.

Optional exercise

Run the config above for 100 steps on 300 of your prompts, with log_completions=True, and then read the completions table rather than the loss curve. Specifically: find a prompt where all 8 group members scored identically. Its advantages are all zero, so it contributed nothing to the gradient.

Count how many of your prompts are like that. If most are, your reward function is not discriminating and no amount of GRPO will help — this is the same difficulty-filtering insight that made Olmo 3 drop prompts its base model solved 8/8 times. The group-relative advantage is only informative when the group actually disagrees.

Session 3 built an elaborate machine to solve one equation. Direct Preference Optimization solves the same equation — same objective, same data, same β — by noticing that the optimal policy has a closed form, and that if you rearrange it to express the reward in terms of the policy, the reward model cancels out of the Bradley-Terry loss entirely.

The paper's subtitle is the whole idea: Your Language Model is Secretly a Reward Model. This is the most commonly misunderstood sentence in post-training. It does not mean DPO avoids reward modelling. DPO is still fitting a reward model — it just uses the policy's own log-ratio as that model, so there is nothing separate to train.

pi_logratios  = policy_chosen_logps    - policy_rejected_logps
ref_logratios = reference_chosen_logps - reference_rejected_logps
logits        = pi_logratios - ref_logratios
losses = -F.logsigmoid(beta * logits)

chosen_rewards   = beta * (policy_chosen_logps   - reference_chosen_logps).detach
rejected_rewards = beta * (policy_rejected_logps - reference_rejected_logps).detach

What β actually does — the one hyperparameter that matters

The variant family — what each one actually changes

There are dozens. These are the ones you will meet, grouped by what problem they were built to solve rather than by name.

Where preference pairs come from

This is the part that decides whether your run works. Source §11, plus the synthetic-data section, plus everything you already know about judge bias from your evaluation stage.

Three current data points

No trainer code yet — S5 is the code-along. This lab designs the dataset, which is the decision that actually determines whether S5 works.

The layered pipeline, concretely

The judge decision, argued

Argued recommendation: external judge for L2 on redacted content, Gemma-as-judge as a second opinion for agreement measurement only. Where redaction is impossible, fall back to Gemma and widen L0 and L1 so that structured and programmatic pairs carry more of the dataset — trading coverage for bias, deliberately, and writing down that you did.

Target dataset shape

{"prompt":   [{"role":"user", "content": "…excerpt + task…"}],
 "chosen":   [{"role":"assistant", "content": "{\"risk_score\":0.78,…}"}],
 "rejected": [{"role":"assistant", "content": "{\"risk_score\":0.5,…}"}],
 // metadata — not consumed by the trainer, essential for you
 "layer":    "L1",
 "reason":   "evidence_present",
 "on_policy": true,
 "judge_agreed_both_orders": null}

Conversational format, because your Gemma chat template must render identically to how it renders at serving time — the same discipline you already applied in SFT. Keep layer and on_policy: in S5 you will slice the curves by them, and in S7 you will need to prove your training data was decontaminated against the golden set.

Composition target

• ~40% structured (L0) — free, unbiased, teaches schema and grounding discipline hard.
• ~40% on-policy sampled + programmatically settled (L1) — satisfies the Tülu 3 on-policy requirement, since one side is always your own model's output.
• ~20% judge-settled (L2) — the only layer that teaches domain taste, and the only one carrying meaningful bias.
• Total 3,000–5,000 pairs. Small by open-recipe standards and entirely adequate for a single narrow task.

Optional exercise

Build 200 L0 structured pairs before anything else — same prompt, once with the full schema and grounding rules in the system prompt, once with them stripped. Then run your current SFT adapter on 50 held-out prompts and check what fraction already satisfies every rule you just used to build those pairs.

If it is above ~90%, L0 pairs are near-saturated and will teach almost nothing — go straight to L1 and L2 for taste. If it is below ~70%, you have found something better than a DPO opportunity: you have found an SFT bug, and rung 1 will fix it faster and more cheaply than rung 2. That check costs an hour and can save a week.

S4 gave you five lines of loss. This session is everything those five lines don't tell you: how two model copies fit on one 24 GB card, which of TRL's three reference-model strategies to use and why, what a healthy margin curve looks like, and the five ways a DPO run fails while appearing to succeed.

This is the designated code-along session — but per the rule, concepts first. Read the recap below before running anything, because every code block afterwards implements one row of it.

Concepts recap — what each block will implement

Every DPO step needs four numbers: log π_θ(y_c), log π_θ(y_r), log π_ref(y_c), log π_ref(y_r). The first two come from the model you are training. The second two come from a model that must be the SFT checkpoint, frozen. Naively that is a second full model in memory — and that is exactly the thing your 24 GB card cannot afford twice.

The insight that rescues you is one you already have from LoRA: a LoRA-adapted model is the base model plus a small delta. Turn the delta off and you are the base model again. So the reference model is not a second set of weights — it is the same weights with the adapter bypassed.

Before writing a line of training code, prove the run fits. This is the arithmetic the brief asked for, at Gemma E4B scale (~4B effective params for the sum), sequence length 1024, batch size 1 with gradient accumulation.

Three grounded reference points for this run

Block 1 · Generate on-policy candidates

Implements the on-policy requirement from S4. Run against your existing vLLM endpoint — prefix caching makes the shared system prompt nearly free, which is the same optimisation you already benchmarked.

import json, asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://…modal.run/v1", api_key="…")
N_PER_PROMPT = 4          # UltraFeedback used 4; enough to find real disagreement

async def sample(prompt, adapter="fintech-sft"):
    r = await client.chat.completions.create(
        model=adapter,                    # multi-LoRA: select the SFT adapter by name
        messages=[{"role": "user", "content": prompt}],
        temperature=0.9, top_p=0.95,      # HIGH temp on purpose — see note
        max_tokens=512, n=N_PER_PROMPT,
        extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    )
    return [c.message.content for c in r.choices]

Block 2 · Label with the layered pipeline

Implements S4's L0–L3 design. The hard_checks and tier functions come straight from S2's lab.

import itertools, random, json

def judge_once(prompt, a, b, client, model):
    """Returns 'A', 'B' or 'TIE'. Temperature 0 to cut rating variance."""
    msg = JUDGE_TEMPLATE.format(question=prompt, answer_a=a, answer_b=b)
    out = client.chat.completions.create(
        model=model, temperature=0,
        messages=[{"role": "user", "content": msg}]).choices[0].message.content
    if "[[A]]" in out: return "A"
    if "[[B]]" in out: return "B"
    return "TIE"

def judge_symmetric(prompt, a, b, client, model):
    """L2. Ask both ways. Disagreement means the pair is genuinely ambiguous."""
    fwd = judge_once(prompt, a, b, client, model)
    rev = judge_once(prompt, b, a, client, model)     # A and B EXCHANGED
    if fwd == "A" and rev == "B": return a, b, True   # both say the first text wins
    if fwd == "B" and rev == "A": return b, a, True
    return None, None, False                          # L3: drop, do not guess

def build(prompt, cands, client, model):
    rows, seen = [], set
    for a, b in itertools.combinations(cands, 2):
        ta, tb = tier(hard_checks(a)), tier(hard_checks(b))
        if ta != tb:                                    # L1 — free, unbiased
            hi, lo = (a, b) if ta > tb else (b, a)
            rows.append(row(prompt, hi, lo, layer="L1", agreed=None))
        else:                                           # L2 — judged, order-swapped
            hi, lo, ok = judge_symmetric(prompt, a, b, client, model)
            if ok:
                rows.append(row(prompt, hi, lo, layer="L2", agreed=True))
    # cap pairs per prompt so one prompt cannot dominate the gradient
    # (the InstructGPT balancing point from S2 — batch by prompt if you keep more)
    return random.sample(rows, min(2, len(rows)))

Block 3 · The dataset TRL expects

from datasets import Dataset

ds = Dataset.from_list([{
    "prompt":   [{"role": "user",      "content": r["prompt"]}],
    "chosen":   [{"role": "assistant", "content": r["chosen"]}],
    "rejected": [{"role": "assistant", "content": r["rejected"]}],
} for r in rows])

ds = ds.train_test_split(test_size=0.05, seed=0)   # keep an eval split!

# sanity: render one row exactly as the trainer will, and LOOK at it
print(tok.apply_chat_template(ds["train"][0]["prompt"] +
                             ds["train"][0]["chosen"], tokenize=False))

That last print is not optional. Chat-template mismatch between training and serving is the single most common silent failure in this whole pipeline, and you already know it from SFT. Confirm the rendered string is byte-identical in shape to what your vLLM endpoint produces.

Blocks 4–6 · The trainer

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
from trl import DPOTrainer, DPOConfig

BASE = "google/gemma-4-E4B-it"
SFT  = "./adapters/fintech-sft"          # your capstone adapter

# --- block 4: 4-bit base --------------------------------------------------
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE, quantization_config=bnb, dtype=torch.bfloat16, device_map={"": 0})

# --- block 5: the reference, for free (option 3) --------------------------
model = PeftModel.from_pretrained(
    model, SFT, is_trainable=True,            # CONTINUE training the SFT adapter
    adapter_name="dpo_adapter", autocast_adapter_dtype=False)
model.load_adapter(
    SFT, is_trainable=False,                  # <-- the frozen reference. IMPORTANT.
    adapter_name="reference_adapter", autocast_adapter_dtype=False)

# --- block 6: config ------------------------------------------------------
cfg = DPOConfig(
    output_dir="gemma-fintech-dpo",
    beta=0.1,                          # the KL leash. S4's whole discussion.
    loss_type="sigmoid",               # "sigmoid_norm" if length creeps — TRL ≥1.4
    learning_rate=5e-6,                # SURPRISINGLY LOW. see note below.
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    num_train_epochs=1,                # DPO overfits fast
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,     # effective batch 8
    max_length=1024,
    max_prompt_length=640,
    precompute_ref_log_probs=True,     # 4 fwd passes → 2. huge win.
    bf16=True,
    gradient_checkpointing=True,
    eval_strategy="steps", eval_steps=50,
    logging_steps=5,
    save_strategy="steps", save_steps=100,   # spot-preemption insurance
    report_to="none",                        # or "trackio" / your Langfuse sink
)

trainer = DPOTrainer(
    model=model,
    ref_model=None,                     # the named adapters handle it
    args=cfg,
    train_dataset=ds["train"],
    eval_dataset=ds["test"],
    processing_class=tok,
    peft_config=None,                   # already a PeftModel — do NOT double-wrap
)
trainer.train
trainer.save_model("./adapters/fintech-dpo")

Block 7 · Reading the curves

DPOTrainer logs the implicit reward model from S4 directly. These four numbers are your instrument panel.

Block 8 · The five failure smells

Block 9 · Ship it — multi-LoRA A/B

# vLLM engine args — the pattern you already run in gemma_modal.py
--enable-lora \
--max-loras 2 --max-lora-rank 16 \
--lora-modules fintech-sft=./adapters/fintech-sft \
               fintech-dpo=./adapters/fintech-dpo \
--enable-prefix-caching --quantization fp8 --gpu-memory-utilization 0.92

Now both policies are addressable by name on one endpoint, on one GPU, and your Langfuse online-eval sampling can route a slice of live traffic to each. That is the A/B substrate S7's win-rate gate runs on — and it costs you no extra hardware, because a LoRA adapter is tens of megabytes.

Execution: Modal or Kubernetes — the argument flips here

Argued choice: Modal for the first run, Kueue for the sweep. Note this is the opposite of the S3 recommendation's reasoning and lands in the same place for a different reason — there, Modal won because the job was a one-off demo; here it wins because the job is short enough that orchestration buys nothing until you have many of them.

Optional exercise

Run the config twice, changing exactly one thing: beta=0.1 and beta=0.5. Plot rewards/accuracies for both on the same axes, then generate 50 completions from each adapter and compute schema-validity and mean token length.

You should see the β=0.5 run reach a lower accuracy but hold schema validity and length closer to the SFT baseline. That is the leash, measured on your own model — and it turns S4's β discussion from a paragraph you read into a trade-off you have personally observed. Keep both adapters; S7 will A/B all three against SFT-only.

Every failure mode in sessions 2 through 5 traces back to one fact: the reward was a learned opinion. Change the scorer from a model to a program, and Goodhart's grip loosens — not because the pressure disappears, but because a unit test cannot be flattered.

That is the third rung. RLVR — reinforcement learning with verifiable rewards — proceeds almost exactly like RLHF, but it makes the reward model optional, replacing it with a scoring function that returns a positive reward when the answer is correct and zero otherwise. It is the engine behind reasoning models, and it is also where post-training stops being a training-cost question and becomes a serving-cost question.

RLVR in three passes

# maths — extract and compare
def math_reward(completion, gold):
    m = re.search(r"\\boxed\{([^}]*)\}", completion)
    return 1.0 if m and normalise(m.group(1)) == normalise(gold) else 0.0

# code — run the tests
def code_reward(completion, tests):
    passed = run_sandboxed(completion, tests)   # all-or-nothing…
    return 1.0 if passed == len(tests) else 0.0
    # …or partial credit: return passed / len(tests)

Inference-time scaling — the other half

RL training and test-time compute are two ways of reaching the same place, and they predate each other's popularity.

The curve nobody puts on the marketing slide

More thinking helps, then stops helping, then actively hurts. This is not a fringe result; it is the standard shape.

You already benchmarked this hardware, so this is arithmetic on your own numbers rather than a vendor's. From your vLLM work: an L4 at batch 32 sustains 934 output tokens/second aggregate, and GCP L4 on-demand is $0.45/hour (≈$324/month for a 730-hour month).

Three grounding points from current practice

You already have an intelligent router that classifies request complexity and picks a model. Thinking is one more field in that same decision — and unlike model selection, it is free to change per request with no extra weights loaded.

from dataclasses import dataclass

@dataclass
class Route:
    adapter: str            # "fintech-dpo" | "fintech-sft"
    thinking: bool
    max_tokens: int
    reason: str

# thresholds are MEASURED, not guessed — see the exercise below
VERIFIABLE_TASKS = {"reconcile_totals", "extract_figures", "cross_foot"}

def route(req) -> Route:
    # 1. latency-bound paths never think. silence is not free for a user.
    if req.sla_ms < 10_000:
        return Route("fintech-dpo", False, 512, "latency SLA")

    # 2. verifiable tasks think, because we can PROVE the benefit
    if req.task in VERIFIABLE_TASKS:
        return Route("fintech-dpo", True, 2048, "verifier-backed uplift")

    # 3. batch/backfill paths think — throughput-bound, silence costs nothing
    if req.channel == "batch":
        return Route("fintech-dpo", True, 2048, "offline batch")

    # 4. everything else: off. the default is the cheap default.
    return Route("fintech-dpo", False, 512, "default off")

Note max_tokens moves with the toggle. An unbounded thinking budget is how you end up on the right-hand side of the overthinking curve; a hard cap is the crudest possible version of the oracle-stopping result, and it costs nothing to implement.

Measuring the uplift on your own stack

import time, json, statistics as st

def trial(prompts, thinking: bool, budget: int):
    correct, tokens, wall = 0, 0, []
    for p in prompts:
        t0 = time.time
        r = client.chat.completions.create(
            model="fintech-dpo", max_tokens=budget, temperature=0,
            messages=[{"role": "user", "content": p.text}],
            extra_body={"chat_template_kwargs": {"enable_thinking": thinking}})
        wall.append(time.time - t0)
        tokens += r.usage.completion_tokens
        correct += verify(r.choices[0].message.content, p.gold)  # YOUR verifier
    n = len(prompts)
    gpu_s   = tokens / 934.0                     # your measured L4 throughput
    cost    = gpu_s / 3600 * 0.45
    return {"accuracy": correct / n,
            "tokens_per_req": tokens / n,
            "p50_latency_s": st.median(wall),
            "cost_per_1k": cost / n * 1000,
            "cost_per_correct": (cost / max(correct, 1))}

for budget in (256, 512, 1024, 2048, 4096):
    print(budget, trial(golden_verifiable, True, budget))
print("off", trial(golden_verifiable, False, 512))

Read the cost_per_correct column, not the accuracy column. That is the whole discipline of this session: an accuracy improvement that costs 48× per correct answer is not an improvement, it is a purchase you would not have authorised if anyone had shown you the invoice.

Monitoring it in Langfuse

• Tag every trace with thinking=true|false and the routing reason. Without the tag you cannot slice anything.
• Track completion tokens split into reasoning vs answer as separate metrics. Aggregate token counts hide the entire story.
• Alert on p95 completion tokens, not the mean. Overthinking is a tail phenomenon — the mean will look fine while a slow-bleeding tail eats your KV cache.
• In Prometheus, watch KV-cache utilisation and queue depth alongside your existing TTFT/ITL panels. Thinking traffic will move those first, before it moves cost.

Optional exercise

Take 100 golden-set items where you have a programmatic verifier. Run the budget sweep above. Plot cost_per_correct against budget and find your peak — then set max_tokens for the thinking route to roughly 1.5× the peak budget.

Then do the more uncomfortable version: run the same sweep on 100 items where you have no verifier, only a judge. Compare the shape of the two curves. If the judge-scored curve keeps rising where the verifier-scored curve has already turned over, you have just caught your judge rewarding long reasoning traces for their own sake — length bias, in your evaluation, on your own data. That is exactly the failure S7 is built to prevent, and finding it yourself first is worth more than reading about it.

Your fine-tuning stage taught you to measure accuracy against a golden set. That instrument was correct for rung 1 and is close to blind for rung 2 — because preference tuning changes which of several acceptable answers you get, and an exact-match accuracy metric cannot see a difference between two answers it both marks correct.

The instrument that can see it is the same one that generated your training data: a pairwise comparison. Which is a pleasing symmetry and a serious hazard, because it means your eval and your training data now share a bias surface. Get this wrong and you build a model that games your judge, then use that judge to certify it.

Win rate, and its family resemblance to everything else in this stage

Contamination — and why your case is more tractable than the field's

Dataset contamination is evaluation data leaking into training data; decontamination is the search-and-remove process. The standard method is n-gram overlap between training prompts and evaluation prompts, or fixed-length character substring matching.

The single most useful piece of arithmetic in this session. You want to know whether your DPO adapter beats SFT-only by 5 points — 55% vs 50%. How many comparisons do you need before that claim means anything?

Three current data points on eval integrity

Tooling, if you want to stop writing harness code

Open evaluation frameworks worth knowing: Inspect AI (UK AI Safety Institute), LightEval (Hugging Face, powered the Open LLM Leaderboard), lm-evaluation-harness (EleutherAI, with a well-curated GPT-3-era setup), OLMES (Ai2), HELM (Stanford CRFM), and Databricks' Eval Gauntlet. For a narrow single-task adapter your own harness plus Langfuse is likely enough — reach for these when you need standardised public benchmarks alongside your golden set.

Everything from seven sessions, assembled into one runnable sequence against your actual infrastructure.

The release checklist

Print this. It is the gate, in order, and the ordering is the point.

What to write down when you ship

A short run record, because in six months you will not remember and neither will anyone else. Six lines:

• Data: pair count, layer composition (L0/L1/L2 percentages), on-policy fraction, decontamination hits removed.
• Judge: which model, which prompt template, measured position-bias flip rate, measured self-preference where applicable.
• Training: β, learning rate, loss type, epochs, effective batch, exact pinned versions of trl, transformers, peft, bitsandbytes.
• Curves: final rewards/margins and rewards/accuracies, and whether the eval callback showed any schema drift.
• Gate: win rate with CI, n, tie rate, length delta, per-task breakdown.
• Decision: shipped / canaried / rejected, and the one sentence of reasoning.

That record is also your defence against the cross-lab-comparison problem from earlier in this tab, applied inward: the only reproducible evaluation is one whose inputs you wrote down.

Optional exercise — the capstone itself

Run the whole loop once, end to end, at the smallest scale that is still honest: 1,000 preference pairs, one DPO run at β=0.1, and a 1,000-prompt win-rate gate against SFT-only. Fill in the six-line run record. Then do the one thing that separates a completed exercise from a working pipeline: take the golden-set items where the DPO adapter lost, and feed them back into stage 1 as next round's prompts.

Iterating on the data with a fixed algorithm is the loop that Olmo 3's team described as making DPO worth using despite its critics — cheap, stable, and highly iterable. You now have that loop, and it runs for about a quarter of a dollar per turn.