Fine-Tuning LLMs — stage workbook

Session 1 · decide

To tune or not — and the memory wall

Two skills, both defensive. First: recognising the small number of problems that actually require fine-tuning. Second: doing the arithmetic that tells you, before you rent a GPU, whether the run you have in mind can physically happen.

01 · why The problem this session solves

Your fintech pipeline takes earnings reports and SEC filings in, and is supposed to emit structured outputs: sentiment, risk scores, trading signals, compliance flags. Two failure modes look identical from the outside and have completely different fixes.

In the first, the model says Q3 revenue was $4.1bn when the filing says $4.6bn. In the second, the model gets every number right and returns them wrapped in a chatty paragraph — “Certainly! Here’s my analysis of the risk factors…” — instead of the strict JSON your gateway is trying to parse. Both are “the model is wrong.” Only one of them is a fine-tuning problem.

02 · concepts Core concepts, from zero

Sharpening the ladder: knowledge versus behaviour

Picture hiring a junior equity analyst. On day one there are two different things they can be missing.

They might not know this company’s numbers. The fix is a filing cabinet: you don’t send them back to school, you give them the documents and teach them to look things up. That is RAG — retrieval-augmented generation — and its defining property is that when the numbers change you swap the documents, not the analyst.

Or they might know everything and still write like a chatbot: burying the recommendation in three paragraphs of hedging when your desk wants a two-line note in a fixed format. No filing cabinet fixes that. You need them to internalise the house style until producing it is automatic. That is fine-tuning: you are changing the analyst, not their reference material.

Mechanically the two interventions touch different parts of the request path. RAG edits the input at request time — it prepends tokens. Fine-tuning edits the weights ahead of time — the token stream at request time gets shorter, not longer.

That second-order effect matters more than people expect. Every behaviour you push into the weights is a behaviour you stop paying for in the prompt on every single request. A 600-token system prompt full of formatting rules and few-shot examples, retired into the weights, is 600 tokens of prefill you never buy again. Generative AI on Kubernetes makes exactly this argument for model customization: a bank embeds its slow-changing knowledge about loans, trading and credit risk into the model so it does not have to ship that context with every request, cutting both context length and inference cost.

The memory wall: four terms

Serving a model and training a model feel like the same activity — load weights, push tokens through — but their memory profiles are not remotely comparable. Serving is like reading a source: you need the material. Training is like editing it: you need the material, a marked-up copy showing every proposed change, a set of notes tracking how each sentence has been trending across all previous edits, and a desk covered in the intermediate drafts you’ll need when you work backwards through the section.

The material is the smallest item on that desk.

Walk one training step and watch memory accumulate. the guide’s framing in the applied fine-tuning reference is the clean one: load the model, forward pass, backward pass, optimizer step, zero the gradients, repeat. Memory demand climbs through the first four and only falls at the end.

Now the picture that should stay with you. Same 4B model, same GPU, three training strategies.

The dtype zoo

A floating-point format spends its bits on two competing jobs: range (how big and how small a number it can represent at all) and precision (how finely it can distinguish nearby numbers). Fewer bits means giving something up. Which thing you give up turns out to matter enormously.

Every float splits into a sign bit, an exponent field (range) and a mantissa field (precision). FP16 and BF16 are both 16 bits and make opposite bets.

Loading a quantized model: bitsandbytes

bitsandbytes is the library that implements this inside the Hugging Face stack. You never call it directly; you hand a configuration object to from_pretrained and it swaps every eligible Linear layer for a Linear4bit (or Linear8bitLt) as the weights land on the GPU.

# the four decisions that matter — everything else can stay default
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # quantile-spaced, not plain fp4
    bnb_4bit_use_double_quant=True,     # ~0.4 bits/param back, free
    bnb_4bit_compute_dtype=torch.bfloat16, # dequantize to bf16 to multiply
)

Five lines that decide roughly 6 GB. The last one is the one people get wrong.

That fourth argument is the conceptual key. A quantized layer stores 4-bit indices but cannot compute in 4 bits. On every forward pass it dequantizes its weights to the compute dtype, multiplies, and casts the result back out. So there are three dtypes in play at once: storage (4-bit), compute (bnb_4bit_compute_dtype), and the dtype of everything that was never quantized (torch_dtype). Mismatch the last two and you get a warning about slow training — and it means it.

The two quantizations — never confuse them

You already know quantization from the inference stage: FP8 and AWQ on your vLLM endpoint, chosen to fit more KV cache and serve more tokens per second. This stage introduces a different use of the same word, and conflating them is the single most common confusion at this point in the curriculum.

03 · reality check The numbers, on your hardware

Worked example: a ~4B Gemma on a 24 GB L4

Term by term, with 4.0B parameters and the activation assumptions from Fig 1.2 (micro-batch 2, sequence 2048, gradient checkpointing on, SDPA attention).

Worked example: how many parameters does LoRA actually train?

The formula is small enough to do in your head. For one linear layer of shape d_in × d_out, LoRA at rank r adds r × (d_in + d_out) parameters. Sum over every targeted layer.

Using an illustrative 4B-class decoder — hidden 2048, 32 layers, intermediate 8192, grouped-query attention with k/v projections at 512:

Worked example: what does the run actually cost?

Take 20,000 SFT examples averaging ~900 tokens after templating, packed into 2048-token sequences, three epochs. That is ~18M tokens per epoch, ~54M tokens total.

What teams actually did

04 · lab Apply to my stack

Two jobs. Measure your real parameter count and memory footprint instead of trusting the table above, and get a 4-bit Gemma loaded and inspected on an L4 before you go anywhere near a trainer.

Lab 1.1 — the memory ledger, on the real model

# memory_ledger.py — run this on the L4 before planning any run.
import torch, json
from transformers import AutoModelForCausalLM, AutoConfig, BitsAndBytesConfig

MODEL = "google/gemma-4-E4B-it"   # your served base
GPU_GB = 24

def ledger(n_params, n_trainable, bytes_per_weight, opt_bytes=8, grad_bytes=2, act_gb=1.5):
    w  = n_params    * bytes_per_weight / 1e9
    g  = n_trainable * grad_bytes       / 1e9
    o  = n_trainable * opt_bytes        / 1e9
    return {"weights": w, "gradients": g, "optimizer": o,
            "activations": act_gb, "peak": w + g + o + act_gb}

# --- 1. the true stored parameter count (NOT the "E4B" marketing number) ---
cfg = AutoConfig.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16,
                                             device_map="cuda:0")
n = sum(p.numel for p in model.parameters)
print(f"stored parameters: {n/1e9:.2f}B")
print(f"bf16 footprint:    {model.get_memory_footprint/1e9:.2f} GB")

# --- 2. the embedding table you will NOT be able to quantize ---
emb = model.get_input_embeddings.weight.numel
print(f"embedding params:  {emb/1e6:.0f}M  ({emb/n:.1%} of the model)")
print(f"  → stays 16-bit:  {emb*2/1e9:.2f} GB inside your '4-bit' model")

# --- 3. the three profiles, with YOUR numbers ---
for name, bpw, trainable in [
        ("full FT  (fp32+Adam)", 4,   n),
        ("LoRA r16 (bf16 base)", 2,   22_500_000),
        ("QLoRA r16 (nf4 base)", 0.6, 22_500_000)]:   # 0.6 ≈ nf4 + double-quant overhead
    L = ledger(n, trainable, bpw, grad_bytes=(4 if bpw == 4 else 2))
    fits = "FITS" if L["peak"] < GPU_GB * 0.92 else f"OVER by {L['peak']-GPU_GB:.1f} GB"
    print(f"{name}: peak {L['peak']:5.1f} GB  [{fits}]  "
          f"w={L['weights']:.1f} g={L['gradients']:.2f} o={L['optimizer']:.2f}")

Lab 1.2 — load Gemma in NF4 and confirm what got quantized

# nf4_load.py — the config from Fig 1.4's left column, made real.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

MODEL = "google/gemma-4-E4B-it"
compute_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported else torch.float32

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=compute_dtype,
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    device_map="cuda:0",
    quantization_config=bnb,
    torch_dtype=compute_dtype,   # MUST match compute_dtype or training crawls
    attn_implementation="sdpa",  # linear-in-seq-len activations, not quadratic
)
tok = AutoTokenizer.from_pretrained(MODEL)

print(f"4-bit footprint: {model.get_memory_footprint/1e9:.2f} GB")

# which layers are actually 4-bit, and which escaped?
from collections import Counter
kinds = Counter(type(m).__name__ for m in model.modules
                if "Linear" in type(m).__name__ or "Embedding" in type(m).__name__)
print(kinds)   # expect many Linear4bit + a plain Embedding + a plain Linear (lm_head)

unquantized = [(nm, p.dtype, p.numel)
               for nm, p in model.named_parameters if p.dtype != torch.uint8]
big = sorted(unquantized, key=lambda t: -t[2])[:5]
print("largest unquantized tensors:")
for nm, dt, cnt in big:
    print(f"  {nm:52s} {str(dt):16s} {cnt/1e6:8.1f}M  = {cnt*2/1e9:.2f} GB")

Expect the embedding table at the top of that last list. That is the pale steel block from Fig 1.2, and now you know its exact size on your model.

Session 2 · train

LoRA and QLoRA, all the way down

Session 1 established that gradients and optimizer states are indexed on trainable parameters. This session is the technique built entirely around abusing that fact — and the four knobs that decide whether it works or quietly underperforms.

01 · why The problem this session solves

The memory arithmetic left you in an awkward place. You cannot train all 4 billion parameters. But the parameters you most want to change — the projections inside every attention block — are precisely the big ones you cannot afford to make trainable.

The obvious escape is to freeze most of the network and train only the last few layers, the way people fine-tuned image classifiers for a decade. It works badly here. A language model’s behaviour is distributed through every block; freezing the bottom 90% leaves you adjusting a thin output film over a model that still thinks the way it always did.

LoRA takes a stranger route. Leave every original weight frozen — and add a small, cheap, trainable detour alongside each one. You are not choosing which layers to train. You are training a tiny shadow of all of them.

02 · concepts Core concepts, from zero

The low-rank hypothesis

Think about what fine-tuning actually asks of a weight matrix. You are not rebuilding the model’s understanding of language — that already works. You are nudging it: be terser, always emit JSON, use the register of a sell-side analyst. Those are narrow, systematic adjustments.

Here is the analogy worth keeping. A published source has been fully typeset; every letter is set. To issue a corrected edition you do not re-typeset the material — you print an errata sheet. The corrections are systematic (“wherever we wrote *colour*, read *color*”), so the sheet is a page long even though the material is 600. The reader carries both and applies one to the other.

LoRA is the errata sheet. The base model is the printed source, frozen. The adapter is a tiny set of systematic corrections applied on top. The low-rank hypothesis is the claim that the corrections fine-tuning wants to make really are that systematic — that the update matrix has low rank, meaning most of its rows are just combinations of a few underlying patterns rather than 4,096 independent ones.

Concretely. A projection inside an attention block might be a 2048 × 2048 matrix — about 4.2 million weights. Full fine-tuning learns a 4.2-million-element update to add to it. LoRA instead learns two skinny matrices whose product has the same shape:

The forward pass never builds that big product. Matrix multiplication distributes, so instead of computing x @ (W + BA)ᵀ you compute x @ Wᵀ and x @ Aᵀ @ Bᵀ separately and add the results. Two cheap passes through skinny matrices, one addition. That is also why the adapter can be switched off at inference — just skip the second path.

The four knobs

A LoraConfig is short. Every field in it is load-bearing.

# the artifact IS the mechanism — this object is the whole technique
cfg = LoraConfig(
    r=16,                          # rank — the bottleneck width
    lora_alpha=32,                 # scaling numerator; effective gain = alpha/r
    lora_dropout=0.05,             # dropout on the adapter path only
    target_modules="all-linear",   # which layers get a detour
    bias="none",                    # leave biases frozen so the base stays recoverable
    task_type="CAUSAL_LM",
)

The rank recommendation deserves scrutiny rather than obedience, because the memory arithmetic from Session 1 does not stop applying just because a good paper said 256.

QLoRA: assembling the parts

QLoRA is not a new algorithm. It is LoRA plus a specific set of memory economies, chosen so that the pieces do not fight each other. The insight: since the base is frozen anyway, its precision matters far less than you would expect — nobody is computing gradients for it. So compress the thing you are not training, and keep full precision only for the thing you are.

Merge, or keep separate?

Training gives you a base plus an adapter. Two artifacts. Before serving you must choose whether to fold them together permanently.

Merging computes W + (alpha/r)·B·A for every adapted layer and writes the result as ordinary weights. You get one self-contained model with zero adapter overhead — and you can no longer switch the behaviour off, swap it, or serve a second one beside it.

Keeping them separate means the serving engine loads one base and applies adapters per request. This is multi-LoRA serving: many tuned behaviours on one GPU’s worth of base weights.

03 · reality check The numbers

What the current evidence actually says

A configuration you can defend

04 · lab Apply to my stack

Lab 2.1 — the QLoRA config for Gemma on the L4, with the parameter count printed

# qlora_setup.py — everything from Fig 2.5, assembled and verified.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

MODEL = "google/gemma-4-E4B-it"
compute_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported else torch.float32

# --- part 1 + 2: NF4 base with double quantization ---
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=compute_dtype,
)
model = AutoModelForCausalLM.from_pretrained(
    MODEL, device_map="cuda:0", quantization_config=bnb,
    torch_dtype=compute_dtype, attn_implementation="sdpa",
)

# --- part 3: numerical hygiene. Do this BEFORE attaching adapters. ---
model = prepare_model_for_kbit_training(
    model, use_gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
)

# --- part 4: bf16 adapters, targeting every linear layer ---
lora = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",   # NOT the PEFT per-architecture default
    bias="none",                  # keeps the base exactly recoverable
    task_type="CAUSAL_LM",
    # use_rslora=True,          # switch on if you go above r≈64
    # modules_to_save=["lm_head","embed_tokens"],  # ONLY if adding new tokens (S3)
)
model = get_peft_model(model, lora, adapter_name="fintech-json-v1")

# --- verify before you spend five hours ---
model.print_trainable_parameters
print(f"footprint: {model.get_memory_footprint/1e9:.2f} GB")

# which modules ACTUALLY received an adapter? assumptions die here.
adapted = sorted({nm.split(".lora_A")[0].split(".")[-1]
                  for nm, _ in model.named_parameters if "lora_A" in nm})
print("adapted modules:", adapted)
assert "down_proj" in adapted, "MLP not adapted — target_modules did not apply"

That final assertion is the point of the lab. The most common silent LoRA failure is a config that looked right and quietly adapted four attention projections.

Lab 2.2 — a rank sweep that reports memory as well as loss

# rank_sweep.py — sweep r, hold everything else fixed, record the loss FLOOR.
# Because LoRA's optimal LR is ~rank-independent, one LR works across the sweep.
import torch, json, gc

RANKS = [8, 16, 32, 64]
results = []

for r in RANKS:
    torch.cuda.reset_peak_memory_stats
    model = build_qlora_model(r=r, alpha=2*r, rslora=(r > 64))  # from Lab 2.1
    trainer = build_trainer(model, max_steps=300)                # short probe, not a full run
    trainer.train

    hist = [h for h in trainer.state.log_history if "eval_loss" in h]
    results.append({
        "rank": r,
        "trainable": sum(p.numel for p in model.parameters if p.requires_grad),
        "best_eval_loss": min(h["eval_loss"] for h in hist),
        "peak_gb": torch.cuda.max_memory_allocated / 1e9,
        "adapter_mb": sum(p.numel for n, p in model.named_parameters
                            if "lora_" in n) * 2 / 1e6,
    })
    del model, trainer; gc.collect; torch.cuda.empty_cache

print(json.dumps(results, indent=2))
# READ IT LIKE THIS:
#   loss floor keeps dropping as r rises  -> you are capacity-limited, go higher
#   loss floor flattens between two ranks -> take the LOWER one, the rest is waste
#   loss floor gets WORSE at high r       -> instability; set use_rslora=True and retry

Session 3 · data

Data and SFT in practice

The memory maths is solved and the adapter is configured. Every remaining way this run can fail is a data failure — and the worst of them produces no error, no warning, and a loss curve that looks fine.

01 · why The problem this session solves

Here is a failure that has cost more GPU-hours than any OOM. A team fine-tunes a small Gemma on a clean, curated, domain-specific dataset. The hardware is happy. The loss curve descends smoothly. Training completes. Then they query the model and it produces circular nonsense — the reported example is a medical model answering “anemia is anemia; anemia is caused by anemia”.

Nothing was broken. Three tokens were missing.

Every instruction-tuned model ships with a chat template: a precise sequence of special tokens marking who is speaking and where the model’s turn begins. It is not a formatting preference. It is a contract burned into the weights during the model’s own instruction tuning. Train on data that does not honour it and you are teaching the model a second, conflicting convention — while at inference time it still expects the first.

02 · concepts Core concepts, from zero

From rows to a training batch

Walk the pipeline once. Every stage has a characteristic failure, and knowing which stage produced a symptom is most of debugging SFT.

<bos><start_of_turn>user
Analyse the risk factors in this filing.<end_of_turn>
<start_of_turn>model
{"sentiment":"negative","risk_score":0.71}<end_of_turn>

Curation and size: how much data, and how good?

SFT is demonstration learning. Every row is you saying “when you see this, do that.” The model has no way to know which parts of your demonstration were the point — so if a fifth of your examples demonstrate something slightly different, you have taught inconsistency and it will reproduce it faithfully.

Which reframes the size question. You are not gathering volume. You are gathering agreement.

Reading the loss curve

Your evaluation stage already covered loss, perplexity and checkpoint selection as concepts. Here is where they get cashed in — as an operational skill, watched live, with a decision attached to each shape.

03 · reality check The numbers

Sizing your actual run

Take 20,000 fintech SFT examples. A filing excerpt plus instruction runs ~700 tokens; the JSON completion ~200. Call it 900 tokens per example after templating.

Two consequences fall out of those numbers. First, at ~1,650 steps an eval_steps of 50 gives you 33 evaluation points — enough resolution to see a turn. An eval_steps of 500 gives you three, and you will miss it. Second, the packing saving is real: at 900 tokens average packed into 2048, padding each example to the batch maximum would waste roughly half of every batch on PAD tokens — and that waste is paid in activation memory, the one term LoRA never reduced.

The effective batch size trap

Your micro-batch is limited by VRAM; your effective batch is what the optimizer sees:

effective batch = per_device_train_batch_size × gradient_accumulation_steps × num_devices

On one L4 with a micro-batch of 2 and 8 accumulation steps, that is 16. Which is comfortably inside the current recommendation to keep LoRA’s effective batch below about 32 — a finding that surprised the people who measured it, and which raising the rank does not fix. If you scale to multiple GPUs in Session 6, remember that num_devices multiplies in, and an eight-GPU job at the same per-device settings gives you an effective batch of 128. Turn the accumulation steps down when you scale out, or you will silently leave the regime where LoRA works well.

Gemma-specific gotchas, from the field

04 · lab Apply to my stack

Lab 3.1 — build the fintech SFT set, and assert the template is right

# build_sft.py — pipeline output -> chat-templated, verified training data.
import json
from datasets import Dataset
from transformers import AutoTokenizer

MODEL = "google/gemma-4-E4B-it"
tok = AutoTokenizer.from_pretrained(MODEL)

# ---- 0. THE PRE-FLIGHT CHECKS. Never skip these. ----
assert tok.chat_template is not None, "no chat template on this tokenizer"
if tok.pad_token_id == tok.eos_token_id:
    # EOS/PAD collision: padding is masked from the loss, so EOS would be too,
    # and the model would never learn to stop. Point PAD somewhere else.
    tok.pad_token = tok.unk_token or "<pad>"
    print("WARNING: EOS==PAD, remapped PAD ->", tok.pad_token)

SYSTEM = ("You are a sell-side equity analyst. Read the filing excerpt and return "
          "ONLY a JSON object with keys: sentiment, risk_score, signal, compliance_flags.")

def to_messages(row):
    # Gemma has no dedicated system role — fold instructions into the first user turn.
    # Check this against YOUR checkpoint's template rather than assuming.
    return {"messages": [
        {"role": "user",
         "content": f"{SYSTEM}\n\n---\n{row['filing_excerpt']}"},
        {"role": "assistant",
         # canonical form: sorted keys, no stray whitespace. Consistency IS the lesson.
         "content": json.dumps(row["structured_output"], sort_keys=True,
                              separators=(",", ":"))},
    ]}

rows = [json.loads(l) for l in open("fintech_pairs.jsonl")]
ds = Dataset.from_list(rows).map(to_messages,
        remove_columns=["filing_excerpt", "structured_output"])

# ---- 1. QUALITY GATE: every completion must be valid against the schema. ----
#      Unfiltered synthetic data is worse than a smaller filtered set.
import jsonschema
SCHEMA = json.load(open("signal_schema.json"))

def valid(ex):
    try:
        jsonschema.validate(json.loads(ex["messages"][1]["content"]), SCHEMA)
        return True
    except Exception:
        return False

before = len(ds); ds = ds.filter(valid)
print(f"schema filter: {before} -> {len(ds)}  ({1-len(ds)/before:.1%} dropped)")

# ---- 2. RENDER AND LOOK AT IT WITH YOUR OWN EYES. ----
rendered = tok.apply_chat_template(ds[0]["messages"], tokenize=False)
print(repr(rendered))

# ---- 3. THE ASSERTIONS THAT WOULD HAVE CAUGHT FIG 3.1 ----
ids = tok.apply_chat_template(ds[0]["messages"], tokenize=True)
assert rendered.count(tok.bos_token) == 1, "double BOS — do not re-tokenize with add_special_tokens"
assert "<start_of_turn>model" in rendered, "no generation prompt: model won't know it's its turn"
assert rendered.rstrip.endswith(("<end_of_turn>", tok.eos_token)), "no terminator: model won't stop"

# ---- 4. token budget, so max_length is measured rather than guessed ----
lens = [len(tok.apply_chat_template(m, tokenize=True)) for m in ds["messages"][:2000]]
lens.sort
print(f"tokens/example  p50={lens[len(lens)//2]}  "
      f"p95={lens[int(len(lens)*.95)]}  max={lens[-1]}")
print("-> set max_length just above p95; truncating the tail beats paying for it on every batch")

ds.train_test_split(test_size=0.05, seed=42).save_to_disk("fintech_sft")

Lab 3.2 — the SFTConfig, TRL v1 syntax, tuned for a 24 GB L4

# train_sft.py — pairs with Lab 2.1's model. TRL v1.x API.
from datasets import load_from_disk
from transformers import EarlyStoppingCallback
from trl import SFTConfig, SFTTrainer

ds = load_from_disk("fintech_sft")

cfg = SFTConfig(
    output_dir="gemma-fintech-json-v1",

    # --- memory (Session 1's four terms, as arguments) ---
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    per_device_train_batch_size=2,      # micro-batch: VRAM decides this
    gradient_accumulation_steps=8,      # effective batch 16 — under the ~32 ceiling
    per_device_eval_batch_size=4,       # no grads at eval, so this can be larger
    bf16=True,

    # --- data ---
    max_length=2048,                    # from the p95 measured in Lab 3.1
    packing=True,
    packing_strategy="wrapped",
    assistant_only_loss=True,           # completions-only; v1 patches the template if needed

    # --- schedule ---
    num_train_epochs=3,                  # small dataset -> 3-4; watch the eval turn
    learning_rate=2e-4,                  # LoRA wants MORE than full FT's 2e-5
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    optim="paged_adamw_8bit",            # negligible saving under LoRA; paging is the point

    # --- the instrument panel: 33 eval points across ~1,650 steps ---
    eval_strategy="steps",   eval_steps=50,
    save_strategy="steps",   save_steps=50,
    load_best_model_at_end=True,        # WITHOUT THIS you ship the last checkpoint
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    save_total_limit=3,
    logging_steps=10,
    report_to="tensorboard",
    seed=42,
)

trainer = SFTTrainer(
    model=model,                          # the PEFT model from Lab 2.1
    args=cfg,
    train_dataset=ds["train"],
    eval_dataset=ds["test"],
    processing_class=tok,                 # v1: NOT tokenizer=
    callbacks=[EarlyStoppingCallback(early_stopping_patience=5)],
)

# ===== THE CHECK THAT SAVES THE RUN — decode a real batch before training =====
batch = next(iter(trainer.get_train_dataloader))
labels = batch["labels"][0]
supervised = labels[labels >= 0]
print("=== TOKENS THE MODEL IS GRADED ON ===")
print(tok.decode(supervised))
# Expect: the JSON completion and its terminator. NOT the filing text.
# All -100  -> mask matched nothing.  Everything visible -> completions-only is off.
assert len(supervised) > 0,               "every label masked — nothing to learn from"
assert len(supervised) < len(labels) * 0.8, "prompt does not appear to be masked"

trainer.train
trainer.save_model("gemma-fintech-json-v1")   # adapter only, ~45 MB

Those three lines that decode labels[labels >= 0] are the highest-value code in this session. They catch template bugs, mask bugs and the EOS/PAD collision, before you spend five hours finding out the slow way.

Session 4 · represent

The representation side — when you should tune the embedder, not the generator

Half the "fine-tuning" problems that land on your desk are not generation problems at all. They are retrieval problems wearing a generation costume: the model writes fine, but it was handed the wrong context. This session is about recognising that case and tuning the cheaper model — the embedder.

04 · why The problem this session solves

Your RAG pipeline returns confident answers built on the wrong paragraphs. The instinct is to fine-tune the generator to "understand the domain." But the generator never saw the right evidence — no amount of tuning fixes what retrieval never fetched. The lever that actually moves the metric is the embedding model: the network that turns queries and documents into vectors. It is typically 100–400× smaller than your generator, which means the entire memory-wall arithmetic from Session 1 collapses into "fits on the GPU you already have."

04 · what Bi-encoders, cross-encoders, and where tuning lands

A bi-encoder embeds query and document independently, so document vectors can be pre-computed and searched at scale — this is your index. A cross-encoder reads query and document together and scores the pair — far more accurate, far too slow to run over a corpus, perfect as a reranker over the top-k. Fine-tuning applies to both, but the training signal differs: bi-encoders learn with contrastive objectives (pull the right pair together, push wrong pairs apart), while cross-encoders train like ordinary classifiers on relevance labels.

The contrastive recipe has one ingredient that decides success more than any hyperparameter: hard negatives. A random wrong document teaches the model almost nothing — it was already far away. A document that looks right (same product name, wrong version; same statute, wrong year) forces the embedder to encode the distinction your domain actually cares about. Mining hard negatives from your own retrieval logs — documents the current system ranked highly but users rejected — is the highest-yield data work in this whole stage.

04 · how LoRA applies here too — and the run is tiny

Everything from Session 2 transfers: attach low-rank adapters to the attention projections of the embedding model, freeze the base, train on (query, positive, negatives) triplets with an InfoNCE-style loss. A 0.3B–1.5B embedder with LoRA trains comfortably inside a single consumer GPU, in minutes to hours rather than days. Batch size matters more than usual — contrastive learning uses the other items in the batch as free negatives, so bigger batches teach faster. If memory pinches, gradient-cache techniques recover large effective batches at the cost of a second forward pass.

04 · prove Evaluation is recall, not vibes

Generation quality metrics do not apply here. Build a small gold set of (query → correct document) pairs from real traffic — even 200 pairs is enough to see movement — and report recall@k and MRR before and after tuning. The number that matters downstream is recall at your pipeline's actual k: if your generator reads 5 chunks, recall@5 is the truth. A tuned embedder that lifts recall@5 from 0.61 to 0.83 will do more for answer quality than any generator-side fine-tune, at a fraction of the cost. Stage 04 picks this thread up and runs the full retrieval-quality playbook.

Session 5 · compress & serve

Compress & serve — from adapter checkpoint to tokens per second

Training ends with a folder of adapter weights. Nobody can use a folder. This session walks the artifact from checkpoint to a served endpoint: merge or hot-load, quantize with your eyes open, and re-run the evaluation gate before anything touches traffic.

05 · why The problem this session solves

The run finished, the loss curve looks beautiful, and now three decisions stand between you and production: whether to merge the LoRA into the base weights or serve it as a detachable adapter; whether to quantize, and to what precision; and how to package the result so the serving layer can load it deterministically. Each choice trades quality, latency, VRAM, and operational flexibility against the others — and each one is reversible only if you kept the pieces.

05 · what Merge vs. hot-load — one model or many personalities

Merging folds the low-rank update into the base matrices, producing a single ordinary checkpoint: simplest to serve, zero runtime overhead, but the base is now permanently "flavored" — serving ten variants means ten full copies. Runtime adapters keep the base frozen and attach LoRA weights per request: modern servers like vLLM can multiplex many adapters over one base, paying a small latency tax for enormous VRAM savings. The decision rule is simple: one product, one voice → merge; many tenants, many voices → hot-load. Either way, keep the unmerged adapter in artifact storage forever — it is your ability to re-merge onto a newer base later.

05 · how Quantization after tuning — the honest sequence

Quantize after the merge, then evaluate — never trust that a recipe which was harmless on the base model is harmless on your tuned one. Weight-only 4-bit schemes in the GPTQ/AWQ family typically cost a point or less on general benchmarks while cutting VRAM roughly 3–4×; your narrow fine-tuned skill can be more fragile than the average benchmark, which is exactly why the gate below exists. The arithmetic callback to Session 1: a merged 8B model at 4-bit weighs in near 5 GB — the same hardware that could barely train with QLoRA now serves with headroom for a real KV-cache.

05 · prove The gate: same eval, new artifact

Re-run the exact evaluation set from your training loop against the served artifact — merged, quantized, behind the real inference stack — not against the training-time checkpoint. Differences here are not noise; they are the cost of every packaging decision you just made, measured. Only when the served numbers clear the bar does the artifact earn a version tag and a place in the registry. Stage 05 goes deep on the serving stack itself; Stage 06 wraps it in an API worth exposing.

Session 6 · operate

Fine-tuning as a Kubernetes Job — the interchange preview

On your laptop, training is a script. In a team, it is a workload: schedulable, restartable, observable, and gone when it finishes. Kubernetes Jobs are the smallest honest way to run one — enough platform to run a real tune on a cluster without pretending to be a platform course.

06 · why The problem this session solves

The GPU box under the desk has three problems: someone else wants it, nothing restarts your run when it dies at 3 a.m., and nobody can reproduce what you did. A Kubernetes Job answers all three: it requests a GPU through the scheduler, restarts on failure with a policy you chose, and its YAML is the reproduction recipe.

06 · what The five lines that matter in the manifest

A training Job is an ordinary Job with a handful of load-bearing fields. resources.limits."nvidia.com/gpu": 1 is the actual GPU request — schedulers treat GPUs as extended resources, and requests equal limits for them. A nodeSelector or toleration steers the pod onto GPU nodes (which are usually tainted so CPU workloads stay off). backoffLimit caps retries; activeDeadlineSeconds caps runaway spend. And the container's command is your training script — the same one from Session 3, now pinned inside an image.

06 · how Checkpoints make preemption boring

Cheap GPU capacity is preemptible capacity, so design for interruption instead of hoping. Write checkpoints (model, optimizer, step counter) to object storage every N steps; on start, the script checks for the latest checkpoint and resumes. With that in place, a spot eviction costs you minutes, not the run — and the Job's restart policy turns "the node vanished" into "the pod moved." Secrets (your model-hub token, bucket credentials) come from Kubernetes Secret objects mounted as env vars, never baked into the image. The finished adapter is pushed to the registry as an artifact, closing the loop with Session 5's packaging.

06 · prove Where this thread continues

You have now touched images, scheduling, GPU resources, and secrets — one honest slice of the platform. The production treatment lives in Stage 08 — K8s & Infra for LLMs: model weights as cluster data, GPU scheduling at fleet scale, and the serving topologies that carry real traffic. And the cloud that all of it runs on is the Platform track — accounts to GPUs to pipelines.