Tab 00 · Stage overview & map

RAG is an open-book exam. The retriever decides what's on the desk.

Generation can only lose quality from whatever retrieval hands it. A perfect generator over the wrong three chunks produces a fluent, confident, wrong answer — and every metric on your generation dashboard will look fine. That single asymmetry is the spine of all seven sessions: fix the desk before you blame the student.

MAP

Every RAG system is two loops

click any box

Almost every RAG confusion dissolves once you separate the two loops. One runs when documents arrive and is allowed to be slow and expensive. The other runs on every user request and has a latency budget measured in hundreds of milliseconds. They share exactly one artifact — the index, the searchable store built by the offline loop and read by the online loop. Every design decision in this stage is really a decision about which loop pays.

Click a box. Each node in the map is one session's home turf — the badges tell you which.

Read it as: Loop A is a batch job you own; Loop B is a request handler your users feel. The index is the contract between them — change the chunker or the embedding model and every vector must be rebuilt.

WHY

RAG is a choice, not a default

You already met the prompt-vs-RAG-vs-fine-tune tree in the prompt-engineering stage. Long context has since become a serious fourth option, so the honest framing is a four-way choice. All four inject knowledge; they differ in when the knowledge enters and who pays for it.

at sources time

Prompt / few-shot

Knowledge is baked into the prompt by you
Zero infra, zero freshness
Ceiling: whatever fits and stays stable
Use when the knowledge is small, fixed and rarely changes

at query time

RAG

Knowledge is fetched per request
Fresh, attributable, access-controllable
Ceiling: retrieval quality
Use when the corpus is large, changing, or must be cited

at prompt time

Long context

Whole documents pasted in
No chunk-boundary loss; no infra
Ceiling: cost, latency, and context rot
Use when the corpus is small, static and each query needs most of it

at train time

Fine-tune

Knowledge is pressed into weights
No retrieval latency, no citations
Ceiling: staleness, and it teaches behaviour better than facts
Use when you need format, tone or task shape — not lookups

The heuristic that survives contact with production

Facts belong in retrieval. Behaviour belongs in fine-tuning. If the answer changes when a new 10-Q drops, it is a retrieval problem. If the answer's shape is wrong — wrong format, wrong hedging, wrong risk vocabulary — that is a fine-tuning or prompting problem. Session 1 turns this into a full guard-clause tree; the rest of the stage assumes you already chose RAG.

Where the materials have aged — mid-2026 reality

All four sources write as if "RAG vs long context" is still an open argument. As of 2026 the argument has largely settled into hybrid: retrieve a bounded but generous slice, then long-context-reason over it. Two findings drove that. First, cost: retrieval-based answers land at a fraction of a cent per query while full-corpus stuffing runs into dollars, and published comparisons put the gap in the three-to-four-orders-of-magnitude range at realistic corpus sizes. Second, quality: Chroma's context rot work (July 2025) tested 18 frontier models and found accuracy degrading well before the advertised window is full — a 200K window can wobble at 50K of input. The window is a capacity, not a competence.

Context rot · Chroma Research, Jul 2025 Degradation of long-context accuracy across 18 models, well below advertised limits — the empirical basis for "retrieve, then reason".
research.trychroma.com/context-rot

Cost framing · practitioner analyses, 2026 Independent write-ups converge on RAG being roughly three orders of magnitude cheaper per query than full-corpus long context at enterprise corpus sizes, with latency differences of the same flavour.
usewire.io — long context vs RAG, Jun 2026

GOAL

What you'll be able to defend at the end

Design

Pick a parser and prove it on your worst PDFs
Pick a chunking strategy and say what it costs
Pick an embedding model on your corpus, not MTEB
Pick a vector store from a defensible tree

Retrieve

Hybrid dense + sparse with RRF, tuned
Metadata filters that don't silently gut recall
A reranker sized to your latency budget
Query rewriting / HyDE only where it pays

Escalate

Know when naive RAG is genuinely enough
Know when to add a grading loop (CRAG-style)
Know when a knowledge graph earns its indexing bill
Know the latency multiplier of each rung

Operate

Component-wise eval wired into your harness
Context budget and prefix-cache-safe assembly
Index refresh, ACLs, and drift detection
A capstone design with every choice justified

PLAN

How the seven sessions build

click a card to open the tab

The order is not arbitrary. S1–S3 build Loop A and the index. S4 makes Loop B actually work. S5 is the escalation ladder you climb only when S4 has run out. S6 tells you whether any of it helped. S7 keeps it alive in production and puts the whole thing together on your fintech corpus.

HOW

How each session tab is built

The real-world problem, in plain words

Every tab opens with the failure that made this topic exist — not a definition. If you can't name the failure, the technique is cargo cult.

Concepts before code

Full working code, pipeline structure and anything about your Gemma-on-Modal stack lives only in each tab's Lab section. The one exception is short snippets inside a mechanism explanation where the config is the concept — a chunker's parameters, the RRF formula, a filtered retrieval query. Everything else is taught in prose, diagrams and numbers first.

Colour is meaning, not decoration

dense / semantic / vector is teal. sparse / lexical / BM25 is violet. fusion, reranking, budgets and aged source claims are amber. Failure modes are red. The colour of a box tells you which half of the retrieval story it belongs to.

Start here

Session 1 builds the mental model this map assumes: why embedding a query and a document with the same model is both the trick and the flaw, why a second model has to re-read the shortlist, and how to decide whether you should be doing RAG at all.

Session 1

Dense retrieval, and the reason a second model has to re-read everything

The whole of RAG rests on one trick — put queries and documents in the same coordinate space and take the nearest neighbours — and on one flaw in that trick, which is that the query never actually saw the document. Reranking exists because of that flaw. Understand this pair and the rest of the stage is engineering.

You are here — S1 spans both loops at the concept level

Why this session exists

≈10%

A language model that has never seen your 10-Q will still answer questions about your 10-Q. It will produce a fluent paragraph with plausible numbers, a confident tone and no way for you to tell it apart from a correct answer. That is the failure that created RAG: not that models don't know things, but that they don't know what they don't know, and they are equally fluent either way.

Search had already solved half of this. Around 2019 both Google and Microsoft rebuilt parts of their ranking stacks on BERT-style transformers and reported it as one of the largest quality jumps in years. The capability they added has a name — semantic search: matching on meaning rather than on shared words. So when the hallucination problem arrived, the fix was obvious in hindsight: bolt a semantic search engine onto the front of the generator and make it answer from what the search returned. That is RAG.

In regulated work the third box usually justifies the project on its own. A wrong-but-cited answer is auditable. A right-but-uncited answer is not defensible.

Core concepts

≈50%

2.1 · Dense retrieval — search as nearest-neighbour lookup

Pass 1 · intuition

Imagine a library where, instead of shelving sources by subject code, you place every paragraph at a physical point in a very large warehouse, positioned so that paragraphs meaning similar things end up standing near each other. Now someone walks in with a question. You don't read the question and go looking — you work out where the question itself would stand in the warehouse, walk to that spot, and hand over whatever is nearest.

That is dense retrieval: turning the query into a vector with the same model that turned the documents into vectors, then returning the closest document vectors. "Dense" because every dimension carries a bit of signal, in contrast to sparse keyword vectors that are mostly zeros.

Pass 2 · mechanism

You already know what sentence embeddings are. What matters here are the three things that only become problems when you use them for retrieval:

concern 1

Asymmetry

A query and its answer are not paraphrases of each other. "How precise was the science?" and a sentence about astronomers praising the film's accuracy share almost no vocabulary and are not "similar" in the everyday sense. Retrieval models are trained on question–answer pairs specifically to pull those two together. Most modern models expose this explicitly with a prefix or an input_type flag — search_query vs search_document. Getting that flag wrong is a silent, permanent recall loss.

concern 2

Domain shift

An embedding model trained on web and Wikipedia text, pointed at SEC filings, is out of distribution. It has no strong notion that "material weakness" is a term of art or that "10-K Item 1A" means risk factors. This is why domain-specific and fine-tuned embedding models routinely beat generic leaderboard winners on the corpus you actually have.

concern 3

There is always an answer

Ask "what is the mass of the moon?" against a corpus about a film and you get three results with distances. Nearest-neighbour search cannot return nothing. Either you set a distance threshold, or you let a later stage decide, or you ship an answer grounded in irrelevant text.

The red points are the whole problem. With k=4 and no threshold, chunk 41 enters the prompt and the model will happily build a sentence out of it.

Pass 3 · trade-offs & limits

Where dense retrieval loses outright. Exact-match queries. A user searching for CUSIP 037833100, Item 1A, PAYMENT_RETRY_EXHAUSTED or a specific executive's surname is doing lexical lookup, and a 1024-dimensional average of meaning is the wrong instrument. This is not a tuning problem — it is a representational one, and it is the single strongest argument for hybrid search, which S3 and S4 build.

Where it degrades quietly. Long chunks. Every chunk becomes one vector regardless of length, so a 2,000-token chunk covering four topics produces a vector that is near the centroid of all four and close to none of them. Recall drops without any error appearing anywhere.

The fix that isn't free. Fine-tuning the embedding model on your own query–document pairs works, and published domain adaptations report double-digit relative gains. But it means you own a model, an eval set, and a re-embedding job for the entire corpus every time you retrain. Budget for the third one — it is the expensive one.

2.2 · Bi-encoder vs cross-encoder — the central trade in all of retrieval

Pass 1 · intuition

Two ways to run a hiring process. Option A: every candidate writes a one-page profile in advance; when a role opens, you write a one-page profile of the role and match profiles. Fast — the candidate profiles were written months ago — but nobody ever read a candidate's CV with the job description in hand. Option B: for each candidate, an interviewer reads the CV and the job description together and scores the fit. Far more accurate, and completely impossible to do for a million candidates.

Option A is a bi-encoder. Option B is a cross-encoder. Every production search system does A then B: A to shortlist, B to order. That is the entire reason reranking exists, and it is not an optimisation — it is a structural necessity.

Pass 2 · mechanism

The asymmetry in one line: a bi-encoder's document work is done before the query exists, so it scales to a billion vectors. A cross-encoder's work cannot start until the query arrives, so it scales to about a hundred. Classic formulation in the literature: framing ranking as pairwise relevance classification with BERT, often called monoBERT.

Why this is the reason for two stages

Bi-encoder = high recall, mediocre ordering, unlimited scale. Cross-encoder = excellent ordering, no scale. Compose them and you get both: the bi-encoder guarantees the right chunk is somewhere in the top 50; the cross-encoder guarantees it ends up at position 1. Neither can do the other's job. A third architecture, late interaction (ColBERT-style: store one vector per token and match token-to-token at query time), sits between them — better ordering than a bi-encoder, far cheaper than a cross-encoder, at the cost of a much larger index.

Pass 3 · trade-offs & numbers

How large is the reranking lift, really? the source material cites a multilingual retrieval benchmark where adding a reranker moved nDCG@10 — normalised discounted cumulative gain over the top 10, an IR metric that rewards putting relevant results high and discounts them logarithmically as they slide down — from roughly 36.5 to 62.8. That is not a tuning gain; that is a different system. On text-and-table financial documents, a 2026 evaluation reports recall@5 improving from around 0.59 with dense-only retrieval to around 0.82 with hybrid plus reranking, which is the same story with your corpus's shape.

What it costs. A cross-encoder is a model inference in your request path. Practitioner measurements in early 2026 put a self-hosted BGE-reranker-v2-m3 at roughly 50–100 ms on a GPU for a realistic candidate list, and considerably worse on CPU; larger 8B-class rerankers are meaningfully slower and need benchmarking against your budget rather than assuming. Hosted rerankers trade that latency for a network hop and a per-call price.

When to skip it. If your first-stage recall@50 is already near 1.0 and your generator handles 10 chunks as well as 3, reranking buys ordering you don't need. Measure recall@k before you buy a reranker; S6 shows you how.

2.3 · The online loop as a funnel

Query in — 1,000,000 candidates

Everything in the index is a candidate. The raw user question may be verbose, conversational, or refer to earlier turns ("and what about last quarter?"). Anything you do to the query before embedding it — rewriting, decomposing, routing — is the cheapest lever in the whole pipeline, and S4 is built around it.

2.4 · Grounded generation and the citation contract

Pass 1 · intuition

The last step is not "give the model the documents". It is changing the model's job from "answer this question" to "answer this question using only the text below, and tell me which part you used". Those are different tasks with different failure modes, and the prompt is where you switch between them.

Pass 2 · mechanism

The RAG prompt has four parts and their order matters for reasons that have nothing to do with quality:

┌─ STATIC ──────────────────────────────────────┐ ← identical every request │ system: "You are a financial analyst…" │ → prefix-cacheable │ output rules, citation format, refusal rule │ ├─ VARIABLE ────────────────────────────────────┤ ← changes every request │ [doc_3 | AAPL 10-K 2025 | Item 1A] …text… │ → forces prefill │ [doc_7 | AAPL 10-Q Q3'25 | MD&A] …text… │ ├─ QUERY ───────────────────────────────────────┤ │ user: "What new supply-chain risks appeared?" │ └───────────────────────────────────────────────┘

You know from the inference stage that a KV prefix cache only helps up to the first token that differs. Put retrieved context above the static instructions and you have destroyed the cache for every request. Put it below, and the static block stays a cache hit while only the variable part pays prefill. This is a free win and it is routinely thrown away.

The citation contract. Give every chunk a stable, short identifier in the prompt and require the model to emit those identifiers. Now a wrong answer is diagnosable: either it cited a chunk that doesn't support the claim (a generation failure) or it cited correctly from a chunk that shouldn't have been retrieved (a retrieval failure). Without IDs you cannot tell those apart, and S6's whole diagnostic method collapses.

Pass 3 · trade-offs

More context is not more accuracy. Two independent effects work against you. Lost in the middle: models attend best to the start and end of the context and worst to the middle, so the 6th of 10 chunks is the weakest position in the prompt. Context rot: Chroma's 2025 study across 18 models found accuracy degrading with input length even on simple tasks, well before the advertised limit. Practical consequence — order your chunks so the reranker's best hit is first or last, not buried, and treat top-k as a budget to be spent, not maximised.

Grounding does not guarantee faithfulness. A model given correct context can still add an unsupported clause. That is why S6 measures faithfulness separately from retrieval — a high faithfulness score with low context recall is the classic silent failure: coherent answers built on partial evidence.

2.5 · The decision tree: should this even be RAG?

Guard-clause tree · follow “no ↓” until a “yes” exits right

Which knowledge-injection strategy?

Guard 1Does the answer depend on facts that change — filings, prices, news, tickets?

yes →RAG. Nothing else stays fresh without a retrain or a re-paste.

no ↓

Guard 2Must every answer cite a source, or be filtered by who is asking?

yes →RAG. Citations and per-user ACLs live in the retrieval layer; weights and pasted context have neither.

no ↓

Guard 3Is the whole relevant corpus under ~200K tokens and does a typical question genuinely need most of it?

yes →Long context. Paste the documents; skip the pipeline. Re-check when the corpus grows or the bill arrives.

no ↓

Guard 4Is the problem the answer's form — tone, schema, domain vocabulary, refusal behaviour — rather than its content?

yes →Fine-tune (or a better prompt first). Facts in retrieval, behaviour in weights.

no ↓

Guard 5Is the knowledge small, stable, and expressible in a page or two of instructions?

yes →Put it in the system prompt. Cheapest option on the board; prefix-caches perfectly.

no ↓

Default (bottom-left) Build RAG — hybrid retrieval + reranking. It is the default because it is the only option that is simultaneously fresh, attributable, permission-aware and affordable at corpus sizes above a few hundred thousand tokens. The 2026 refinement: it is rarely RAG instead of long context. Retrieve a bounded but generous slice, then let a long-context model reason over it.

Worked example — three fintech questions, three different exits

“Summarise the risk factors added in Apple's latest 10-K versus the prior year.” → Guard 1 fires (new filings arrive) and Guard 2 fires (an analyst will ask which paragraph). RAG, and specifically a version that can filter by ticker and fiscal_year.

“Given this one 90-page credit agreement, list every covenant and flag conflicts.” → Guards 1 and 2 don't fire (single static document, the whole thing is relevant). Guard 3 fires: ~120K tokens, and covenant conflict detection genuinely needs the whole document at once. Long context wins, and chunking would actively hurt by splitting a clause from its exception.

“Every answer must open with a one-line signal, then a confidence band, then evidence.” → Guard 4. That is format, not knowledge. Prompt it; fine-tune only if prompting keeps drifting at scale.

Term ledger · defined in this tab

semantic search: Search by meaning rather than shared words.
dense retrieval: Retrieval by nearest-neighbour lookup over embeddings; every dimension carries signal.
asymmetric search: Queries and documents are different kinds of text; models expose separate query/document encoding modes.
bi-encoder: Encodes query and document separately; document vectors are precomputed. Fast, scalable, ordering is approximate.
cross-encoder: Encodes query and document jointly with full cross-attention, emitting one relevance score. Accurate, unscalable.
late interaction: ColBERT-style middle ground: per-token vectors matched at query time. Better ordering than bi-encoder, bigger index.
reranker: The second-stage model that reorders a shortlist. Almost always a cross-encoder.
first-stage retriever: The shortlisting step — dense, sparse, or hybrid — that feeds the reranker.
nDCG@10: Ranking metric over the top 10 that rewards relevant results high in the list and discounts them logarithmically by position.
grounded generation: Generation constrained to retrieved text, ideally with per-claim citations.
context rot: Measured degradation of model accuracy as input length grows, well before the context limit.
lost in the middle: Positional weakness: information in the middle of a long prompt is used least reliably.

Reality check

≈25%

On paper: why you cannot cross-encode your corpus

Take a modest fintech index — 1 million chunks from filings, notes and news. Assume a small cross-encoder that scores about 700 query–document pairs per second on one L4 when batched. Compare scoring everything against the two-stage funnel.

# NAIVE — cross-encode every chunk pairs = 1,000,000 throughput = 700 pairs/s (small CE, batched, one L4) latency = 1,000,000 / 700 = 1,428 s ≈ 24 minutes per query

# TWO-STAGE — bi-encoder shortlist, then cross-encode 50 embed query = 8 ms (1 forward pass, small embed model) ANN search = 6 ms (HNSW over 1M vectors — S3) rerank 50 = 71 ms (50 / 700 pairs·s⁻¹) assemble prompt = 2 ms TTFT (Gemma) = 340 ms (prefill of ~2.4K ctx tokens on L4)

total to first token = 427 ms → 3,300× faster than naive, same reranker reranker share = 71 / 427 = 17% of the budget

Two things fall out of this arithmetic. First, the funnel is not an optimisation you add later — without it there is no product. Second, the reranker is affordable: at 17% of time-to-first-token it is one of the best accuracy-per-millisecond purchases in the stack, which is exactly why the 2026 consensus is to add hybrid search and a reranker before reaching for anything exotic.

In the wild — three cited data points, 2026

Financial documents · retrieval benchmark, 2026 On T²-RAGBench — 23,088 question–context–answer triples
arxiv.org/pdf/2604.01733 — From BM25 to Corrective RAG

FinanceBench trajectory · 2023 → 2026 The original 2023 FinanceBench evaluation found GPT-4-Turbo with retrieval failing or hallucinating on a large majority of a 150-case sample. A later multi-agent RAG system over SEC filings reported 56% against a 19% baseline for the same setup. The lesson is not that models got better — it is that the retrieval architecture around the model moved the number.
pathway.com — LiveAI for SEC filings

Reranker latency · practitioner measurement, Feb 2026 A hands-on comparison of BGE-reranker-v2-m3, MiniLM cross-encoders and hosted APIs found the open multilingual reranker landing around 50–100 ms once moved onto a GPU — competitive with managed APIs and with no per-call cost — while CPU deployment was the thing that made reranking feel expensive.
docs.bswen.com — best reranker models, Feb 2026

Where the source material has aged

the source material (2024) demonstrates dense retrieval with a Cohere embedding endpoint returning 4,096-dimensional vectors and recommends bge-small-en-v1.5 / gte-small as strong small local models. Both statements were true and both have moved. The current landscape is covered properly in S3, but the headline: the open/API gap has closed and in places inverted, Qwen3-Embedding-8B and successors sit at or near the top of the multilingual boards, Matryoshka-style truncation lets you buy 256 or 512 dimensions instead of 4,096 at a few points of accuracy, and MTEB itself moved to a v2 whose scores are not comparable with the v1 numbers printed in older sources. the source material's concepts — dense retrieval, monoBERT-style reranking, MAP/nDCG — have not aged at all.

Apply to my stack — lab

≈10%

Lab S1 · A two-stage retriever in front of Gemma-on-Modal, with cache-safe prompt assembly

Goal: the smallest honest RAG loop, wired to your existing endpoint, that already does the two things this session argued are non-negotiable — a shortlist-then-rerank funnel, and a prompt whose static prefix survives the KV cache. No vector database yet (that is S3); an in-memory index is fine at this size and keeps the concepts visible.

# rag/s1_two_stage.py — concepts made runnable
from dataclasses import dataclass
import numpy as np, httpx
from sentence_transformers import SentenceTransformer, CrossEncoder

BI    = SentenceTransformer("BAAI/bge-m3")            # bi-encoder: doc vectors precomputed
CROSS = CrossEncoder("BAAI/bge-reranker-v2-m3")      # cross-encoder: query-time only
GEMMA = "https://<your-modal-app>.modal.run/v1"     # vllm serve, OpenAI-compatible

@dataclass
class Chunk:
    id: str; text: str; ticker: str; form: str; section: str; filed: str

# ---------- LOOP A (offline): embed once ----------
def build_index(chunks: list[Chunk]):
    # normalize_embeddings=True so dot product == cosine (see 2.1)
    V = BI.encode([c.text for c in chunks],
                  normalize_embeddings=True,
                  prompt_name="passage")          # ← the asymmetry flag. Do not skip.
    return np.asarray(V, dtype="float32"), chunks

# ---------- LOOP B (online): shortlist -> rerank ----------
def retrieve(q: str, V, chunks, k1=50, k2=5, floor=0.15):
    qv = BI.encode([q], normalize_embeddings=True, prompt_name="query")[0]
    sims = V @ qv                                    # stage 1: 1M dot products, still milliseconds
    top  = np.argpartition(-sims, k1)[:k1]
    cand = [chunks[i] for i in top]

    scores = CROSS.predict([(q, c.text) for c in cand])   # stage 2: k1 forward passes
    ranked = sorted(zip(cand, scores), key=lambda p: -p[1])
    # the "there is always an answer" guard from 2.1, concern 3
    kept = [(c, s) for c, s in ranked[:k2] if s >= floor]
    return kept or []                                   # empty == "I don't know", not "guess"

# ---------- prompt assembly: STATIC first, always ----------
SYSTEM = ("You are a financial analyst. Answer only from the EVIDENCE block. "
          "Cite every claim as [doc_id]. If the evidence does not contain the answer, "
          "reply exactly: INSUFFICIENT EVIDENCE.")     # byte-identical every request

def assemble(q, kept):
    # best hit first and worst hit last: fight 'lost in the middle' (2.4 pass 3)
    ordered = kept[::2] + kept[1::2][::-1]
    ev = "\n\n".join(
        f"[{c.id} | {c.ticker} {c.form} {c.filed} | {c.section}]\n{c.text}"
        for c, _ in ordered)
    return [{"role": "system", "content": SYSTEM},          # ← cached prefix
            {"role": "user",   "content": f"EVIDENCE\n{ev}\n\nQUESTION\n{q}"}]

def answer(q, V, chunks):
    kept = retrieve(q, V, chunks)
    if not kept:
        return "INSUFFICIENT EVIDENCE", []       # cheapest correct answer there is
    r = httpx.post(f"{GEMMA}/chat/completions", timeout=60, json={
        "model": "google/gemma-4-E4B-it",
        "messages": assemble(q, kept),
        "temperature": 0.0,
        "max_tokens": 600,
        "chat_template_kwargs": {"enable_thinking": False},  # grounded QA: no thinking budget
    })
    return r.json["choices"][0]["message"]["content"], [c.id for c, _ in kept]

Three things in there are the session, not boilerplate. (1) prompt_name="query" vs "passage" — the asymmetry flag from 2.1; drop it and you lose recall with no error. (2) The floor on the reranker score — a calibrated cross-encoder score is a far better abstention signal than a cosine distance, because it is trained to mean "relevant", not "nearby". (3) SYSTEM is byte-identical on every call and sits first, so the --enable-prefix-caching you already turned on in gemma_modal.py actually hits. Move the evidence above it and every request pays full prefill.

Watch this in your own telemetry

vLLM exports prefix-cache hit rate. Log it per request alongside retrieved-token count. If hit rate is high but TTFT still climbs with corpus size, your variable block is the problem — that is a top-k and chunk-size question, and S2 plus S4 give you the levers.

Optional exercise Take 20 real questions from your analysts. Run retrieve twice — once returning the raw bi-encoder top-5, once the reranked top-5 out of a 50-candidate shortlist. Don't score the answers yet; just count how often the chunk you consider correct is in each set, and at what position. That hand-built number is the first row of the golden set you formalise in S6, and it will tell you within an hour whether your problem is recall (the right chunk is missing) or ordering (it's there, at position 9).

Bridge → S2

Everything above assumed a list of chunks already existed. That assumption is where most real RAG systems actually fail: the clause got split from its exception, the table became two unrelated tables, and no metric in this session would notice. Session 2 goes back one step and builds Loop A properly — parsing, cleaning, chunking and metadata.

Session 2 · + §4 “Data Preparation”

Ingestion: the failures nobody sees, because nothing throws an exception

Parsing and chunking are the only stage of RAG where a catastrophic bug produces perfectly valid output. A table split across a page boundary becomes two well-formed tables. A clause severed from its exception becomes two coherent paragraphs. The pipeline succeeds, the index builds, and the system now confidently answers the opposite of what the document says.

You are here — Loop A, stages 1–3

Why this session exists

≈10%

the source material opens with a legal-assistant failure worth memorising. A contract clause was split at a fixed character boundary. The half containing the clause landed in one chunk; the half containing a common exception to that clause landed in another. A user asked a question, retrieval returned the first chunk and not the second, and the system answered the exact opposite of what the contract said. Nobody on the team had thought hard about chunking; they had been busy arguing about which LLM to use.

The financial-document version is the same shape. Feed a large annual report through a naive text extractor and a dense exhibit table spanning two physical pages emerges as two unrelated tables: the schema breaks at the page boundary, exhibit numbers lose their descriptions, and when the model is later asked "what filing date goes with Exhibit 4.5?" it produces a confident date that appears nowhere in the document.

Nothing in this pipeline errored. This is why S6 measures retrieval separately — and why S2 is the highest-leverage hour in the stage.

The reframe that makes this session make sense

A chunk is not a piece of a document. A chunk is a unit of evidence that must stand alone, because at query time nothing else will be there to explain it. Every technique below is an answer to one question: how do I make each fragment self-contained without paying to store the whole document next to it?

Core concepts

≈50%

2.1 · Parsing — three families, and why you will use two of them

Pass 1 · intuition

A PDF does not contain a document. It contains drawing instructions: put this glyph at these coordinates in this font. There are no paragraphs, no reading order, no notion that these twelve glyph runs form a table row. Every parser is reconstructing structure that was thrown away at export. That is why parsers disagree, why they disagree most on exactly the documents you care about, and why "just use PyPDF" works until it silently doesn't.

Pass 2 · mechanism

family 1

Heuristic / geometric

pypdf, pdfplumber, PyMuPDF, pdftotext -layout. Read the PDF's own text objects and their bounding boxes; infer columns and rows from x/y positions.

Milliseconds per page, no GPU
Exact when the PDF has a clean text layer
Breaks on: multi-column, borderless tables, rotated headers, scans
pdfplumber stays useful forever as a debugger — it shows you the geometry

family 2

Layout models

Docling, MinerU, Unstructured, Marker. A layout-detection model (DocLayout-YOLO lineage) labels regions — title, paragraph, table, figure, caption — then specialised models read each region; OCR fills scans.

Handles reading order, spanning tables, formulas
Emits typed elements or clean Markdown
GPU helps a lot on Docling and MinerU
Cost: seconds per page, and a model dependency in your ingest job

family 3

Vision-language models

Render the page to an image and ask a VLM to transcribe it as structured Markdown or JSON. the source material's recipes 3.7–3.10 are exactly this.

Best on hostile layouts: charts, infographics, scanned forms
Can describe a figure, which no geometric parser can
Cost: most expensive per page; can hallucinate cells
Use selectively, on pages the cheap parser flagged

The pattern that actually ships

Cascade, don't choose. Cheap parser on every page → a structural confidence check (did we get a text layer? do table rows have consistent column counts? is the character-per-page count plausible?) → escalate only the failures to a layout model, and only their failures to a VLM. Most corpora are 90% easy pages, and a cascade costs roughly the cheap parser's price with roughly the expensive parser's quality.

Tables: the part that decides whether fintech RAG works

A table is the densest information per token in your corpus and the worst possible input to an embedding model. Flattened to text, every quarterly results table in every filing looks nearly identical — the same row labels, the same units, different digits — so dense retrieval cannot tell them apart. Three treatments, usually combined:

1 · keep it as a table

Emit Markdown or HTML with the header row repeated into every chunk of a split table. Never split a table without carrying its header and its caption.

2 · summarise it

the source material recipe 3.9: have a multimodal model write a sentence describing what the table shows. Embed the summary for retrieval; return the table for generation. This is the single most effective fix for table retrieval.

3 · extract it out

If the table is numeric and recurring — income statements, holdings — parse it into rows in Postgres and let the query router send numeric questions to SQL instead of to the vector index (S5).

Pass 3 · trade-offs

Parsing quality is not a leaderboard question, it is a corpus question. A 2026 tool-agnostic evaluation of ten PDF-parsing frameworks across six document categories — financial reports, scientific papers, patents, law — found no single winner; the ranking flips by category. Which is why the one habit that pays: build a ten-document torture set from your worst filings, and diff every candidate parser against it cell by cell before committing.

The cost you'll forget: reparsing. Any parser change invalidates every chunk, which invalidates every embedding, which means a full re-index. Version your parser in chunk metadata from day one (S7 turns this into an operational discipline).

GPU asymmetry. Docling and MinerU accelerate meaningfully on GPU; the open-source Unstructured path does not benefit in the same way. If you are running ingest on your existing L4 capacity, that difference is worth more than a couple of accuracy points.

2.2 · Cleaning — small, boring, and worth more than a model upgrade

Pass 2 · mechanism

Strip the furniture

Running headers, page numbers, footers, "Table of Contents" leaders, watermark text. In filings this can be 10–15% of extracted characters and it is identical across documents, which makes it actively harmful: it pushes unrelated chunks together in embedding space.

Deduplicate boilerplate

Financial corpora are pathologically redundant — the same risk-factor language recurs across issuers and across years. Near-duplicate chunks split your retrieval budget across copies of the same evidence. Hash or MinHash at ingest and keep one, with a list of the documents it appears in.

Expand the jargon

the source material recipe 4.2. Replace or annotate abbreviations and internal terms — YoY, bps, MD&A, CAM, internal desk codes. The embedding model may never have seen your ticker aliases. A 30-line lookup table is a cheaper recall win than any model swap.

Generate hypothetical questions

the source material recipe 4.3. For each chunk, have a small model write 2–3 questions it answers, and embed those alongside the chunk. This attacks the asymmetry problem from S1 directly — you are moving the chunk's vector toward query-space. Costs one cheap LLM call per chunk at ingest.

2.3 · Chunking — six strategies on one axis

Pass 1 · intuition

Chunking is a single trade dressed up in many names. Small chunks are precise but stupid — they match a query sharply and then arrive at the model without enough surrounding context to be useful. Large chunks are informative but blurry — they contain the answer but their single vector is an average of four topics, so retrieval can't find them. Every strategy below is an attempt to get precision at retrieval time and context at generation time, without paying for both.

Pass 2 · mechanism

A · Fixed-size (character or token) splitting

Cut every N characters or tokens, optionally with overlap. the source material recipe 4.4. Deterministic, instant, and completely blind to meaning — it is the strategy that cut the contract clause in half. Its one real virtue is predictability: you know exactly how many chunks you will get and exactly what they cost. Use as: a baseline to beat, and a fallback for formats with no structure at all (chat logs, OCR sludge).

Overlap geometry — what the sliding window actually buys

Overlap is boundary insurance, and insurance has a premium. With size S and overlap O, chunk count scales by S/(S−O): 15% overlap on 512-token chunks is +18% vectors, +18% index RAM, +18% embedding bill, and a higher chance that two near-identical chunks both occupy slots in your top-k.

Where the materials have aged · overlap is no longer automatic

the source material and the source material both present overlap as standard practice, and the common advice is 10–20%. A January 2026 systematic analysis using SPLADE retrieval on Natural Questions found overlap delivering no measurable retrieval benefit while adding indexing cost. That is one benchmark on one corpus shape, not a repeal — but it does mean overlap has moved from "default on" to "measure it". Overlap earns its keep where facts genuinely straddle boundaries (legal clauses, narrative MD&A) and wastes money where they don't (already structured records, tables with repeated headers).

Two 2026 techniques that changed the ceiling

contextual retrieval

Prepend the chunk's place in the world

Before embedding, a small model writes 50–100 tokens describing where this chunk sits in its document ("This is from Apple's Q3 FY2025 10-Q, MD&A, discussing Services revenue…") and prepends it. Anthropic's original write-up reports up to a two-thirds reduction in top-20 retrieval failures when combined with BM25 and reranking; practitioner cost estimates land around $1 per million document tokens to preprocess, and prompt caching over the parent document is what makes that affordable.

Why it works: it directly repairs the "chunk is not self-contained" problem in the vector, not just in the returned text.

late chunking

Embed first, split second

Run a long-context embedding model over the whole document to get token-level embeddings, then pool them into chunk vectors after the fact. Each chunk's vector was computed with the rest of the document attending to it, so it carries context for free. Jina's work reports gains that grow with document length.

Trade: no extra LLM calls (much cheaper than contextual retrieval) but requires a long-context embedding model and a pipeline that can hold a document in one pass. A 2025 comparison found contextual retrieval preserved coherence better while late chunking was substantially more efficient.

Pass 3 · trade-offs & numbers

How much does chunking actually move the needle? Published 2026 comparisons put the swing at up to ~9% recall on the same corpus from chunking choice alone — larger in domains with strong inherent boundaries (contracts, filings with numbered Items) and smaller in flowing prose. That is comparable to a significant embedding-model upgrade, at a fraction of the cost.

Where the ceiling actually is. A January 2026 systematic analysis found sentence-based chunking matching semantic chunking up to roughly 5,000 tokens of context at a fraction of the compute, and identified a "context cliff" around 2,500 tokens where response quality starts dropping. Semantic chunking is real but it is not free and it is not always better — the honest default in 2026 is recursive splitting at 400–512 tokens, then measure before escalating.

Hierarchical is the pattern that won. Small child chunks for retrieval, large parent chunks (or a sentence window) returned for generation, is the most widely adopted production shape in 2025–26 precisely because it dissolves the trade rather than balancing it. the source material's auto-merging retriever (7.5) and sentence-window retriever (7.6) are the query-side half of this; you build the index-side half here.

2.4 · Metadata — the cheapest retrieval feature you will ever ship

Pass 1 · intuition

Semantic similarity cannot express "only Apple", "only 2025", "only risk factors", "only documents this user may read". Those are not fuzzy notions — they are exact predicates, and trying to make an embedding model approximate them is how you get an analyst reading Microsoft's risk factors under an Apple heading. Metadata turns a 1-million-vector search into a 4,000-vector search and makes it correct.

Pass 2 · mechanism

the source material recipe 4.1 makes the point; here is the schema shape that matters for a fintech corpus. Design it once, at ingest, because retrofitting metadata means reprocessing everything.

Field	Type	Why it exists	Used by
chunk_id	str	Citation target; stable across re-embeds	S1 prompt, S6 eval
doc_id, page, bbox	str/int	Deep-link back to the source page for audit	S7 compliance
ticker, cik	str	Hard filter; also the primary KG node key	S4 filter, S5 graph
form_type	enum	10-K / 10-Q / 8-K / analyst_note / news — different trust levels	S4 filter, S5 router
section	enum	Item 1A / Item 7 MD&A / footnote / table — the single highest-value filter in filings	S4 filter
period_end, filed_at	date	Recency and point-in-time correctness; "as of" queries	S4 filter, S7 refresh
acl_labels	str[]	Who may retrieve this. Must be at chunk granularity	S7 access control
content_type	enum	prose / table / table_summary / figure_caption	S4 routing, S6 slicing
parser_version, chunker_version, embed_model	str	Lets you re-index incrementally and diagnose regressions	S7 operations
content_hash	str	Skip unchanged chunks on refresh; detect near-duplicates	S7 refresh

Trap you will hit in S3

Metadata filtering and approximate indexes fight each other. Post-filtering an HNSW result set can silently return far fewer rows than you asked for — the index fetches 40 candidates, your WHERE ticker='AAPL' removes all 40, and the query returns zero. This is the single most common production surprise with pgvector, and S3 covers the fix (hnsw.iterative_scan) in detail. Design metadata now; learn the index interaction there.

2.5 · The decision tree: which chunking strategy?

Guard-clause tree · follow “no ↓” until a “yes” exits right

How should I split this corpus?

Guard 1Do the documents carry explicit machine-readable structure — HTML/XBRL sections, numbered Items, Markdown headings, database rows?

yes →Layout/document-aware splitting. Split on the structure that already exists; never invent boundaries a schema already gave you.

no ↓

Guard 2Is the unit of meaning a table, a form, or a figure rather than a paragraph?

yes →One chunk per table/figure, header repeated, plus a generated summary embedded separately. Do not let a token counter cut a table.

no ↓

Guard 3Do users need precise pinpoint answers and the model needs surrounding context to interpret them?

yes →Hierarchical / parent-child, or sentence-window. Retrieve small, return large. The 2026 production default.

no ↓

Guard 4Is the corpus long-form narrative where topics shift mid-section, and did recursive splitting measurably underperform on your eval set?

yes →Semantic chunking (embedding-distance boundaries), or agentic chunking for high-value low-volume documents. Verify the gain first — it is not free.

no ↓

Guard 5Is the whole corpus under ~200K tokens?

yes →Reconsider chunking entirely — inject whole documents (S1 guard 3). Chunking a small static corpus adds boundary risk for no benefit.

no ↓

Default (bottom-left) Recursive character/token splitting at 400–512 tokens, splitting on paragraph → sentence → word in that order, with overlap set to 0 until measured and 10–15% if boundary straddling shows up in your error analysis. Then layer contextual retrieval (prepended chunk context) if your eval shows fragments being misinterpreted out of context — that layer, not a smaller chunk size, is the fix.

Term ledger · defined in this tab

fixed-size chunking: Cut every N characters/tokens, structure-blind.
recursive splitting: Try progressively finer separators (paragraph → sentence → word) until chunks fit the size budget.
layout-aware / document-aware: Split on the document's own structure: headings, Items, sections, table boundaries.
semantic chunking: Place boundaries where consecutive-sentence embedding distance spikes.
agentic chunking: An LLM reads the document and decides the boundaries. Highest quality, highest cost per document.
hierarchical / parent-child: Index small child chunks; return their larger parent for generation.
sentence-window: Same idea without a second index: retrieve a sentence, return it plus n neighbours.
contextual retrieval: Prepend an LLM-written 50–100 token situating description to each chunk before embedding.
late chunking: Embed the whole document with a long-context model, then pool token embeddings into chunk vectors.
hypothetical questions: Generate the questions a chunk answers and embed those too, to close the query/document asymmetry gap.
metadata filtering: Exact predicates (ticker, section, date, ACL) applied alongside vector similarity.
boilerplate deduplication: Collapsing near-identical recurring text so it doesn't consume the retrieval budget.

Reality check

≈25%

On paper: what one chunking decision costs across the whole corpus

A realistic fintech corpus: 12,000 documents averaging 18,000 tokens — filings, analyst notes, news. That is 216 million tokens. Compare a 512-token chunk with no overlap against a 256-token chunk with 15% overlap, both embedded with a 1024-dimension model.

# corpus tokens = 12,000 docs × 18,000 tok = 216,000,000

# option A — 512 tok, 0% overlap chunks = 216,000,000 / 512 = 421,875 embedding cost = 216M tok × $0.02/1M tok = $4.32 (one pass) index RAM (1024-d fp32) = 421,875 × 1024 × 4 B = 1.73 GB + HNSW graph ≈ 2.2 GB

# option B — 256 tok, 15% overlap (stride 218) chunks = 216,000,000 / 218 = 990,826 (2.35× option A) embedding cost = 254M tok (overlap re-embedded) = $5.08 index RAM = 990,826 × 1024 × 4 B = 4.06 GB + graph ≈ 5.2 GB

# option B + contextual retrieval (80-token prepend, written by a small model) context tokens added = 990,826 × 80 = 79M tokens stored preprocess LLM cost ≈ 216M doc-tok × ~$1.02/1M = ~$220 one-off (cache the parent doc!) index RAM = unchanged — vectors are fixed-width, text is not

verdict the embedding bill is noise; RAM and the contextual-retrieval LLM pass are the real costs — and both are one-off per re-index, which is exactly why S7 cares so much about incremental refresh.

In the wild — three cited data points, 2026

Parser comparison on hostile documents · May 2026 A hands-on comparison of Docling, MinerU and Unstructured, built by rendering bounding boxes on real pages and diffing extracted tables cell by cell. Findings that matter for filings: MinerU is strongest on visually complex PDFs including SEC filings and cross-page table reconstruction; Unstructured optimises for breadth of formats rather than depth of PDF comprehension; GPU acceleration differs sharply between them.
Document parsing for production RAG, May 2026

Chunking benchmarks · 2026 retrieval playbook Consolidates the current evidence: chunking choice can swing recall by up to ~9% on a fixed corpus; the "universal overlap rule" is no longer safe to assume; structure-aware splitting beats naive splitting by a wide margin in domains with strong inherent boundaries. It also puts contextual retrieval's reported figure — around a two-thirds cut in top-20 retrieval failures with BM25 and reranking — next to its ~$1/M-token preprocessing cost.
RAG chunking strategies, Jun 2026

Parsing + chunking, jointly, on financial QA · 2026 An empirical study arguing that PDF parsing and chunking must be evaluated together for a task, not in isolation: misaligned tables and merged columns from the parser and split logical units from the chunker compound, and retrieval then cannot recover the answer at any k.
arxiv.org/pdf/2604.12047

Apply to my stack — lab

≈10%

Lab S2 · Replace your fixed-size chunker with a structure-aware, metadata-rich fintech ingester

You already have fixed-size-with-overlap utilities from the long-context experiments. This lab keeps them as the fallback leaf of a cascade and puts real structure above them, with the metadata schema from 2.4 attached at birth.

# ingest/pipeline.py — Loop A, structure-first
from dataclasses import dataclass, asdict
import hashlib, re
from docling.document_converter import DocumentConverter
from langchain_text_splitters import RecursiveCharacterTextSplitter, MarkdownHeaderTextSplitter

PARSER_V, CHUNKER_V = "docling-2.x", "fin-structural-v3"   # stamped into every chunk

# SEC Item boundaries are real structure — use them before any token counter
ITEM_RE = re.compile(r"^\s*(Item\s+\d+[A-Z]?\.?)\s+(.{0,90})$", re.M | re.I)

@dataclass
class FinChunk:
    chunk_id: str; doc_id: str; text: str
    ticker: str; cik: str; form_type: str; section: str
    period_end: str; filed_at: str; page: int
    content_type: str                # prose | table | table_summary | figure_caption
    acl_labels: list[str]            # enforced in S7, carried from day one
    parser_version: str = PARSER_V
    chunker_version: str = CHUNKER_V
    content_hash: str = ""

def parse(path):
    """Cascade: cheap first, escalate only what fails the structure check."""
    doc = DocumentConverter.convert(path).document
    md  = doc.export_to_markdown
    if _structure_ok(md, doc):
        return md, doc.tables
    return _vlm_reparse(path)          # pages the layout model mangled → multimodal pass

def _structure_ok(md, doc):
    # cheap, corpus-specific tripwires — tune these on YOUR torture set
    if len(md) < 500: return False                       # scanned page, no text layer
    if not ITEM_RE.search(md): return False              # a 10-K with no Items is a broken parse
    for t in doc.tables:                                  # ragged tables = column merge
        widths = {len(r) for r in t.data}
        if len(widths) > 1: return False
    return True

# --- the three-level split: Item -> heading -> recursive fallback ---
recursive = RecursiveCharacterTextSplitter(
    chunk_size=512, chunk_overlap=0,          # overlap OFF until S6 says it helps
    separators=["\n\n", "\n", ". ", " "],
    length_function=lambda s: len(TOK.encode(s)))   # tokens, not characters

def chunk_filing(md, tables, meta):
    out = []
    for item_name, body in _split_on_items(md):        # GUARD 1: real structure exists
        for piece in recursive.split_text(body):
            out.append(_mk(piece, meta, section=item_name, ctype="prose"))
    for t in tables:                                    # GUARD 2: tables are atomic
        table_md = _with_header_repeated(t)             # header travels with every split
        out.append(_mk(table_md, meta, section=t.section, ctype="table"))
        out.append(_mk(_summarise(table_md), meta,      # the source material 3.9: embed the summary,
                       section=t.section, ctype="table_summary"))   # return the table
    return out

# --- contextual retrieval: 80 tokens of "where am I" per chunk ---
CTX_PROMPT = ("Document: {ticker} {form} filed {filed}, section {section}.\n"
              "Write ONE sentence (max 40 words) situating the chunk below inside this "
              "document, naming the company, period and topic. No preamble.\n\n{chunk}")

def contextualize(chunks):
    # Cache the parent document as a shared prefix — this is what makes it ~$1/M
    # instead of ~$15/M. Same prefix-cache logic you tuned for Gemma serving.
    for c in chunks:
        c.text = _small_llm(CTX_PROMPT.format(**asdict(c), chunk=c.text))"\n\n" + c.text
    return chunks

def _mk(text, meta, section, ctype):
    h = hashlib.sha256(text.encode).hexdigest[:16]
    return FinChunk(chunk_id=f"{meta['doc_id']}:{h}", text=text, section=section,
                    content_type=ctype, content_hash=h, **meta)

What each guard in the tree turned into. Guard 1 became _split_on_items — SEC filings have machine-readable structure and a token counter must never override it. Guard 2 became the table branch, with the header repeated and a summary embedded separately. Guard 3 is your next upgrade: promote section to a parent chunk so retrieval hits a paragraph but generation receives the whole Item. Overlap stays at zero until S6 gives you a reason.

Why content_hash and the version stamps matter now

They cost nothing today and they are the difference between a 20-minute incremental refresh and a six-hour full re-index when you change parsers in S7. Chunks whose hash is unchanged keep their vectors; only new hashes get re-embedded.

Optional exercise · build the torture set Pick ten documents from your corpus that you know are ugly: a scanned analyst note, a 10-K exhibit index that spans pages, a filing with a landscape table, an 8-K with an embedded chart, a news page with a cookie banner. Run pdfplumber, Docling and one VLM pass over each. For every one, count three things by hand: rows recovered, header rows preserved, and reading-order errors. That table is your parser decision — and unlike a leaderboard, it is about the documents you actually have.

Bridge → S3

You now have clean, self-contained, metadata-rich chunks. They are still text. Session 3 turns them into the index: which embedding model to trust when the leaderboard changes monthly, how HNSW and IVF actually work and which knobs move recall, and why every serious system also stores a sparse representation next to the dense one.

Session 3 · + §6 “Vector Databases and Similarity Searches”

The index: two irreversible choices and a graph you have to understand

Embedding model and index type are the two decisions in this stage you cannot change cheaply — both invalidate every vector you have stored. Everything else is tunable at runtime. So this session is about making those two choices for reasons you can defend, and about the one data structure, HNSW, whose knobs you will actually turn.

You are here — Loop A, stages 4–5: the artifact both loops share

Why this session exists

≈10%

Two facts make this session different from the rest. First, an embedding is only meaningful relative to the model that produced it. Vectors from two different models are not comparable, not convertible, and not mixable — swapping models means re-embedding the entire corpus and rebuilding the entire index. Second, an approximate index is approximate. It will confidently return five neighbours that are not the five nearest, and nothing in your logs will say so. The gap between "the answer wasn't retrieved" and "the index didn't look hard enough" is invisible unless you deliberately measure it.

Core concepts

≈50%

2.1 · Choosing an embedding model without trusting the leaderboard

Pass 1 · intuition

An embedding model is a compression scheme with an opinion. It has decided what "similar" means, and it decided that during training on somebody else's data. MTEB — the Massive Text Embedding Benchmark — tells you whose opinion generalises best across a fixed set of public tasks. It cannot tell you whose opinion matches an analyst asking about supply-chain risk in a 10-K. Those are different questions and only one of them is answerable from a leaderboard.

Pass 2 · mechanism

Six axes decide the choice. Rank them for your corpus before looking at any scores.

Axis	What it controls	The trap
retrieval quality	Does the right chunk make the top-k at all	MTEB averages eight task families; only the retrieval split is relevant to you. A model can win overall and lose at retrieval.
dimensions	Index RAM, ANN speed, storage — all linear in d	Bigger is not proportionally better. Going 1024 → 3072 triples your RAM for a couple of points.
max sequence length	The largest chunk the model can see without truncating	Silent truncation. If your table chunks are 900 tokens and the model caps at 512, the bottom half never existed.
asymmetry support	Separate query/document modes (prefixes or `input_type`)	Models differ; getting it wrong costs recall with no error (S1, 2.1).
cost model	Per-token API vs GPU-hours self-hosted	The re-embed bill, not the first embed, is what bites. Multiply by how often you expect to change your mind.
licence & residency	Whether text may leave your perimeter at all	In regulated finance this often eliminates the top of the leaderboard before you start.

Matryoshka: buy fewer dimensions without retraining

Matryoshka representation learning trains a model so that the first k dimensions of its output are themselves a usable embedding. That means you can truncate a 3072-dim vector to 512 or 256 and lose only a small amount of accuracy — reported figures cluster around 2–3 points of precision loss for a 4× storage reduction at 256 dims. Most 2026-era models support it. Practical pattern: store full-dimension vectors, index truncated ones, and rescore the shortlist with the full vectors — an adaptive-retrieval trick that gets you small-index speed with large-index accuracy.

Where the materials have aged · the model landscape, July 2026

the source material recommends bge-small-en-v1.5 and gte-small; the source material's §5 recipe is written around the same generation. Since then: MTEB moved to v2, and v2 scores are not comparable to the v1 numbers printed in sources. The open-vs-API gap has closed and in places reversed — Qwen3-Embedding-8B sits around 70.6 on the multilingual board, above several commercial APIs, and vLLM and SGLang both shipped first-class embedding endpoints in Q1 2026, which made self-hosting the default cost path for teams that already own GPUs. Google's Gemini Embedding 001 held the top English MTEB spot at 68.32 through the spring; NVIDIA's Llama-Embed-Nemotron-8B topped the multilingual board as a fully open-weight option. Multimodal embedding — one shared space for text, images and PDFs — went from novelty to product with Cohere Embed v4 and Google's 2026 releases. All of these numbers are self-reported: MTEB accepts vendor submissions with no independent verification step, so treat the leaderboard as a shortlist generator, never as a decision.

self-host, open

Qwen3-Embedding / BGE-M3 / Nemotron

Apache-2.0-class licences, weights stay inside the perimeter
Serve on your existing L4 via vLLM's embedding endpoint
BGE-M3 gives you dense + sparse + multi-vector from one model — convenient for hybrid
Break-even around the 10M-embeddings-per-month mark in most published comparisons

API

Gemini / Cohere / Voyage / OpenAI

No infrastructure, instant start, batch discounts
Domain-specialised variants (finance, legal, code) report 10–15% gains over generic in-domain
Data leaves your network — often the deciding factor, not the score
Re-embedding a large corpus is a real invoice, not a rounding error

NVIDIA stack

NeMo Retriever embedding NIM

Containerised embedding models with TensorRT-LLM-compiled kernels, OpenAI-compatible API
Matryoshka-configurable output dims (e.g. 2048 down to 384/512/768/1024)
Pairs with a reranking NIM that returns per-passage indices for citation attribution
NIM profiles are compiled per GPU architecture — running an H100 profile on an A100 works but loses throughput

Pass 3 · trade-offs

Fine-tuning beats model shopping, eventually. Reported gains from domain fine-tuning land in the +10–30% range for specialised corpora — larger than the gap between the top five leaderboard models. But it adds a training set, an eval set, a retraining cadence, and a full re-embed per retrain. Do it after you have the golden set from S6, never before.

The economics have inverted since the materials were written. Self-hosting an 8B embedding model on capacity you already pay for is now frequently cheaper and higher-scoring than a mid-tier API. The constraint is no longer money, it is whether you want to own an inference service. You already do — that is what gemma_modal.py is — so the marginal operational cost for you is unusually low.

2.2 · Approximate nearest neighbour — HNSW and IVF

Pass 1 · intuition

With ten thousand vectors you compare the query to all of them; that is a matrix multiply and it takes milliseconds. At ten million it is not, and the fix is the same one humans use: don't check everything, navigate.

HNSW — hierarchical navigable small world — is a road network with motorways. The top layer has a handful of nodes connected by very long edges; each layer down is denser with shorter edges. You enter at the top, take the biggest hop that moves you closer, drop a layer, repeat. Six or seven hops and you are in the right neighbourhood having examined a few hundred of ten million points.

IVF — inverted file — is a filing cabinet instead. Cluster all vectors once into nlist cells with a centroid each. At query time, find the nearest few centroids and only search inside those drawers.

Pass 2 · mechanism

Both structures trade recall for speed, at different knobs. HNSW's knob is how many candidates you keep in flight (ef_search). IVF's is how many drawers you open (nprobe). Neither guarantees the true nearest neighbours, and that is the point — 98% recall at 5 ms beats 100% recall at 900 ms in almost every product.

The four knobs you will actually turn

M

Edges per node in HNSW (typ. 16). Higher M = better recall and bigger index; memory grows roughly linearly in M. Build-time only.

ef_construction

Candidate breadth while building (typ. 64–200). Higher = better graph, slower build. Build-time only; costs nothing at query time.

ef_search

Candidate breadth while querying (default 40 in pgvector). The one runtime dial: raise it for recall, pay latency. Must be ≥ your k.

nlist / nprobe

IVF's pair. nlist ≈ √N is the usual starting point; nprobe is the runtime dial. IVF builds 5–6× faster than HNSW but needs data present to learn centroids.

The filtering trap — and why pgvector 0.8 mattered

This is the most common production surprise in the whole stage, so it gets its own worked failure. With approximate indexes, filters are applied after the index scan by default. Suppose hnsw.ef_search = 40 and your filter matches 10% of rows: the index returns 40 candidates, roughly four survive the filter, and your request for 10 results returns four. Worse — if the filter is selective and unlucky, all 40 are removed and the query silently returns zero rows.

-- the failure, exactly as it appears in EXPLAIN SET hnsw.ef_search = 40; SELECT id FROM chunks WHERE ticker = 'AAPL' ORDER BY embedding <=> :q LIMIT 10; -- Index Scan using idx_chunks_hnsw -- Rows Removed by Filter: 40 ← the whole candidate set -- (0 rows) ← your app just showed "no results" -- the fix (pgvector 0.8+): keep scanning until the filter is satisfied SET hnsw.iterative_scan = strict_order; -- or relaxed_order, faster SET hnsw.ef_search = 100; -- and give it room to work

pgvector 0.8 introduced iterative index scans exactly for this, with hnsw.max_scan_tuples and ivfflat.max_probes as the safety limits. It also improved the planner's cost estimates, so Postgres is now more willing to use a B-tree instead of the ANN index when that is genuinely faster — which is a good thing, because a B-tree path gives you 100% recall.

Pass 3 · trade-offs & limits

HNSW's real constraint is memory residency, not disk. The graph is stored on disk but performance depends on the active working set staying in cache. When graph pages get evicted, latency degrades sharply and no parameter fixes it. Size shared_buffers and OS cache so the working set stays resident; if it cannot, move to a disk-oriented index (pgvectorscale / StreamingDiskANN, or Milvus DiskANN) rather than pretending.

Quantisation is the lever that changes the arithmetic. Scalar quantisation (fp32 → int8) is a 4× reduction for typically ~1% recall loss. Product quantisation goes further at a larger cost. Binary quantisation is 32× and needs a rescoring pass over full vectors to be usable. Combined with Matryoshka truncation, a 1M × 1024-dim index can go from ~4 GB to well under 1 GB.

Write churn is HNSW's other enemy. In Postgres, high-frequency re-embedding creates MVCC bloat in the index; this is one of the standard signals that you have outgrown pgvector. The other two are billions of vectors and a hard sub-20 ms p99 requirement.

2.3 · Sparse retrieval — BM25, and why it belongs in your index

Pass 1 · intuition

Dense retrieval knows what words mean. BM25 knows which words are rare. When an analyst types CUSIP 037833100 or Item 1A, meaning is not the signal — the exact rare token is. BM25 is a fifty-year lineage of that idea, refined into one formula, and it is still the strongest single thing you can add to a dense-only system.

Pass 2 · mechanism

Three ingredients, each fixing a flaw in naive keyword counting:

score(q, d) = Σ over terms t in q: IDF(t) × ( f(t,d) · (k₁+1) ) / ( f(t,d) + k₁·(1 − b + b·|d|/avgdl) ) └──┬──┘ └─────────────────┬─────────────────┘ rare terms term-frequency saturation + length normalisation count more (the 10th "revenue" adds almost nothing; long documents don't win by being long) typical: k₁ ≈ 1.2–1.5, b ≈ 0.75

IDF

Inverse document frequency. A term appearing in every filing carries no signal; a term appearing in three carries a lot. This is why BM25 nails identifiers and proper nouns.

Saturation (k₁)

Term frequency has diminishing returns. Without it, a keyword-stuffed page beats the right page.

Length norm (b)

Divides out document length so a 40-page Item 1A doesn't outrank a precise paragraph purely by containing more words.

Learned sparse is the modern middle ground: SPLADE and BGE-M3's sparse head produce a sparse vector over the vocabulary where a transformer decided the weights, including terms that never appeared in the text (term expansion). You get BM25's exact-match behaviour plus some semantics, stored in the same inverted-index machinery. Qdrant treats sparse vectors as a first-class type and frames BM25 as a special case of them; Milvus supports BM25 full-text and SPLADE-style sparse embeddings in the same collection.

Decide this at index time, not query time

Hybrid search is an S4 topic, but the storage decision is made here: you must build a sparse representation alongside the dense one during ingest. Retrofitting BM25 onto a dense-only store means another full pass. Add the tsvector (Postgres) or sparse-vector field (Qdrant/Milvus/Weaviate) now.

Pass 3 · trade-offs

Cost of the second representation: roughly a 1.4× storage footprint and a few milliseconds of extra query latency in published measurements — trivial against what it buys. On financial documents, a 2026 evaluation reports recall@5 moving from about 0.59 (dense only) to about 0.82 with hybrid plus reranking.

Where sparse alone still wins: exact identifier lookup, quoted phrases, and any corpus where users search using the document's own vocabulary. Where it loses: paraphrase, cross-lingual, and questions whose answer shares no words with the question — precisely the source material's "how precise was the science" example, where BM25's top hit contained the word "science" and did not answer the question.

2.4 · The decision tree: which vector store?

Guard-clause tree · follow “no ↓” until a “yes” exits right

pgvector, dedicated, or managed?

Guard 1Do you need billions of vectors, or GPU-accelerated index builds, or hard sub-20 ms p99 at high QPS?

yes →Dedicated. Milvus for extreme scale and GPU index building (cuVS); Qdrant for lowest single-node latency and strong filtering.

no ↓

Guard 2Is high write churn — continuous re-embedding, hourly document turnover — the dominant workload?

yes →Dedicated. Postgres HNSW accumulates MVCC bloat under heavy churn; purpose-built engines handle segment compaction for you.

no ↓

Guard 3Is “nobody on this team should be running a database” a hard constraint, and is the per-month floor acceptable?

yes →Managed. On AWS: Bedrock Knowledge Bases (backends include OpenSearch, Aurora pgvector, S3 Vectors, Neptune Analytics, third parties). On GCP: Vertex AI Vector Search. Check the floor — OpenSearch Serverless has a minimum OCU charge that surprises people.

no ↓

Guard 4Do your queries need to join vectors against relational data in one statement — positions, entitlements, prices, user state?

yes →pgvector, emphatically. One SQL statement instead of “query vectors, get IDs, query the database”. This is pgvector's real advantage and it is not a performance argument.

no ↓

Default (bottom-left) pgvector in the Postgres you already run, with HNSW, halfvec or int8 quantisation, hnsw.iterative_scan enabled, and a tsvector column for BM25. Independent 2026 guidance converges on this for roughly 70% of workloads and up to the 10M–50M-vector range: same backups, same IAM, same migrations, same on-call as the rest of your system. Add pgvectorscale/StreamingDiskANN before you add a second database. Revisit when a guard above starts firing — and measure, don't guess, because every engine here shipped a major release in 2026.

Term ledger · defined in this tab

MTEB: Massive Text Embedding Benchmark. Eight task families; only the retrieval split is directly relevant. v2 scores are not comparable with v1. Self-reported.
Matryoshka (MRL): Training so that a vector's leading dimensions are themselves a valid embedding, enabling truncation.
ANN: Approximate nearest neighbour: trading exactness for speed at large N.
recall@k (index sense): Fraction of the true k nearest neighbours the approximate index actually returned.
HNSW: Layered proximity graph searched by greedy descent. Knobs: M, ef_construction, ef_search.
IVF / IVFFlat: Partition vectors into nlist clusters; search the nprobe nearest at query time.
DiskANN: Graph index designed to stay correct when most of it lives on SSD rather than RAM.
quantisation: Storing vectors at lower precision: scalar (int8, ~4×), product (PQ), binary (~32×, needs rescoring).
pre-filter / post-filter: Applying metadata predicates before vs after the ANN scan. Post-filtering is the default and the cause of empty result sets.
iterative scan: pgvector 0.8+ behaviour that keeps pulling candidates until enough survive the filter.
BM25: Lexical ranking function: IDF weighting × saturated term frequency × length normalisation.
sparse vector: High-dimensional, mostly-zero representation over vocabulary. BM25 is one; SPLADE-style learned sparse adds term expansion.
cuVS: NVIDIA's GPU-accelerated vector search library, used to speed index build and search in Milvus.

Reality check

≈25%

On paper: index RAM for 1M vectors, four ways

The formula everyone needs and nobody memorises. Raw vector bytes are N × d × bytes_per_component. HNSW adds a graph on top: roughly N × M × 2 × 4 bytes for bidirectional neighbour lists at layer 0, plus upper layers, which in practice lands around 30–45% overhead at M=16. Everything must be resident or latency falls off a cliff.

# N = 1,000,000 chunks · HNSW M = 16 graph overhead ≈ 1e6 × 16 × 2 × 4 B ≈ 128 MB (+ upper layers ≈ 10%)

A · 1536-d fp32 1e6 × 1536 × 4 B = 6.14 GB + 0.14 = 6.28 GB needs a big box B · 1024-d fp32 1e6 × 1024 × 4 B = 4.10 GB + 0.14 = 4.24 GB C · 1024-d fp16 1e6 × 1024 × 2 B = 2.05 GB + 0.14 = 2.19 GB pgvector halfvec D · 512-d int8 1e6 × 512 × 1 B = 0.51 GB + 0.14 = 0.65 GB MRL-truncated + SQ

A → D 9.6× smaller for a few points of recall you buy back by rescoring the top-100 against full-precision vectors.

# the sparse side, same corpus tsvector / inverted ≈ 0.3–0.5× the raw text size — call it 1.4× total footprint extra query latency ≈ +6 ms in published hybrid measurements

The recall / latency curve you must measure yourself

There is no universal ef_search. The shape is always the same — steep gains up to about 2–3× your k, then a long flat tail where you buy milliseconds and no recall. The position of the knee depends on your data's intrinsic dimensionality. Find yours by computing exact top-k with a brute-force scan over a 1,000-query sample and comparing.

In the wild — three cited data points, 2026

pgvector at scale · Jun 2026 A scaling write-up naming the three failure signals precisely: fix filtered-search recall with 0.8+ iterative scans; handle memory limits with quantisation, partitioning and disk-oriented indexes such as pgvectorscale/StreamingDiskANN; and move to a dedicated engine when you need sub-20 ms p99, billions of vectors, or when re-embedding churn causes MVCC bloat.
clickhouse.com — scaling vector search in Postgres

Engine releases · mid-2026 benchmark refresh Every major engine shipped a significant release this year, so old benchmarks are stale: Milvus 2.6 replaced its Kafka/Pulsar dependency with a purpose-built WAL (Woodpecker), removing a heavy operational component; Weaviate 1.37 (April 2026) shipped a native MCP server so agents can query it directly; Qdrant continues to lead single-node latency on Rust with no GC pauses. Vendor numbers assume ideal hardware — re-benchmark on your own.
Vector database benchmarks, refreshed Jul 2026

NVIDIA NeMo Retriever · docs, 2026 The reference enterprise shape: multimodal extraction → embedding NIM → insert into Milvus accelerated by cuVS → semantic + hybrid retrieval with the embedding and reranking NIMs. Worth noting for your capstone: NeMo Retriever does not bundle a vector database — the store is still your decision, and the reranking NIM returns a per-passage index that makes citation attribution mechanical.
docs.nvidia.com/nemo/retriever

Apply to my stack — lab

≈10%

Lab S3 · One Postgres table that serves dense, sparse and metadata — and a swappable embedder

The schema below is the physical form of every decision in this tab: halfvec to halve RAM, a tsvector so hybrid is possible in S4 without a re-ingest, ACL labels indexed for S7, and a embed_model stamp so a model change becomes a partial migration rather than a rewrite.

-- migrations/003_chunks.sql
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;

CREATE TABLE chunks (
  chunk_id      text PRIMARY KEY,
  doc_id        text NOT NULL,
  text          text NOT NULL,
  -- DENSE: halfvec = fp16, half the RAM of vector(1024) for ~0 recall loss
  embedding     halfvec(1024),
  embed_model   text NOT NULL,        -- lets two models coexist during a migration
  -- SPARSE: built at ingest so S4 hybrid needs no reprocessing
  tsv           tsvector GENERATED ALWAYS AS (to_tsvector('english', text)) STORED,
  -- METADATA from S2
  ticker text, cik text, form_type text, section text,
  period_end date, filed_at date, page int,
  content_type text, acl_labels text[],
  parser_version text, chunker_version text, content_hash text
);

-- ANN index. M=16 is the right default; ef_construction buys build-time quality free.
CREATE INDEX chunks_hnsw ON chunks
  USING hnsw (embedding halfvec_cosine_ops) WITH (m = 16, ef_construction = 128);
-- Sparse index for BM25-style ranking (ts_rank_cd) in S4
CREATE INDEX chunks_tsv  ON chunks USING gin (tsv);
-- B-tree indexes on EVERY filter column. Without these the planner has no
-- alternative to the ANN path and the empty-result trap gets much likelier.
CREATE INDEX ON chunks (ticker, period_end DESC);
CREATE INDEX ON chunks (form_type, section);
CREATE INDEX ON chunks USING gin (acl_labels);

# index/embedder.py — one interface, three backends, so the irreversible
# decision stays swappable while you are still deciding.
from typing import Protocol
import httpx, numpy as np

class Embedder(Protocol):
    name: str
    dims: int
    def docs(self, texts: list[str]) -> np.ndarray: ...
    def query(self, texts: list[str]) -> np.ndarray: ...   # separate: asymmetry (S1 2.1)

class LocalBGE:                       # open weights, stays in the perimeter
    name, dims = "bge-m3", 1024
    def __init__(self):
        from sentence_transformers import SentenceTransformer
        self.m = SentenceTransformer("BAAI/bge-m3")
    def docs(self, t):  return self.m.encode(t, normalize_embeddings=True)
    def query(self, t): return self.m.encode(t, normalize_embeddings=True,
                                       prompt_name="query")

class NimEmbedder:                     # NeMo Retriever embedding NIM — the NVIDIA path
    name, dims = "nemo-retriever-nim", 1024   # Matryoshka-truncated from 2048
    def __init__(self, url): self.url = url          # OpenAI-compatible /v1/embeddings
    def _call(self, t, kind):
        r = httpx.post(f"{self.url}/embeddings", timeout=120, json={
            "input": t, "model": "nvidia/llama-3.2-nv-embedqa-1b-v2",
            "input_type": kind,           # "passage" | "query" — the asymmetry flag
            "dimensions": self.dims})     # MRL truncation, server-side
        return np.array([d["embedding"] for d in r.json["data"]], dtype="float32")
    def docs(self, t):  return self._call(t, "passage")
    def query(self, t): return self._call(t, "query")

The runtime dial you now own

SET LOCAL hnsw.ef_search per transaction, not globally. Cheap listing queries can run at 40; the analyst-facing question path can run at 200. Same index, two latency profiles, no rebuild. Set hnsw.iterative_scan = relaxed_order on any query that carries a metadata filter.

Optional exercise · find your knee Sample 500 real queries. For each, compute exact top-10 with a brute-force ORDER BY embedding <=> :q over a sequential scan (SET enable_indexscan = off) — slow, but it is ground truth. Then run the same queries at ef_search ∈ {10, 20, 40, 80, 160, 320}, recording index-recall@10 and p95 latency. Plot it. You will almost certainly find you can drop below the default on cheap paths and that your analyst path needs more than 40. That single plot justifies your index configuration in the capstone.

Bridge → S4

The index exists and it can be searched two ways. Session 4 is where the largest quality gains in this whole stage live: combining those two ways with RRF, filtering without destroying recall, reranking the survivors, and rewriting the query before any of it happens.

Session 4 · (7.1–7.8)

Retrieval that actually works — where the cheapest wins in the whole stage live

If your system is failing, the fix is almost certainly in this tab and almost certainly not in a more exotic architecture. Hybrid search plus a reranker resolves the majority of retrieval failures, ships in a week, and costs a few tens of milliseconds. Everything in S5 is what you reach for after this tab is exhausted.

You are here — Loop B, the query path

Why this session exists

≈10%

top_k=5 looks like a knob. It is actually four assumptions stacked on top of each other, and each one is false often enough to break your system:

Each of the four fixes is a section of this tab. None of them requires an agent, a graph, or a bigger model.

Core concepts

≈50%

2.1 · Metadata filtering — exactness that similarity cannot express

Pass 1 · intuition

"What did Apple say about supply chain in its most recent 10-K?" contains one fuzzy requirement (supply chain) and three hard ones (Apple, 10-K, most recent). Asking an embedding model to honour the hard three is asking a blurry instrument to draw a straight line. Filters are the straight line.

Pass 2 · mechanism

Two orderings, and the difference is the whole game:

pre-filter

Restrict, then search

Build the candidate set from the metadata predicate first, then rank within it. Correct by construction — you always get k results if k exist. Costs: the engine needs a filterable index structure, and a very broad filter degrades to a scan. Qdrant and Weaviate are strong here; Postgres can do it with a partial index or partitioning.

post-filter · the default

Search, then discard

The ANN index returns its ef_search candidates and the predicate deletes most of them. Fast, and quietly wrong: a 10%-selective filter over 40 candidates leaves about four rows for a LIMIT 10. This is the empty-result trap from S3.

-- one filtered hybrid-ready retrieval, with the guards from S3 turned on SET LOCAL hnsw.iterative_scan = relaxed_order; -- keep scanning past the filter SET LOCAL hnsw.ef_search = 120; -- and give it room SELECT chunk_id, text, 1 - (embedding <=> :q) AS sim FROM chunks WHERE ticker = 'AAPL' AND form_type = '10-K' AND period_end >= :as_of - INTERVAL '18 months' AND acl_labels && :user_labels -- ACL is a filter, not an afterthought (S7) ORDER BY embedding <=> :q LIMIT 50; -- 50, not 5 — this feeds the reranker

Design rule

Every hard constraint a user states in words should become a filter, not a hope. Build a small extraction step that pulls tickers, form types, sections and date ranges out of the question and turns them into predicates. That extraction is the cheapest accuracy you will ever buy, and it is also the seed of the query router in S5.

Pass 3 · trade-offs

Filters can hurt recall too. An over-eager date filter on "recent guidance" will drop the 8-K that actually contains it. Prefer soft recency (boost recent, don't exclude old) unless correctness demands a hard cut — point-in-time compliance queries do; exploratory analyst questions don't.

Cardinality matters. High-selectivity filters (one ticker out of 3,000) argue for pre-filtering or partitioning. Low-selectivity filters (form_type in a corpus that is 80% 10-Ks) are nearly free either way.

2.2 · Hybrid retrieval and Reciprocal Rank Fusion

Pass 1 · intuition

You have two witnesses with different blind spots. The dense retriever understands paraphrase and misses exact identifiers. BM25 nails identifiers and misses paraphrase. You do not want to pick one — you want to ask both and trust the documents they agree on. That is the entire idea of fusion, and it is why fusing two similar retrievers buys nothing: agreement is only informative between systems that fail differently.

Pass 2 · mechanism

The obstacle is that a cosine similarity of 0.83 and a BM25 score of 14.2 are not on the same scale and their distributions shift per query, so averaging them is meaningless. Reciprocal Rank Fusion (Cormack, Clarke & Büttcher, SIGIR 2009) sidesteps normalisation entirely by throwing the scores away and keeping only positions:

RRF(d) = Σ over result lists L: 1 / (k + rank_L(d)) k = 60 by default rank 1 → 1/61 = 0.01639 rank 10 → 1/70 = 0.01429 (only 15% less than rank 1) rank 100→ 1/160 = 0.00625 ← about 38% of the rank-1 contribution Consequence: a document ranked #4 by BOTH retrievers beats a document ranked #1 by one and #40 by the other. RRF rewards consensus, not pole position.

Note chunk_A. It topped the dense list and sat at rank 37 in BM25. RRF demotes it — correctly, in this example — because only one witness liked it. That behaviour is the feature.

Tuning k, honestly

k=60 is the near-universal default and the right starting point: it is what Elasticsearch, OpenSearch, Azure AI Search, MongoDB Atlas, Weaviate and Chroma ship. It is sensible, not sacred. Low k (1–10) sharply favours whichever retriever put a document first — use it for short lists where you trust pole position. High k (60–100) rewards agreement — use it for long candidate lists and for fusing a reliable retriever with an experimental one, where you want the outlier damped. Some production systems tune down to k=10 for top-200 fusion; measure on your own query mix.

The alternative worth knowing: score-based fusion. Weaviate's relativeScoreFusion normalises and blends the actual scores rather than the ranks, keeping magnitude information that RRF discards. That matters when one retriever is well calibrated and genuinely knows document A is far better than document B — RRF cannot express that. Trade-off: it needs the score distributions to be stable, which RRF explicitly does not.

Pass 3 · trade-offs

Weighting by query type beats one global weight. Practitioner guidance converges on: BM25 weighted 0.8+ for exact lookups (identifiers, error codes, SKUs, tickers), dense weighted 0.8+ for conceptual questions, roughly even for mixed intent. A cheap classifier on the query — does it contain a token that looks like an identifier? — is enough to switch between two weight profiles.

Fusion only helps between genuinely different retrievers. Fusing three BM25 variants is close to useless; the errors are correlated. Dense + sparse is the canonical pair precisely because they fail on disjoint query classes.

Cost: around +6 ms query latency and about a 1.4× storage footprint in published measurements. Against a reported recall@5 jump from ~0.59 to ~0.82 on financial text-and-table documents, that is not a close decision.

2.3 · MMR — buying diversity with a tunable amount of relevance

Pass 2 · mechanism

Maximal Marginal Relevance builds the result list greedily. At each step it picks the candidate that maximises relevance to the query minus its maximum similarity to anything already selected:

pick_next = argmax over remaining d of: λ · sim(d, query) − (1 − λ) · max over s in selected of sim(d, s) λ = 1.0 → pure relevance (identical to plain top-k) λ = 0.7 → sensible default: mostly relevance, actively penalise near-duplicates λ = 0.3 → aggressive diversity; use for "give me the range of views" questions

Use MMR when the question is “what are all the…”. Skip it when the question has one right answer — diversity is a cost there, not a benefit. In a near-duplicate-heavy corpus like filings, a cheap alternative that costs nothing: deduplicate by content_hash before you rerank.

2.4 · Reranking — how wide should the shortlist be?

Pass 2 · mechanism

S1 established why a cross-encoder exists. The engineering question here is k₁ — how many candidates you hand it. There is exactly one principle:

The sizing rule

k₁ is set by first-stage recall; k₂ is set by the generator's context budget. Choose k₁ as the smallest value where recall@k₁ of your first stage exceeds your target (typically 0.95+). A reranker cannot recover a document the first stage never returned — widening k₂ is useless if recall@k₁ is 0.7. Measure recall@{10, 20, 50, 100} on your golden set (S6) and read k₁ off that curve.

Pointwise cross-encoder

Scores each pair independently (BGE-reranker-v2-m3, the NeMo reranking NIM). Trivially parallel, calibrated-ish scores you can threshold on. The default.

Listwise reranker

Scores candidates in a shared context so relative ordering is decided jointly (jina-reranker-v3 and similar). Often better ordering; harder to batch and to threshold.

LLM-as-reranker

Prompt a general model to order the list. Highest quality ceiling and highest cost/latency. Reasonable for <20 candidates on high-stakes queries; wasteful as a default.

A reranker score is also a refusal signal. Because a cross-encoder is trained to output "relevant / not relevant" rather than "nearby", its score is a far better abstention threshold than a cosine distance. If the best reranked score is below your floor, answer "insufficient evidence". That one line removes a whole category of hallucination and is cheaper than any guardrail model.

2.5 · Query transformation — fixing the input instead of the index

Pass 1 · intuition

Everything so far improves how you search. This improves what you search for, and it is frequently the bigger win — because users do not write queries, they write messages. "We're prepping the client deck tomorrow, they hold a lot of semis, I think NVDA maybe AMD, anyway what's the guidance situation" is not a query. There is a query inside it.

Pass 2 · mechanism

Query rewriting — turn a message into a query

An LLM condenses conversational, verbose or context-dependent input into a standalone search string, resolving pronouns and references to earlier turns. "And what about last quarter?" becomes "NVIDIA Q1 FY2026 revenue guidance". Cheap (one small-model call, ~100ms), and in chat interfaces it is the single largest retrieval improvement available, because without it every follow-up question retrieves nothing useful.

Pass 3 · trade-offs

Every transformation is a latency and failure surface. Multi-query triples your retrieval calls; HyDE adds a generation before the retrieval; decomposition serialises rounds. Apply them conditionally, not always. A one-line classifier — is this query short and specific? — routing to a fast path handles the 60–80% of traffic that is simple lookup and reserves the expensive path for the rest. That is Adaptive RAG in miniature, and S5 formalises it.

HyDE's failure mode is confident fiction. It works by having the model write a hypothetical answer and embedding that, on the theory that a fake answer sits closer to the real answer than the question does. When the model has no idea, the hypothetical document is fluent nonsense and it drags retrieval toward nonsense-shaped chunks. It is strongest in domains where the model knows the form of the answer (financial disclosure language) even when it doesn't know the content.

2.6 · The decision tree: dense, sparse, or hybrid — and what to add next

Guard-clause tree · follow “no ↓” until a “yes” exits right

My retrieval is missing things. What do I add?

Guard 1Are the misses on exact strings — tickers, CUSIPs, Item numbers, executive names, error codes?

yes →Add sparse and fuse with RRF. No amount of dense tuning fixes lexical misses; this is representational.

no ↓

Guard 2Are you retrieving the wrong entity or wrong period — right topic, wrong company or year?

yes →Metadata filters, plus an extraction step that turns stated constraints into predicates. Turn on iterative scan so the filter doesn't gut recall.

no ↓

Guard 3Is the right chunk present in the top-50 but not in the top-5?

yes →Cross-encoder reranker. This is exactly the failure it was built for — you have recall, you lack ordering.

no ↓

Guard 4Are the top-5 five copies of the same paragraph?

yes →Deduplicate by content hash first (free), then MMR at λ≈0.7 if clusters remain.

no ↓

Guard 5Is the user's message conversational, vague, or multi-part — or does the answer need facts from two different sources?

yes →Query transformation: rewriting first (cheapest), then multi-query, then decomposition. Gate it behind a difficulty classifier.

no ↓

Guard 6Is the right chunk genuinely absent from the top-100 — recall itself is broken?

yes →Go back to S2/S3. This is a chunking, parsing or embedding-model problem. Nothing in the query path can retrieve a chunk that doesn't exist or was never indexed correctly.

no ↓

Default (bottom-left) Hybrid dense + BM25 → RRF(k=60) → cross-encoder rerank of the top 50 → top 5 with a score floor. Build this before anything else. It is the 2026 consensus baseline, it fixes most failures, and every advanced architecture in S5 assumes it already exists underneath. If this configuration still fails, you have earned the right to climb the S5 ladder — and you will know which rung, because you will know which guard kept firing.

Term ledger · defined in this tab

top-k: How many chunks reach the prompt. A token budget, not a quality dial.
pre-filter / post-filter: Metadata predicates applied before vs after the ANN scan.
hybrid retrieval: Running dense and sparse retrieval and merging the results.
RRF: Reciprocal Rank Fusion: sum 1/(k+rank) across lists. Rank-based, so no score normalisation needed.
relative score fusion: Score-based alternative that normalises and blends magnitudes, keeping calibration RRF discards.
MMR: Maximal Marginal Relevance: greedy selection trading query relevance against redundancy with a λ knob.
k₁ / k₂: Shortlist size into the reranker, and final chunk count into the prompt.
score floor / abstention: Refusing to answer when the best reranked score is below threshold.
query rewriting: Turning a conversational message into a standalone search query.
multi-query: Generating several paraphrases, retrieving for each, fusing the results.
query decomposition: Splitting a compound question into sub-questions retrieved separately.
HyDE: Hypothetical Document Embeddings: generate a fake answer, embed it, retrieve with that vector.
query routing: Choosing which index, store or tool a query should go to.
auto-merging retriever: Retrieve child chunks; if enough siblings hit, return the parent instead.
sentence-window retriever: Retrieve on single sentences, return the sentence plus its neighbours.

Reality check

≈25%

On paper: top-k as a context-token budget, priced

Your Gemma deployment runs max_model_len = 10,000. The static system prompt is ~350 tokens and prefix-cached. Chunks average 512 tokens plus ~40 tokens of citation header. Here is what k actually buys and costs.

per chunk = 512 + 40 header = 552 tokens static prefix = 350 tokens (cached — near-zero prefill after first hit) answer budget = 600 tokens reserved

k variable-prefill tokens total ctx % of 10,000 window ────────────────────────────────────────────────────────────── 3 1,656 2,606 26% tight, fast, risks missing evidence 5 2,760 3,710 37% ← the usual sweet spot 10 5,520 6,470 65% lost-in-the-middle territory 20 11,040 11,990 120% — exceeds max_model_len

# what k costs in TIME, on your L4 (prefill ≈ 7,000 tok/s for this model class) k=5 prefill 2,760 / 7,000 = 394 ms + rerank 71 ms + retrieve 14 ms = 479 ms TTFT k=10 prefill 5,520 / 7,000 = 789 ms + rerank 71 ms + retrieve 14 ms = 874 ms TTFT

# the prefix-cache effect you already tuned for static 350 tok cached → saves ~50 ms/req and, more importantly, is only a cache hit if EVERY byte before the evidence block is identical. Sorting retrieved chunks into the prompt above the system message = 0% hit rate.

verdict doubling k roughly doubles TTFT and is the single biggest latency lever you own. Buy ordering (reranker, 71 ms) instead of volume.

recall@5 vs recall@20 — the trade in one paragraph

Going from k=5 to k=20 almost always raises recall — the right chunk is more likely to be somewhere in twenty than in five. It also always lowers precision, quadruples prefill, and pushes evidence into the weak middle of the context. The correct move is to raise k₁ (the reranker's input) and keep k₂ (the prompt's input) small. Recall is bought in the shortlist, where it is nearly free; precision is bought in the prompt, where it is expensive. Systems that conflate the two end up slow and inaccurate at the same time.

In the wild — three cited data points, 2026

Hybrid search on financial documents · 2026 A hybrid-search reference consolidating benchmark results: on the WANDS e-commerce dataset a tuned hybrid setup beat either method alone by ~7.4% nDCG; on financial documents, hybrid plus reranking reached recall@5 of about 0.816 against 0.587 for dense-only. Same reference also documents the RRF k-tuning behaviour and the ~6 ms / 1.4× storage cost of adding the sparse side.
denser.ai — hybrid search for RAG, Jun 2026

RRF as an industry default · 2026 RRF is now the default hybrid ranking method in OpenSearch, Elasticsearch, Azure AI Search, MongoDB Atlas and Weaviate, all shipping k=60. Practitioner guidance: k∈[1,10] for short top-10 lists where pole position should dominate, k∈[60,100] for long lists; and the method only pays when the fused rankers fail differently.
RRF — how it works and when to use it, May 2026

The boring middle beats the exotic top · Jul 2026 A survey of eight RAG architecture patterns lands on the same conclusion this tab argues: the discourse is loud about GraphRAG and agents, but the biggest cheapest wins are hybrid retrieval and a reranker. Its escalation advice — add hybrid when you miss names and codes, add a reranker when the answer is in the candidate set but not at the top, only then reach for query transformation and grading loops — is the same ladder as the tree above.
Eight RAG architecture patterns, Jul 2026

Apply to my stack — lab

≈10%

Lab S4 · One hybrid+RRF+rerank retriever, one typed FastAPI route, one SSE stream

Everything fuses inside Postgres in a single round trip, then a reranker trims. The route slots into your existing typed async gateway with the streaming and rate-limiting you already have.

-- retrieval/hybrid_rrf.sql — dense + BM25 + RRF in one statement.
-- No application-side merging: one round trip, planner-visible, filterable.
WITH dense AS (
  SELECT chunk_id, row_number OVER (ORDER BY embedding <=> :qvec) AS r
  FROM chunks
  WHERE (:ticker   IS NULL OR ticker = :ticker)
    AND (:form     IS NULL OR form_type = :form)
    AND (:since    IS NULL OR period_end >= :since)
    AND acl_labels && :user_labels                 -- S7: never optional
  ORDER BY embedding <=> :qvec LIMIT :pool           -- pool = 100
), sparse AS (
  SELECT chunk_id,
         row_number OVER (ORDER BY ts_rank_cd(tsv, plainto_tsquery(:q)) DESC) AS r
  FROM chunks
  WHERE tsv @@ plainto_tsquery(:q)
    AND (:ticker IS NULL OR ticker = :ticker)
    AND acl_labels && :user_labels
  LIMIT :pool
)
SELECT c.chunk_id, c.text, c.ticker, c.form_type, c.section, c.period_end,
       COALESCE(:w_dense  / (:rrf_k + d.r), 0)COALESCE(:w_sparse / (:rrf_k + s.r), 0) AS rrf     -- the formula from 2.2
FROM chunks c
LEFT JOIN dense d  USING (chunk_id)
LEFT JOIN sparse s USING (chunk_id)
WHERE d.chunk_id IS NOT NULL OR s.chunk_id IS NOT NULL
ORDER BY rrf DESC LIMIT :k1;                                -- k1 = 50 → reranker

# api/routes/rag.py — into your existing typed async gateway
from pydantic import BaseModel, Field
from fastapi import APIRouter, Depends
from sse_starlette.sse import EventSourceResponse

router = APIRouter(prefix="/rag")

class AskRequest(BaseModel):
    question: str
    ticker:   str | None = None
    form:     str | None = None
    since:    date | None = None
    k1: int = Field(50, le=200)          # shortlist — recall is bought here, cheaply
    k2: int = Field(5,  le=12)           # prompt — precision is bought here, expensively
    diversify: bool = False              # MMR for "what are all the…" questions

async def hybrid_rerank(req: AskRequest, user) -> list[Hit]:
    # 1. cheap constraint extraction: stated facts become predicates, not hopes
    hints = await extract_constraints(req.question)      # tickers, Item nos, dates
    ticker, form, since = req.ticker or hints.ticker, req.form or hints.form, \
                          req.since or hints.since

    # 2. weight profile by query shape (2.2 pass 3)
    lexical = bool(IDENTIFIER_RE.search(req.question))
    w_dense, w_sparse = (0.4, 1.0) if lexical else (1.0, 0.6)

    async with pool.acquire as con:
        await con.execute("SET LOCAL hnsw.ef_search = 120")
        await con.execute("SET LOCAL hnsw.iterative_scan = relaxed_order")
        rows = await con.fetch(HYBRID_RRF_SQL, qvec=await embed_query(req.question),
                                q=req.question, ticker=ticker, form=form, since=since,
                                user_labels=user.labels, pool=100, rrf_k=60,
                                w_dense=w_dense, w_sparse=w_sparse, k1=req.k1)

    rows = dedupe_by(rows, "content_hash")                # free diversity, do it first
    scored = await reranker.score(req.question, rows)   # cross-encoder, ~70 ms
    kept   = [h for h in scored if h.score >= RERANK_FLOOR]
    if req.diversify:
        kept = mmr(kept, lam=0.7, k=req.k2)
    return kept[:req.k2]

@router.post("/ask")
async def ask(req: AskRequest, user=Depends(current_user)):
    hits = await hybrid_rerank(req, user)
    if not hits:                                        # abstention is a feature (2.4)
        return {"answer": "INSUFFICIENT EVIDENCE", "citations": []}
    return EventSourceResponse(stream_gemma(req.question, hits))  # static prefix first

Why the weight profile switch is worth its four lines

An analyst pasting CUSIP 037833100 and an analyst asking "is the balance sheet getting riskier?" want opposite retrievers. One regex and two weight tuples get you most of what a learned router would, at zero latency. Promote it to a real classifier only when your S6 eval shows the regex misfiring.

Optional exercise · the ablation that settles arguments Take the 20-question set you started in S1. Run five configurations and record recall@5 and MRR for each: (a) dense only, (b) sparse only, (c) RRF hybrid, (d) hybrid + rerank, (e) hybrid + rerank + query rewriting. Then run the same five and record p95 latency. You now have a two-column table — quality and cost — for every rung of the ladder on your corpus. That table is the evidence base for your capstone in S7, and it will very likely show (d) as the knee.

Bridge → S5

Everything so far is a fixed pipeline: one retrieval, one shot, no matter the question. Session 5 breaks that assumption in two directions — letting the system decide whether and how often to retrieve (agentic RAG), and replacing chunk lookup with relationship traversal when the question is about how entities connect (GraphRAG). Both are upgrades you should only buy after the tree above stops firing.

Session 5 · + §9 “Graph RAG” · Source D §7

When one retrieval is not enough — agentic loops and knowledge graphs

Everything up to here is a fixed pipeline: one query in, one retrieval, one answer. This tab breaks that assumption twice. Agentic RAG lets the system decide whether, how, and how many times to retrieve. GraphRAG replaces “find text that resembles the question” with “follow relationships that were actually stated”. Both are genuine capability jumps and both are expensive — the most valuable thing in this tab is the ladder that tells you which rung you actually need.

You are here — the escalation layer wrapped around Loop B

Why this session exists

≈10%

Take the pipeline you designed in S4 — hybrid retrieval, RRF, cross-encoder rerank, top-5 into Gemma — and point it at two questions from a real analyst’s day. It will fail on both, and it will fail confidently, because the failures are structural rather than tuning problems.

Two distinct structural failures. Iteration fixes A; it does nothing for B. Structure fixes B; it is overkill for A. Reaching for “agents” when the problem is B — or a graph when the problem is A — is the most common way to spend a quarter and improve nothing.

The framing that keeps this tab honest

Both techniques in this tab are escalations, not upgrades. An escalation is something you buy after a measurement tells you the cheaper rung has been exhausted — the ladder in §2.5 and the widget in §3 exist to make that a numeric decision instead of a fashionable one. If you have not yet done the S4 ablation, you are not ready to buy anything here.

Core concepts

≈50%

2.1 · Agentic RAG — giving the pipeline a control loop

Pass 1 · intuition

Naive RAG is a vending machine: coin in, one item out, no matter what you wanted. It makes exactly one trip to the shelf, and if the shelf did not have the answer, it hands you the nearest packet anyway (S1: there is always an answer).

Agentic RAG is a research assistant. They read what came back. They notice the report covers 2024 and you asked about 2025. They go back. They realise the question was really two questions and split it. They notice this one is a database lookup, not a document search, and go somewhere else entirely. When nothing supports an answer, they say so.

Everything in agentic RAG follows from a single change: the number of retrieval calls, and their content, becomes a function of what the system has already seen instead of being fixed in advance. That is the whole idea. The frameworks, the graphs of nodes, the tool schemas — all of it is plumbing for that one sentence.

Pass 2 · mechanism

Four components turn a pipeline into a loop. Each is a small model call or a classifier; none is exotic on its own. The behaviour comes from wiring them into a cycle with a bound on it.

The router and the grader are the two components that matter. Both can start as a gpt-oss-class small model or even a regex, and both should return a structured verdict, never prose — you are branching on this value.

Four terms you will meet constantly, defined once:

termRouter

A classifier that picks a destination or a strategy before retrieval happens. Cheapest agentic component and usually the highest-return one, because most traffic is easy and does not deserve the expensive path.

termPlanner

Decomposes a question into an ordered set of sub-questions (S4’s decomposition, promoted to a first-class step with dependencies between the steps).

termGrader / critic

Scores retrieved evidence (or a draft answer) against the question and returns a verdict the control flow branches on. Also called a retrieval evaluator or reflection step.

termTool calling

The model emits a structured call — name plus JSON arguments — that your code executes, returning the result as a new message. Retrieval becomes one tool among several.

Your serving stack already supports this — check the flags

Your Modal vllm serve command already carries --enable-auto-tool-choice, --tool-call-parser gemma4 and --reasoning-parser gemma4. That means Gemma can emit tool calls that arrive as parsed tool_calls in the OpenAI-compatible response rather than as text you have to scrape. It also means the per-request enable_thinking switch becomes a routing lever: thinking off for the cheap path, on for the hard path. Most teams building agentic RAG discover this capability months after they needed it.

The named patterns you will see in papers and framework docs are all the same four components wired differently. Step through them — the differences are small and the cost differences are not.

Pass 3 · trade-offs and limits

Three properties change the moment you add a loop, and all three are things your S1–S4 pipeline did not have to think about.

costMultiplication, not addition

A rerank adds ~70 ms once. A retry edge multiplies everything before it. Two laps through a 900 ms pipeline is 1.8 s plus two grader calls plus a rewrite. Budget agentic features against the blended average (§3 widget), because only the hard slice pays the multiplier.

riskCompounding error

Chain four steps at 90% accuracy each and end-to-end reliability is 0.9⁴ ≈ 66%. Loops make this worse because a bad grader verdict early sends the whole run down a wrong branch. Fewer, better steps beat more steps — every time.

riskUnbounded anything

Always cap iterations, wall-clock, and total tokens, and always have a terminal “abstain with what we found” state. An agent without a budget is an outage with a personality. Your gateway’s rate limiter is protecting the model, not the loop — the loop needs its own budget.

The trap: agentic as a substitute for fixing retrieval

A loop that re-retrieves against a bad index just retrieves badly several times and pays several times for it. If the grader’s verdict is INCORRECT on 40% of first attempts, the finding is not “we need better agents”, it is “our recall is 60% and S2/S3/S4 are unfinished”. Agentic RAG converts a quality problem into a latency and cost problem; it does not remove it.

2.2 · Graph RAG — retrieving relationships instead of passages

Pass 1 · intuition

Vector retrieval answers “what text resembles this question?” Graph retrieval answers “what is connected to this thing, and how?” Those are different questions and most fintech work needs both.

The analogy: a chunk index is the index at the back of a source — brilliant at “where is X mentioned”, useless at “how are X and Y related”. A knowledge graph is the org chart plus the cap table plus the supplier list — it holds relationships explicitly, so following them is a lookup rather than an inference.

Two definitions before anything else, because they are used loosely and mean different things: a knowledge graph (KG) is a set of entities (nodes) connected by typed, directed relationships (edges), where both can carry properties — the atomic unit is the triple: (NVIDIA) −[SUPPLIED_BY]→ (TSMC). A taxonomy is a pure hierarchy (“is-a”); an ontology is the schema of the graph — which entity types and relationship types are legal, and what they mean. Source D’s point is that the ontology is the design decision; the graph is just data that conforms to it.

Pass 2 · mechanism

Source D breaks GraphRAG into three stages, and they map cleanly onto the two loops from Tab 0: G-indexing is Loop A, G-retrieval and G-generation are Loop B.

Note the honest detail in stage 2: GraphRAG does not replace vector search. You almost always find the entry points with an embedding lookup and then traverse. It is an addition to your S3 index, not a replacement for it.

Source D’s single most useful contribution is naming the thing that replaces “chunk size” as your central tuning decision: retrieval granularity. In vector RAG you choose how big a chunk is; in GraphRAG you choose what shape of graph object you return. Step through them from cheapest to richest — the precision/context trade-off from S2 reappears here in a new costume.

Source D also classifies how you decide what to traverse. Three families, in increasing order of cost and capability:

retrieverNon-parametric

Fixed rules: k-hop expansion, shortest path, highest-degree neighbours, a hand-written Cypher template per query type. Microseconds, fully deterministic, auditable. Blind to what the question actually asked — Source D’s stated weakness is insufficient similarity measurement: a text query and a graph structure are not in the same space, so heuristics substitute for relevance.

retrieverGNN-based

A graph neural network scores nodes and edges for relevance given the query encoding. Learns structural relevance rather than assuming it. Needs training data, a training pipeline, and a serving path — realistically a research project, not a sprint.

retrieverLLM-based

The model writes the traversal — most commonly by generating Cypher (the source material 9.3), sometimes by acting as an agent with expand_node / find_path tools. Most flexible, and the natural meeting point of §2.1 and §2.2. Costs a generation per query and needs the schema in the prompt plus a read-only, timeout-bounded execution path.

Text-to-Cypher is the practical entry point — and its failure mode

Handing the model your schema and asking for a Cypher query is the highest-value, lowest-effort GraphRAG pattern, and it is what the source material 9.3 builds. The failure mode is not syntax — models write valid Cypher — it is silent semantic wrongness: a query that runs, returns rows, and answers a subtly different question. Two mitigations, both cheap: execute against a read-only role with a statement timeout, and show the generated query and its row count in the answer so a human can catch it.

// entry points by vector search, then two hops of typed traversal — the common shape
CALL db.index.vector.queryNodes('entity_embedding', 8, $qvec) YIELD node AS seed
MATCH path = (seed)-[r:SUPPLIED_BY|COMPETES_WITH|AUDITED_BY*1..2]-(n:Company)
WHERE n.ticker IN $portfolio AND ALL(x IN r WHERE x.as_of >= date($since))
RETURN DISTINCT n.ticker, [x IN relationships(path) | type(x)] AS how,
       [x IN relationships(path) | x.chunk_id] AS evidence   // citations survive the traversal
LIMIT 25;

Pass 3 · trade-offs and limits

GraphRAG’s costs are almost entirely in Loop A, which is why it looks cheap in a demo and expensive in production. Four of them, concretely.

costIndexing is an LLM pass over the corpus

Entity and relation extraction means at least one generation per chunk — the same order of magnitude as contextual retrieval in S2, but with a larger output. For your 12,000 filings this is a real four-figure decision, quantified in §3.

riskExtraction errors are permanent

A missed edge is a question that can never be answered; a hallucinated edge is a confident wrong answer with a citation attached. Unlike a bad chunk — which a reranker can demote — a wrong edge is invisible at query time. Sample and audit extraction accuracy explicitly.

riskEntity resolution is the hidden project

“NVIDIA”, “NVIDIA Corporation”, “NVDA”, and CIK 0001045810 must be one node or your graph silently fragments into disconnected islands that traverse to nothing. In fintech you are lucky: tickers and CIKs give you a real join key. Use it as the node identity, not the surface string.

costSchema drift and staleness

Relationships have as-of dates. A supplier relationship from a 2023 filing may be false today. Every edge needs a validity window and a source chunk, and your S7 refresh job has to update edges, not just re-embed text.

Where the materials have aged · GraphRAG cost, 2024 → 2026

Source D and most GraphRAG material describe the original Microsoft Research design: extract entities and relations from every chunk, detect communities with the Leiden algorithm, then pre-generate a natural language summary of every community so that “global” questions can be answered by map-reducing over summaries. It works, and its indexing bill is brutal — the summarisation pass, not the extraction pass, is what makes it unaffordable at corpus scale.

The 2025–26 correction is LazyGraphRAG: build the cheap part of the graph up front and defer summarisation to query time, generating only the summaries a specific question needs. Microsoft reported indexing cost falling to roughly a thousandth of full GraphRAG while matching answer quality on local queries and remaining competitive on global ones. The practical read for 2026: do not price GraphRAG using the 2024 numbers, and do not treat community summarisation as a required stage. Start with entity extraction plus text-to-Cypher, and add summarisation only for genuinely global questions.

Two further things the materials predate: graph capability has moved into the databases you already run — Neo4j is no longer the only option, with pgvector-alongside-recursive-CTE, Memgraph, and the graph extensions in Postgres making “a graph” a table design rather than a new system to operate. And the published comparisons have become much less flattering than the 2024 wave suggested: several 2025–26 evaluations found well-tuned hybrid vector RAG matching or beating GraphRAG on question sets that were not specifically global or multi-hop, at a fraction of the indexing cost. microsoft.github.io/graphrag LazyGraphRAG announcement

The one-line test for whether you need a graph

Could the answer, in principle, appear verbatim inside one passage? If yes, a graph is the wrong tool — fix retrieval instead. If the answer only exists as a composition of facts stated in different documents, that composition is exactly what a graph stores, and no amount of chunking will produce it. “What did NVIDIA say about supply concentration?” is a chunk question. “Which of our holdings share a supplier?” is a graph question.

2.3 · The escalation ladder — the decision tree for this whole tab

Which rung do I actually need? Follow “no ↓”. First “yes” exits right. Bottom-left is the default.

Start: your S4 pipeline (hybrid + RRF + rerank) is deployed and you have eval numbers for it.

Is retrieval recall@20 below ~0.8 on your eval set?

STOP — you are not ready for this tab. Go back to S2 (chunking, contextual retrieval) and S3 (embedding model, index parameters). Every technique below multiplies the cost of a retriever that is still missing one document in five. Escalating now buys you slower wrong answers.

no ↓

Do failures concentrate in conversational follow-ups, vocabulary mismatch, or “which source should this even hit?”

Rung 1 — query transformation + a router. Rewriting, multi-query and a destination classifier (S4 §2.6). ~1.4× latency on the affected slice, no new infrastructure, no loop. This is still the cheapest real win available and most teams skip straight past it.

no ↓

Is your traffic clearly bimodal — a large easy majority plus a small expensive minority?

Rung 2 — Adaptive RAG. Classify complexity up front and send each class down a different path: no-retrieval / single-shot / full loop. This is the only rung that can make your system cheaper as well as better; the widget below shows why the saving depends entirely on the hard-query share.

no ↓

Are answers wrong because the retrieved evidence was bad, in a way a reader could detect from the evidence alone?

Rung 3 — a grader with one bounded retry (CRAG-style). Add the evaluator, allow exactly one rewrite-and-retry, and make “abstain” the terminal state. One retry captures most of the available gain; the second and third mostly burn budget.

no ↓

Do questions require a second query whose content depends on the first result (true multi-hop), or actions beyond retrieval (SQL, a calculation, an API)?

Rung 4 — a real agentic loop with tools. Now you need a state machine, tool schemas, per-run budgets and step-level tracing. Use your existing tool-calling flags. Expect p99 to triple and your eval to get substantially harder — S6 has to cover paths, not just outputs.

no ↓

Are the unanswerable questions about relationships between entities or themes across the whole corpus — answers that appear in no single passage?

Rung 5 — GraphRAG, scoped. Build a graph over the entities you can resolve reliably (tickers, CIKs, filings, people), start with text-to-Cypher over that graph as one tool beside your existing retriever, and keep chunk retrieval for everything else. Do not start with community summarisation.

no ↓

Default: stay on the S4 pipeline and spend the quarter on evaluation instead. If none of the guards above fired, your failures are not architectural. They are in chunking, in the prompt, in your metadata, or — most often — in the fact that nobody has measured which of those it is. That is S6, and it is a better investment than anything in this tab.

Read the ladder as a cost curve, not a maturity model

There is no prize for being on rung 5. A team running rung 1 with a measured 0.86 recall and a 500 ms p95 is beating a team running rung 4 with an unmeasured pipeline and a 4-second p99, and it is beating them on every axis that a user or a CFO can perceive. Escalate on evidence, and be willing to de-escalate when S6 shows a rung is not earning its multiplier.

Reality check

≈25%

3.1 · Worked example — what a graph over your corpus actually costs

Same corpus as S2 and S3: 12,000 filings, ~216M tokens, 421,875 chunks at 512 tokens. Price the two GraphRAG designs against each other and against the S4 baseline. Extraction is priced at a small-model rate of roughly $0.20 per million tokens in, $0.60 out — the sort of rate you would pay for a hosted small model, or approximate on your own L4 by counting GPU-seconds.

CORPUS

chunks421,875

tokens in corpus216,000,000

DESIGN A · ENTITY + RELATION EXTRACTION ONLY (recommended start)

input tokens (chunk + schema prompt ≈ 700)295,312,500

output tokens (~180 of JSON triples per chunk)75,937,500

extraction cost$59.06 + $45.56 = $104.62

entity-name embeddings (~90k unique entities)$0.90

wall-clock on 1× L4 at ~14 chunks/s≈ 8.4 hours

one-off total≈ $105

DESIGN B · + COMMUNITY DETECTION AND SUMMARISATION (classic GraphRAG)

communities at 3 levels (est.)≈ 11,400

summarisation input (members + edges, ~4,000 tok each)45,600,000

summarisation output (~600 tok each)6,840,000

summarisation cost at small-model rates$9.12 + $4.10 = $13.22

…but summaries need a capable model to be worth reading×15 ≈ $198

one-off total (A + B)≈ $303, and it must be redone as the graph changes

THE TERM THAT ACTUALLY DECIDES IT · REFRESH

new filings per month (est. 4% of corpus)480 docs · 16,875 chunks

Design A monthly re-extraction$4.19/mo

Design B monthly (communities shift → resummarise ~30%)$4.19 + $59 = $63/mo

QUERY-SIDE, PER 1,000 QUESTIONS

S4 baseline (hybrid + rerank + generate)~900 ms · 1.0× tokens

text-to-Cypher + traverse + generate~1,750 ms · 2.0× tokens

global search over community summaries (map-reduce)~9,000 ms · 12–40× tokens

Three things fall out of those numbers, and none of them is the one people expect. First: the extraction pass is affordable — about a hundred dollars, once, for a corpus that took months to accumulate. GraphRAG’s reputation for expense comes almost entirely from the summarisation stage, which is why LazyGraphRAG’s deferral changes the economics so completely. Second: the recurring number matters far more than the one-off; $4/month is a rounding error and $63/month plus the operational burden of keeping community summaries coherent is a commitment. Third: global search is 12–40× the tokens of a normal query. That is not a latency problem you tune away — it is a different product, and it belongs behind a “generate thematic report” button with a progress bar, not behind the chat box.

3.2 · The escalation ladder, priced against your own traffic

The single number that determines whether any of this is worth buying is the hard-query share: what fraction of your traffic actually needs the expensive path. Every multiplier below applies only to that slice. Move the sliders to your real numbers.

Push the hard-query share to 5% and the expensive rungs become almost free on average — that is the entire argument for Adaptive RAG, and it is why routing is rung 2 rather than rung 4. Push it to 80% and the classifier is pure overhead: you are paying for a decision whose answer is nearly always the same. The crossover sits somewhere around a third, and you can measure yours in an afternoon by labelling 200 logged queries as easy or hard.

3.3 · Grounding · three current sources

LazyGraphRAG and the cost correction · Microsoft Research Microsoft’s own follow-up to GraphRAG reports that deferring community summarisation to query time cuts indexing cost to roughly a thousandth of the original design while matching it on local queries and staying competitive on global ones. The practically important detail for anyone budgeting a graph in 2026 is which stage the money was in: extraction is cheap, eager summarisation is not.
microsoft.com/research — LazyGraphRAG GraphRAG docs and query modes

Agentic RAG survey and pattern taxonomy · 2025–26 The surveys that consolidated this space converge on the same decomposition used above — routing, planning, reflection/critique, and tool use — and on the same warning: reported quality gains are accompanied by latency and token multipliers in the 2–7× range, with reliability governed by the product of per-step accuracies rather than by the best step. The consistent recommendation across them is bounded loops with an explicit terminal abstain state.
arXiv 2501.09136 — Agentic RAG: a survey

Graph retrieval mechanics · GraphRAG survey (cited by Source D) The survey Source D builds its section on formalises the G-indexing / G-retrieval / G-generation split, the granularity ladder from nodes through subgraphs, and the two structural obstacles this tab flags: explosive candidate subgraph growth with hop count, and the mismatch between a text query and a graph structure that makes “similarity” ill-defined on the retrieval side.
arXiv 2408.08921 — Graph Retrieval-Augmented Generation: a survey

Apply to my stack — lab

≈10%

Lab S5 · a bounded corrective loop and a scoped fintech graph

Two deliverables, deliberately in ladder order: rung 3 first (a grader with one retry, which you can ship this week), then a scoped graph exposed as one additional tool. Both call the same Modal Gemma endpoint you already run; neither adds a framework dependency you cannot remove.

File 1 — rag/loop.py: the corrective loop as an explicit state machine. No LangGraph required to start; the shape below is the graph, and porting it to LangGraph later is mechanical if you keep the state in one dataclass.

# rag/loop.py — rung 3: route → retrieve → grade → (one) retry → generate | abstain
from dataclasses import dataclass, field
from enum import Enum
import time, re

MAX_LAPS      = 2          # the single most important constant in this file
WALL_CLOCK_MS = 6000       # hard ceiling; the SSE client gives up at 8s

class Verdict(str, Enum):
    CORRECT = "correct"; AMBIGUOUS = "ambiguous"; INCORRECT = "incorrect"

class Route(str, Enum):
    NONE = "none"; FILINGS = "filings"; NEWS = "news"; POSITIONS = "positions"; GRAPH = "graph"

@dataclass
class RunState:                      # one object = one traced run (Langfuse span, S6)
    question: str
    route: Route | None = None
    query: str = ""
    hits: list = field(default_factory=list)
    laps: int = 0
    t0: float = field(default_factory=time.monotonic)
    trail: list[str] = field(default_factory=list)   # why it did what it did

    def budget_left(self) -> bool:
        return self.laps < MAX_LAPS and (time.monotonic-self.t0)*1000 < WALL_CLOCK_MS

# --- 1. ROUTER: regex first, model only when the regex is unsure -------------
IDENT   = re.compile(r"\b(CIK\s*\d{4,10}|CUSIP\s*[\w\d]{9}|10-[KQ]|8-K|[A-Z]{1,5}:[A-Z]{2,4})\b")
GRAPHY  = re.compile(r"\b(same|shared|both|between|connected|overlap|which of (our|the))\b", re.I)
POSNS   = re.compile(r"\b(our position|we hold|portfolio weight|exposure to)\b", re.I)

async def route(s: RunState) -> Route:
    q = s.question
    if POSNS.search(q):  s.trail.append("route=positions (regex)"); return Route.POSITIONS
    if GRAPHY.search(q): s.trail.append("route=graph (regex)");     return Route.GRAPH
    if IDENT.search(q):  s.trail.append("route=filings (identifier)"); return Route.FILINGS
    label = await classify(q, ["filings", "news", "none"])   # ~90ms, thinking OFF
    s.trail.append(f"route={label} (model)");  return Route(label)

# --- 2. GRADER: structured verdict, never prose ------------------------------
GRADE_SYS = ("You judge whether retrieved evidence can answer a question. "
             "Reply with ONE word: correct | ambiguous | incorrect. "
             "correct = the evidence contains the facts needed. "
             "ambiguous = partially there, a better query might find the rest. "
             "incorrect = the evidence is off-topic or empty.")

async def grade(s: RunState) -> Verdict:
    if not s.hits: return Verdict.INCORRECT
    ctx = "\n---\n".join(h.text[:600] for h in s.hits[:5])
    out = await gemma(GRADE_SYS, f"Q: {s.question}\n\nEVIDENCE:\n{ctx}",
                     max_tokens=4, temperature=0, enable_thinking=False)
    try:    return Verdict(out.strip.lower)
    except: return Verdict.AMBIGUOUS          # unparseable == not confident

# --- 3. THE LOOP -------------------------------------------------------------
async def answer(question: str, user):
    s = RunState(question=question, query=question)
    s.route = await route(s)
    if s.route is Route.NONE:
        return await generate_no_context(s)

    while True:
        s.laps += 1
        s.hits = await retrieve(s.route, s.query, user)      # the whole S4 pipeline
        v = await grade(s)
        s.trail.append(f"lap{s.laps} route={s.route} n={len(s.hits)} verdict={v}")

        if v is Verdict.CORRECT or not s.budget_left:
            break
        if v is Verdict.AMBIGUOUS:
            s.query = await rewrite(s.question, s.hits)         # keep rare tokens verbatim
        else:                                                 # INCORRECT → change strategy,
            s.route = Route.NEWS if s.route is Route.FILINGS else Route.FILINGS
            s.query = s.question                                # not just the wording

    if not s.hits or (await grade(s)) is Verdict.INCORRECT:
        return {"answer": "INSUFFICIENT EVIDENCE", "citations": [], "trail": s.trail}
    return await generate_grounded(s)                        # static prefix first (S1)

Four details in that file that are easy to get wrong

1 · The regex router runs before the model router. Roughly two-thirds of fintech queries carry an identifier or an unmistakable phrase; those cost zero milliseconds to route. Only the remainder pays for a classification call. 2 enable_thinking=False on the grader and router. These are classification calls; thinking tokens make them slower and no more accurate. Save the switch for the final generation on hard queries. 3 · An unparseable verdict degrades to AMBIGUOUS, not CORRECT. Failing towards “try again” is safe; failing towards “ship it” is how a broken grader silently disables itself. 4 trail is not logging, it is the eval artefact. S6 needs to know which path a run took; without it you can only measure outcomes, and you will not be able to tell a good answer from a lucky one.

File 2 — graph/schema.cypher: a deliberately small ontology. The discipline here is to include only entity types you can resolve to a canonical identifier, because an unresolvable entity type produces fragments, and fragments produce empty traversals that look like missing data.

// graph/schema.cypher — constraints ARE the entity-resolution strategy
CREATE CONSTRAINT company_cik  IF NOT EXISTS FOR (c:Company)  REQUIRE c.cik IS UNIQUE;
CREATE CONSTRAINT filing_acc   IF NOT EXISTS FOR (f:Filing)   REQUIRE f.accession IS UNIQUE;
CREATE CONSTRAINT person_key   IF NOT EXISTS FOR (p:Person)   REQUIRE p.key IS UNIQUE;
CREATE CONSTRAINT risk_slug    IF NOT EXISTS FOR (r:RiskTheme)REQUIRE r.slug IS UNIQUE;

// nodes:  Company(cik, ticker, name, sector)   Filing(accession, form, period_end, filed_at)
//         Person(key, name)                     RiskTheme(slug, label)
// edges carry provenance and validity — both are mandatory, not optional:
//   (:Company)-[:FILED {filed_at}]->(:Filing)
//   (:Company)-[:SUPPLIED_BY {as_of, chunk_id, confidence}]->(:Company)
//   (:Company)-[:COMPETES_WITH {as_of, chunk_id}]->(:Company)
//   (:Person)-[:OFFICER_OF {role, as_of, chunk_id}]->(:Company)
//   (:Filing)-[:DISCLOSES {salience, chunk_id}]->(:RiskTheme)

CREATE VECTOR INDEX entity_embedding IF NOT EXISTS
FOR (c:Company) ON c.embedding
OPTIONS {indexConfig: {`vector.dimensions`: 1024, `vector.similarity_function`: 'cosine'}};

The rule that saves the project: every edge carries a chunk_id

The moment an edge exists without a pointer back to the passage it came from, your graph answers stop being citable — and an uncitable answer is exactly the failure mode S1 built the whole stage to prevent. Provenance on edges also gives you a free correction path: when the S6 eval flags a wrong answer, the chunk_id tells you whether the extraction was wrong or the source was.

File 3 — graph/tool.py: the graph exposed as one tool your loop can call, with the guardrails that make LLM-generated Cypher safe to run.

# graph/tool.py — text-to-Cypher behind a read-only role and a timeout
SCHEMA_CARD = open("graph/schema_card.txt").read   # ~40 lines, cached in the prompt prefix

TOOL_SPEC = {"type": "function", "function": {
    "name": "query_entity_graph",
    "description": ("Answer questions about relationships BETWEEN companies, people, "
                    "filings and risk themes: shared suppliers, common officers, "
                    "competitors, who disclosed the same theme. NOT for the content "
                    "of a single document — use search_filings for that."),
    "parameters": {"type": "object",
                   "properties": {"question": {"type": "string"}},
                   "required": ["question"]}}}

FORBIDDEN = re.compile(r"\b(CREATE|MERGE|DELETE|SET|DROP|CALL\s+apoc|LOAD\s+CSV)\b", re.I)

async def query_entity_graph(question: str, user) -> dict:
    cypher = await gemma(f"{SCHEMA_CARD}\n\nWrite ONE read-only Cypher query. "
                        "Always RETURN chunk_id values as `evidence`. LIMIT 25.",
                        question, temperature=0, enable_thinking=True)
    cypher = strip_fences(cypher)
    if FORBIDDEN.search(cypher):                      # belt; the role is the braces
        return {"error": "write operation refused"}
    async with driver.session(database="fin", default_access_mode=READ_ACCESS) as ses:
        res = await ses.run(cypher, timeout=3.0)          # unbounded traversal = outage
        rows = [r.data async for r in res]
    rows = [r for r in rows if acl_ok(r.get("evidence"), user)]   # ACL at the edge
    return {"cypher": cypher, "rows": rows, "n": len(rows)}   # show the query in the UI

Where this maps onto your career stack

Open source: LangGraph for the state machine once loop.py outgrows a while-loop; Neo4j Community or Memgraph for the graph; the official graphrag package if you ever do need community summaries. NVIDIA: the router and grader are perfect NIM candidates — small, latency- critical, high-QPS classification — and NeMo Retriever’s reranking NIM slots in under the same tool interface, so the loop code does not change when you swap the implementation. Managed: Bedrock Agents and Vertex AI Agent Engine both provide the orchestration and tracing layer if you would rather not run the state machine yourself; Neptune Analytics (AWS) and Spanner Graph (GCP) are the managed graph stores. The portable design decision, in all three cases, is that retrieval and graph access are tools behind a schema — that boundary is what lets you change your mind later.

Optional exercise · earn the escalation, or don’t Take the 20-question set from S1 and add ten questions you know your S4 pipeline cannot answer — five true multi-hop, five relational (“which of X share Y”). Run all thirty through the plain S4 pipeline and through loop.py, recording for each: correct/incorrect, p50 and p95 latency, total tokens, and the trail. Then answer two questions with the data. (a) On the original twenty, did the loop change any answer — and were the changes improvements, or did the grader send good evidence back for a pointless lap? (b) What is the blended cost multiplier across all thirty, and does it match what the §3.2 widget predicted at your measured hard-query share? If the loop helped only on the ten you added, you have just proven the case for routing rather than for looping — which is rung 2, and much cheaper.

Term ledger · defined in this tab

agentic RAG: Any RAG design where the number and content of retrieval calls is decided at run time from what the system has already seen, rather than fixed in advance.
router: A classifier choosing a destination or strategy before retrieval. Cheapest agentic component, highest typical return.
grader / retrieval evaluator: A step that scores retrieved evidence and returns a structured verdict the control flow branches on.
Adaptive RAG: Route by question complexity: no-retrieval, single-shot, or iterative. The rung that can lower average cost.
CRAG (Corrective RAG): Grade retrieved documents; on a poor grade, refine the query or fall back to another source before generating.
Self-RAG: Train or prompt the model to emit reflection tokens deciding when to retrieve and whether its own output is supported.
ReAct: Interleaved reason/act loop where the model alternates between thinking and calling tools.
tool calling: Structured function invocation emitted by the model and executed by your code; retrieval becomes one tool among several.
knowledge graph (KG): Entities as nodes, typed directed relationships as edges, both carrying properties. Atomic unit: the triple.
triple: (subject, predicate, object) — the smallest complete statement a graph can store.
taxonomy vs ontology: A taxonomy is a pure “is-a” hierarchy; an ontology is the full schema of legal entity and relationship types and their meaning.
G-indexing / G-retrieval / G-generation: Source D’s three GraphRAG stages: build the graph offline, traverse it at query time, serialise the result for the model.
retrieval granularity: The graph analogue of chunk size: whether you return nodes, triplets, paths, subgraphs, or a graph/chunk hybrid.
entity resolution: Collapsing surface forms (“NVIDIA”, “NVDA”, CIK 1045810) onto one canonical node. The hidden project inside every graph build.
community detection / summarisation: Clustering the graph (e.g. Leiden) and pre-writing a summary per cluster to answer global questions. The expensive stage LazyGraphRAG defers.
text-to-Cypher: Having the model generate a graph query from the schema plus the question. The practical entry point to GraphRAG.
hard-query share: The fraction of traffic needing the expensive path. Determines whether any escalation in this tab is affordable.

Bridge → S6

Every decision in this tab was conditional on a measurement — recall@20 below 0.8, a hard-query share, a grader’s accuracy — and you do not yet have any of them. Session 6 builds the instrument: component-level metrics that tell you which stage failed, a golden set worth trusting, and online evaluation wired into the Langfuse tracing you already run.

Session 6 · (10.1–10.6)

Evaluation — the instrument that makes every earlier decision falsifiable

Five sessions of trade-offs, and every one of them ended in “measure it on your corpus”. This is that measurement. The point is not a score; a single end-to-end number tells you that something is wrong and nothing about what. The point is attribution — a set of metrics arranged so that a bad answer points at the component that produced it.

You are here — the feedback arc that closes both loops

Why this session exists

≈10%

Here is the situation this tab exists to prevent. Your system scores 72% correct on a hundred questions. You have a week to improve it. Where do you spend the week?

With one number, you cannot know. So you do what everyone does: you guess, you change three things at once, the number goes to 74%, and you learn nothing — because you cannot tell whether the new embedding model helped and the prompt change hurt, or the reverse, or whether 72 and 74 are the same number wearing different clothes.

Both panels describe the identical system. The distribution on the right is the deliverable of this session; the number on the left is what most teams have. Note the shape — 19 of 28 failures are retrieval, which is the ratio you should expect and the reason S2–S4 got the weight they did.

The law that governs everything below

RAG is an open-book exam, and that gives you a hard ceiling: no prompt change can recover a fact that retrieval never delivered. End-to-end correctness is bounded above by retrieval recall. If recall is 0.75, then 25% of your questions are unanswerable no matter how good the generator is — and every hour spent on prompt engineering in that state is an hour spent decorating a locked door.

Core concepts

≈50%

2.1 · The two layers — and why they need different instruments

Pass 1 · intuition

Keep the exam analogy and it does all the work. A student gets a question wrong for exactly two reasons: the page they needed was not in the material they were handed, or the page was there and they wrote something else anyway. These have nothing in common. They have different causes, different fixes, different owners, and — critically — different metrics.

So you measure twice. Retrieval metrics ask: given this question, did the right material end up in the context, and how far up? They compare a ranked list against a set of known-relevant documents and need no LLM at all — they are arithmetic, fast, deterministic, and free. Generation metrics ask: given this context, is the answer supported by it and responsive to the question? They need judgement, which in practice means another model, which is where the cost and the caveats live.

One consequence worth stating plainly: you should build the retrieval half first. It is cheaper, it runs in seconds, it catches the majority of failures, and it is the half that is actually reliable.

Pass 2 · mechanism

Retrieval metrics all operate on the same object — a ranked list with some items marked relevant — and differ only in what they reward. Seeing them scored against one identical list is the fastest way to internalise the differences.

Definitions, once: recall@k = relevant found ÷ relevant that exist. precision@k = relevant found ÷ k. MRR = mean of 1/(rank of first hit). nDCG@k = position-discounted gain divided by the best achievable ordering, so 1.0 means perfect ranking. hit rate = fraction of questions with at least one relevant result.

the source material 10.4 adds a metric that behaves differently from all of these, and it is the one that connects retrieval to generation: context precision@k — of the chunks you actually placed in the prompt, what fraction were useful for producing the answer. Unlike classic precision it does not need a pre-labelled relevance set; a judge decides per chunk against the reference answer. It is the metric that catches the S4 failure where you retrieved the right thing plus nine distractors and then lost the model in the middle of them.

On the generation side, three metrics form the standard triad. Step through what each one actually computes — the mechanics matter, because two of them are frequently reported as if they meant the opposite of what they measure.

The two most common misreadings on any RAG dashboard

“Faithfulness is 0.94, we’re in good shape.” Faithfulness only asks whether the answer follows from the context. An answer that faithfully summarises the wrong three chunks scores a perfect 1.0. High faithfulness with low context recall is the signature of a silent failure, and it is the most dangerous state a RAG system can be in, because the dashboard is green.

“Answer relevancy is high, so the answer is right.” Answer relevancy measures whether the response addresses the question — not whether it is true. A fluent, on-topic, entirely fabricated answer scores well. Relevancy is a coherence check, not a correctness check; correctness needs a reference answer or a citation audit.

Pass 3 · trade-offs and limits

The triad is not three independent dials. They compose into a ceiling, and the pattern of which ones are low is itself the diagnosis. That relationship is what the simulator in §3 makes tangible — but three limits apply to every metric above.

limitAverages hide the failures that matter

A mean of 0.85 over 200 questions can be 170 perfect answers and 30 catastrophic ones. In a fintech product the 30 are the entire risk. Always slice: by question type, by form type, by document age, by whether the answer required a table. Report the worst slice next to the mean or the mean will lie to you every single week.

limitCost and latency are part of the score

A configuration that gains 2 points of nDCG for 3× the tokens is a regression in most products. Every eval row should carry latency and token counts beside quality, so that the ladder decisions from S5 are made on the same table as the quality decisions. Quality-only leaderboards produce systems nobody can afford to run.

limitYour golden set decays

A test set built in March against a corpus that grows 4% a month is describing a system that no longer exists by autumn. Version the golden set alongside the index, re-verify a sample quarterly, and treat a question whose ground-truth chunk has been superseded as a bug in the set — not as a model regression.

2.2 · LLM-as-judge — powerful, cheap, and biased in known ways

Pass 2 · mechanism

Every generation metric above is computed by prompting a model to make a judgement. That is the only affordable way to score free text at scale, and it works — judge/human agreement in the 80s is routinely reported for well-designed rubrics, which is roughly the agreement two humans achieve with each other. But a judge is a model, and it inherits model failure modes. Five are well documented, and each has a cheap mitigation.

The last panel is the one that decides whether the other four matter. An uncalibrated judge is not a measurement — it is a second opinion from a model with no track record.

The decomposition that makes judging reliable

The single technique that most improves judge quality is claim decomposition: instead of asking “is this answer faithful?”, split the answer into atomic statements and ask, per statement, whether the context supports it. Faithfulness becomes supported ÷ total — a ratio, not an opinion. This is exactly how Ragas computes it, and it kills verbosity bias as a side effect, because each claim is judged on its own and a longer answer simply has more claims to defend.

2.3 · The golden set — the part everybody skips

Pass 2 · mechanism

A golden set (test set, eval set) is a list of questions with, at minimum, the chunk IDs that should be retrieved, and ideally a reference answer. Without it, none of the retrieval metrics can be computed at all — recall needs a denominator. This is the deliverable that unblocks everything else, and it is the one nobody wants to build.

source 1Real questions from real users

The highest-value source and the one you already have if you log queries. Sample across the whole distribution, not just the interesting failures, or your set will over-represent hard cases and every measurement will look worse than reality. Anonymise, then label the relevant chunks.

source 2Synthetic, generated from your documents

the source material 10.3. Take a chunk, have a model write a question it answers, keep the chunk as ground truth. Scales to hundreds in an hour and is how you bootstrap before launch. Requires care — see the trap below.

source 3Adversarial and unanswerable

The set that separates a demo from a product. Include questions whose answers are not in the corpus, questions about the wrong period, and near-miss questions where a plausible-but-wrong chunk exists. The correct output is a refusal, and nothing else in your eval tests the refusal path.

The trap that makes synthetic sets useless

A model given a chunk writes a question using that chunk’s vocabulary. Your retriever then matches that vocabulary trivially and you measure a recall of 0.95 — for a system that scores 0.6 on real questions, because real users do not phrase things the way filings do. Three mitigations, all cheap: generate from the summary of a chunk rather than its text; explicitly instruct the generator to paraphrase and to use the vocabulary of an analyst rather than a filing; and generate a slice of multi-hop questions from pairs of chunks so that some items in the set cannot be solved by lexical overlap. Then sanity-check: if BM25 alone scores above about 0.9 recall on your synthetic set, the set is measuring vocabulary echo, not retrieval.

How many questions? Enough that a change you care about is distinguishable from noise. For a pass/fail metric near 0.8, the 95% interval on 50 questions is roughly ±11 points — so a 50-question set cannot detect anything smaller than a 10-point swing. At 200 questions it narrows to about ±5.5, and at 500 to ±3.5. The practical shape: 200 questions as the working set for CI and day-to-day decisions, a 30–50 question hand-labelled subset as the calibration anchor for your judge, and a held-out set you look at rarely, so that months of tuning against the working set does not quietly turn it into a training set.

2.4 · Offline versus online — and what each is for

offlineThe golden set, in CI

Fixed questions, known ground truth, run on every change. Answers “did this commit make retrieval worse?” in about ninety seconds. Deterministic, comparable across weeks, and the only place a regression gate belongs. Blind to anything your set does not contain — which is most of what users will actually ask.

onlineSampled production traffic

Score a percentage of live traffic with the judge on the metrics that need no ground truth — faithfulness, context precision, answer relevancy, refusal rate. Answers “is it working now, on what people actually ask?” Catches distribution shift, new document types, and the slow drift that offline sets never see. Costs a judge call per sampled request, so sample at 1–5% and always score the refusals.

You already have the substrate for this

Your stack has Langfuse tracing and an existing eval harness. That means online eval is not a new system — it is a scoring consumer attached to traces you already emit. The one structural change worth making now is ensuring every trace carries the retrieval span (query, retrieved chunk IDs, scores) and the S5 trail, not just the prompt and completion. Without the chunk IDs on the trace, no online retrieval metric can ever be computed, and you will be back to a single aggregate number.

2.5 · The diagnostic tree — “an answer was wrong; what do I fix?”

Attributing a single bad answer Follow “no ↓”. First “yes” exits right. Bottom-left is the default.

Start: one failed question, with its trace — the retrieved chunk IDs, their scores, the assembled context, and the answer.

Does the corpus contain the answer at all? (grep for it; check the document exists and is in scope)

Not a RAG bug. The correct behaviour was a refusal. If the system answered anyway, the bug is a missing abstention path (S1) or a score floor set too low (S4) — fix that, and move this question into your adversarial slice where it belongs.

no ↓

Is the ground-truth chunk absent from the first-stage results (k₁ = 50)?

Recall failure — the expensive one. Now sub-diagnose in this order: (a) is the evidence split across two chunks? → S2 chunking. (b) does BM25 alone find it? → your dense side is failing; hybrid weighting or embedding model, S3/S4. (c) does neither find it? → embedding model or a query/document vocabulary gap; try contextual retrieval (S2) or HyDE (S4). (d) does an exact-match query find it but the filtered query does not? → your metadata filter is wrong, S3.

no ↓

Was it in the 50 but not in the final k₂ = 5?

Ranking failure — the cheap one. The document was there and the ordering lost it. Raise k₂, check the reranker is actually running on the right text field, and verify your score floor is not truncating above the correct chunk. This is the single most fixable category on the tree.

no ↓

Was it in the final context but buried among near-duplicates or low-value chunks?

Context precision failure. Lost-in-the-middle plus context rot (S1). Deduplicate by content_hash, apply MMR, or simply cut k₂ — fewer, better chunks beat more chunks at essentially every k above about 5.

no ↓

Was the evidence clean and present, and the answer still contradicts it or invents a figure?

Generation failure — now, and only now, touch the prompt. Strengthen the grounding instruction, enforce citations per claim, make the refusal path explicit and rewarded, and check the assembled order (S1: static block, then context, then query). Also check enable_thinking — numeric reasoning over a table is a place where turning it on measurably helps.

no ↓

Default: the question is ambiguous, or your ground-truth label is wrong. This is far more common than teams expect — a meaningful share of “failures” in a new eval set turn out to be bad labels, questions with two defensible answers, or ground truth that a corpus update superseded. Fix the set, note it, and move on. A golden set nobody audits becomes fiction within two quarters.

Reality check

≈25%

3.1 · Worked example — what evaluation costs, and what the numbers mean together

Price a real eval programme on your corpus. Assume a 200-question working set, a judge at roughly $0.30/M in and $1.20/M out (a mid-tier hosted model — deliberately not Gemma, to avoid self-preference), and your production traffic from S5.

BUILDING THE GOLDEN SET (one-off)

synthetic generation · 200 questions from 200 sampled chunks~0.3M tok $0.24

human verification at ~2.5 min/question≈ 8.3 hours

hand-labelled judge calibration subset (50 items)≈ 3 hours

honest total~1.5 engineer-days. This is the whole cost.

ONE OFFLINE RUN · 200 QUESTIONS

retrieval metrics (recall/precision/MRR/nDCG)pure arithmetic $0.00 · ~40 s

generation: 200 answers on your own L4~$0.11 of GPU time

judge: faithfulness (claim-decomposed, ~9 claims/answer)1.9M in / 0.14M out · $0.74

judge: answer relevancy + context precision@5$0.61

full run cost$1.46 · ~7 min wall-clock

retrieval-only gate (what runs on every commit)$0.00 · 40 s

ONLINE SAMPLING · 20,000 QUERIES/DAY AT 3%

sampled per day600

judge cost$4.38/day $131/month

as a share of the ~$1,300/mo serving bill from S7≈ 10%

The shape of that table is the argument. Retrieval evaluation is free and instant — there is no defensible reason not to gate every commit on it, and a team that has not done so is choosing to fly blind for zero saving. Generation evaluation costs about a coffee per run, which means nightly, not per-commit. And the entire cost of the programme is the day and a half of human labelling at the top — everything downstream is rounding error. When someone says evaluation is too expensive, they mean they do not want to spend the day and a half, and it is worth naming that plainly, because the alternative is spending quarters instead.

3.2 · The triad, as a system — read the diagnosis, not the scores

Move the three sliders to the numbers your system reports. The point is not the ceiling arithmetic; it is that the pattern of which metric is low is a diagnosis, and the same aggregate score can mean four completely different repair jobs.

The default position of those sliders is not arbitrary — recall in the sixties with faithfulness in the nineties is the most common real-world configuration, and it is the silent failure. The system is behaving impeccably: it is faithfully summarising evidence that is missing a third of what it needed. Every instinct says the model is doing well. Every hour spent on the prompt in that state is wasted. Drag recall up and watch the ceiling move; drag faithfulness up and watch it not.

3.3 · Grounding · three current sources

Ragas · reference-free metric definitions The framework the source material builds its section on. Its documented definitions are the de-facto vocabulary of the field: faithfulness computed by decomposing an answer into atomic claims and checking each against the retrieved context, answer relevancy computed by reverse-generating questions from the answer and measuring similarity to the original, and the separation of context precision from context recall. Its most useful property remains that most of these need no reference answer, so they can run on live traffic.
docs.ragas.io — metric concepts

LLM-as-judge reliability · the calibration literature The body of work established that strong judges reach human-level agreement on well-scoped rubrics — roughly the same rate at which two human annotators agree with each other — while also documenting the position, verbosity and self-preference biases that make an uncalibrated judge unreliable. The consistent recommendations are the ones in §2.2: decompose into atomic claims, randomise presentation order, avoid judging a model with its own family, and anchor to a human-labelled subset.
arXiv 2306.05685 — Judging LLM-as-a-Judge

Tooling landscape · 2026 The evaluation stack has consolidated into three roles that are worth keeping separate: a metric library (Ragas, DeepEval, TruLens) that defines and computes scores; a tracing and online-scoring platform (Langfuse, Phoenix, LangSmith) that attaches those scores to production spans; and a CI gate that is usually just pytest with thresholds. Keeping the metric definitions independent of the platform is what lets you change either without losing your history — the mistake to avoid is expressing your metrics only inside a vendor’s UI.
langfuse.com/docs — scores and online evaluation DeepEval docs

Apply to my stack — lab

≈10%

Lab S6 · extend the harness you already have — do not start a new one

You have an eval harness and Langfuse tracing. The work here is three additions, in this order: (1) a golden-set format that carries chunk-level ground truth, (2) retrieval metrics as a free CI gate, (3) judge-based metrics on a nightly job and a 3% online sample. Nothing below replaces existing code; each piece attaches to it.

File 1 — eval/golden.jsonl: the format. Chunk IDs are the whole point; without them the only metrics you can compute are the expensive ones.

// eval/golden.jsonl — one object per line, versioned in git next to the migration
{"id": "q0042",
 "question": "How did NVIDIA characterise supply concentration risk in its FY2026 10-K?",
 "gold_chunk_ids": ["0001045810-26-000023:1A:07", "0001045810-26-000023:1A:08"],
 "reference": "Identifies dependence on a limited number of foundries, principally TSMC…",
 "slice": {"type": "single_hop", "form": "10-K", "needs_table": false, "period": "FY2026"},
 "index_version": "2026-07-14"}

{"id": "q0043",
 "question": "What was the effective tax rate in the quarter after the restatement?",
 "gold_chunk_ids": [],                          // deliberately unanswerable
 "reference": "INSUFFICIENT EVIDENCE",
 "slice": {"type": "adversarial", "form": "10-Q", "needs_table": true},
 "index_version": "2026-07-14"}

File 2 — eval/retrieval_metrics.py: forty lines, no dependencies, no model calls. This is the piece that runs on every commit.

# eval/retrieval_metrics.py — free, deterministic, 40 seconds for 200 questions
import math
from statistics import mean

def recall_at_k(retrieved, gold, k):
    if not gold: return None                          # unanswerable: scored elsewhere
    return len(set(retrieved[:k]) & set(gold)) / len(gold)

def precision_at_k(retrieved, gold, k):
    return len(set(retrieved[:k]) & set(gold)) / max(1, min(k, len(retrieved)))

def mrr(retrieved, gold):
    for i, c in enumerate(retrieved, 1):
        if c in gold: return 1.0 / i
    return 0.0

def ndcg_at_k(retrieved, gold, k):
    dcg  = sum((1.0 if c in gold else 0.0) / math.log2(i1)
               for i, c in enumerate(retrieved[:k], 1))
    idcg = sum(1.0 / math.log2(i1) for i in range(1, min(len(gold), k)1))
    return dcg / idcg if idcg else 0.0

async def run_retrieval_eval(golden, retrieve_fn, k1=50, k2=5):
    rows = []
    for item in golden:
        stage1, final = await retrieve_fn(item["question"])   # BOTH stages, always
        s1 = [h.chunk_id for h in stage1]
        s2 = [h.chunk_id for h in final]
        g  = item["gold_chunk_ids"]
        rows.append({"id": item["id"], **item["slice"],
                     "recall@k1":  recall_at_k(s1, g, k1),      # THE ceiling
                     "recall@k2":  recall_at_k(s2, g, k2),
                     "ndcg@k2":    ndcg_at_k(s2, g, k2),      # reranker's report card
                     "prec@k2":    precision_at_k(s2, g, k2),
                     "mrr":        mrr(s2, g),
                     "abstain_ok": (not g) and (not s2)})    # adversarial slice
    return rows

def by_slice(rows, metric, dim):
    """The function that stops averages from lying. Always print this, never just the mean."""
    out = {}
    for r in rows:
        if r.get(metric) is not None: out.setdefault(r[dim], []).append(r[metric])
    return {k: (round(mean(v), 3), len(v)) for k, v in sorted(out.items)}

File 3 — eval/test_gate.py: the CI gate. It fails on the ceiling metric and on the worst slice, not on the mean.

# eval/test_gate.py — pytest. Runs on every commit. Costs nothing.
import pytest, json
from eval.retrieval_metrics import run_retrieval_eval, by_slice

BASELINE = json.load(open("eval/baseline.json"))      # committed; updated deliberately
TOLERANCE = 0.02                                        # noise band, not a licence to drift

@pytest.mark.asyncio
async def test_no_retrieval_regression(golden, retriever):
    rows = await run_retrieval_eval(golden, retriever)
    r1 = mean_of(rows, "recall@k1")
    assert r1 >= BASELINE["recall@k1"] - TOLERANCE, \
        f"first-stage recall {r1:.3f} below baseline — nothing downstream can fix this"

    for dim in ("type", "form", "needs_table"):        # the slice gate
        for name, (val, n) in by_slice(rows, "recall@k1", dim).items:
            if n >= 15:                                   # ignore tiny slices
                assert val >= BASELINE["slices"].get(f"{dim}:{name}", 0) - 0.05, \
                    f"slice {dim}={name} regressed to {val:.3f} (n={n})"

@pytest.mark.asyncio
async def test_abstains_on_unanswerable(golden, pipeline):
    adv = [g for g in golden if g["slice"]["type"] == "adversarial"]
    got = [(await pipeline(g["question"]))["answer"] for g in adv]
    rate = sum("INSUFFICIENT EVIDENCE" in a for a in got) / len(got)
    assert rate >= 0.9, f"abstention rate {rate:.2f} — the refusal path is decaying"

File 4 — eval/online.py: the 3% sample, attached to traces you already emit.

# eval/online.py — score live traffic; requires chunk_ids on the retrieval span
import random
from langfuse import Langfuse

lf = Langfuse
SAMPLE = 0.03

async def maybe_score(trace_id, question, contexts, answer, latency_ms, trail):
    # always score refusals and slow runs; sample the rest
    forced = ("INSUFFICIENT EVIDENCE" in answer) or latency_ms > 3000
    if not forced and random.random > SAMPLE:
        return

    claims  = await decompose(answer)                    # atomic statements
    support = await judge_support(claims, contexts)      # NOT Gemma — different family
    faith   = sum(support) / max(1, len(claims))
    cprec   = await judge_context_precision(question, contexts)

    lf.score(trace_id=trace_id, name="faithfulness",      value=faith)
    lf.score(trace_id=trace_id, name="context_precision", value=cprec)
    lf.score(trace_id=trace_id, name="n_laps",            value=len([t for t in trail if t.startswith("lap")]))
    if faith < 0.7:
        lf.score(trace_id=trace_id, name="triage", value=0,
                 comment="low faithfulness — queue for human review")

Three decisions in those files worth defending in a review

1 · The gate fails on recall@k1, not on end-to-end correctness. First-stage recall is the ceiling; a regression there is unrecoverable downstream, and it is measurable for free in forty seconds. End-to-end scores are noisier, slower and more expensive, which makes them a poor gate and a good nightly report. 2 · The slice gate has a minimum n. Without it a five-question slice will fail your build on noise every third day and the team will disable the gate — which is worse than not having one. 3 · Online scoring always fires on refusals and slow runs. Those are the two populations where random sampling is least informative and the failures are most concentrated; forcing them costs almost nothing because they are rare.

Where this maps onto your career stack

Open source: Ragas or DeepEval for metric definitions, Langfuse for traces and scores, pytest for the gate — the split in §3.3 keeps them independent. NVIDIA: NeMo Evaluator covers this territory for teams standardising on the NeMo stack, and a small judge served as a NIM is a sensible way to make online scoring cheap at high sample rates. Managed: Bedrock’s evaluation jobs and Vertex AI’s Gen AI Evaluation Service both offer managed LLM-as-judge with the same metric families; both are worth it if you want the judge managed and are content to keep the metric definitions in their vocabulary.

Optional exercise · calibrate the judge before you trust a single number it produces Take 50 question/answer pairs from a real run and label each yourself as faithful or not — binary, no scale, about three hours. Then run your judge on the same 50 and compute plain agreement and Cohen’s κ. Three outcomes, all useful: κ above 0.7 — the judge is usable; record the number, put it in the README, and re-run this check whenever you change judge models. κ between 0.4 and 0.7 — the rubric is under-specified; the fix is almost always claim decomposition plus explicit criteria per verdict, not a bigger judge. κ below 0.4 — your judge is producing noise, and every generation metric on your dashboard so far has been decoration. Now repeat the whole exercise with the judge’s two options presented in reversed order and see how many verdicts flip; that number is your position bias, measured rather than assumed.

Term ledger · defined in this tab

recall@k: Fraction of all relevant chunks that appear in the top k. The ceiling metric — nothing downstream can exceed it.
precision@k: Fraction of the top k that are relevant. Measures noise in the context window.
MRR: Mean reciprocal rank: the average of 1/(rank of the first relevant result). Only sees the first hit.
nDCG@k: Position-discounted gain normalised by the ideal ordering; 1.0 means perfectly ranked. The right metric after reranking.
hit rate@k: Fraction of questions with at least one relevant result in the top k. Coarse; flatters weak systems.
context recall: Whether the retrieved context contains everything the reference answer needed. Retrieval’s report card, judged against a reference.
context precision@k: What fraction of the chunks actually placed in the prompt were useful. Catches retrieved-plus-noise.
faithfulness / groundedness: Fraction of the answer’s atomic claims supported by the retrieved context. Says nothing about whether the context was right.
answer relevancy: Whether the response addresses the question asked. A coherence check, not a correctness check.
claim decomposition: Splitting an answer into atomic statements and judging each separately. The main reliability technique for LLM judges.
LLM-as-judge: Using a model to score free-text output. Reliable only when calibrated against human labels.
position / verbosity / self-preference bias: The three documented judge biases: favouring the first option, the longer answer, and its own model family.
golden set: Questions plus ground-truth chunk IDs and reference answers. The prerequisite for every retrieval metric.
synthetic eval data: Questions generated from your own documents. Fast to build; prone to vocabulary echo unless deliberately paraphrased.
adversarial slice: Questions whose correct answer is a refusal. The only test of the abstention path.
slicing: Reporting metrics per question type, form type or document age rather than only as a mean. The defence against hidden catastrophic subsets.
offline vs online eval: Fixed golden set in CI versus judge-scored samples of live traffic. Different questions; you need both.
Cohen’s κ: Agreement between two raters corrected for chance. The number that tells you whether your judge is a measurement.

Bridge → S7

You can now tell which component is failing and prove whether a change helped. What remains is everything that only appears once real documents, real users and real deadlines arrive: parsing at scale, keeping the index fresh, enforcing access control on retrieval, and paying for it. S7 closes the stage with the capstone — the whole fintech system, every choice justified by the trees you have collected.

Session 7 · → “The RAG Platform” · stage capstone

Production RAG — the wall, and the system you build to get over it

the source material opens by naming the thing this session is about: teams get a prototype working in an afternoon and then spend nine months at the production wall. Nothing on the other side of that wall is intellectually difficult. It is documents that do not parse, an index that drifts out of date, permissions that must survive retrieval, a bill nobody modelled, and failure modes with no obvious owner. This tab is that list, followed by the capstone: the whole fintech system, assembled, with every choice traced back to a decision tree you have already used.

You are here — everything outside the happy path

Why this session exists

≈10%

The story in the source material’s foreword is the best one-paragraph argument for this entire session. A legal RAG system was asked whether a contract permitted a particular action. It said yes. The contract said no. The clause and its exception had been separated by a fixed-character chunker; retrieval found the exception, the model read it faithfully, and the system produced the exact opposite of the truth with complete confidence.

Notice what that failure is not. Not a hallucination — the answer was grounded in retrieved text. Not a bad model. Not a bad embedding. It was a data engineering defect, introduced 400 characters at a time by a utility function nobody reviewed, and it was invisible to every metric that was not slice-level retrieval evaluation. The production wall is made almost entirely of defects with that shape.

Every row on the right is a system-design problem with a known answer. The reason teams take nine months is not that the answers are hard — it is that the questions do not appear until a prototype meets real data, and by then the architecture has already assumed they do not exist.

The through-line for this tab

Production RAG is a data pipeline with a language model attached, not a language-model application with a data source attached. Once you accept that framing, most of what follows is standard data engineering — idempotency, versioning, incremental updates, backfills, dead-letter queues, access control, cost attribution — applied to a slightly unusual payload. That is good news: these are solved problems with decades of practice behind them.

Core concepts

≈50%

2.1 · Keeping the index true — the refresh problem

Pass 1 · intuition

Your index is a cache of a corpus that keeps changing, and every cache has the same two failure modes: it can be stale, and it can be inconsistent with its source. A stale RAG index does not error. It confidently answers last quarter’s question with last quarter’s number, correctly cited, and nothing in your monitoring notices — which makes staleness the most under-monitored failure in production RAG.

The intuition that organises all of §2.1: four kinds of change arrive, and each demands a different response. Most teams build for the first and discover the other three in an incident.

Pass 2 · mechanism

Class 4 is why S3’s reversibility ladder mattered: the embedding model is the least reversible choice in the stack, and this is the bill that makes it so. Design the swap on day one — version in the table name or a column, build the new index alongside, run both, compare on the S6 golden set, then cut over. A rebuild you planned is a weekend; one you did not is a quarter.

Three mechanisms handle all four classes, and none of them is exotic:

mechanismcontent_hash as the idempotency key

Hash the extracted text (not the file — PDFs differ byte-wise on identical content). Unchanged hash means skip the entire pipeline for that document. On a monthly refresh where 96% of the corpus is unchanged, this one column is the difference between a $4 job and a $105 job. It is also the join key that lets you detect the class-2 case reliably.

mechanismTransactional replace, not delete-then-insert

Wrap the class-2 update in a transaction, or write the new chunks with a new doc_version and flip a pointer. In Postgres this is free and you should simply use it. In a separate vector store it usually is not, which is one of the strongest practical arguments for keeping vectors in the same database as your metadata.

mechanismBlue/green indexes for class 4

Build chunks_v2 beside chunks_v1, backfill in the background, evaluate both on the golden set, then move an alias. Costs double storage for a few days and removes the entire category of “we cannot upgrade the embedding model because the migration is too scary”.

The staleness monitor nobody builds, in one query

Emit a metric for max(now − indexed_at) per source, and alert when it exceeds your promised freshness. Also emit count(*) where indexed_at > filed_at + interval '2 hours'. Both are one line of SQL and together they catch the silent-staleness failure that no quality metric will ever surface — because a stale answer is a perfectly faithful answer to a question about the past.

2.2 · Access control — the constraint that must live inside retrieval

Pass 2 · mechanism

In fintech this is not a feature, it is the licence to operate: research under embargo, material non-public information, client-confidential notes, and data with jurisdictional restrictions all sit in the same corpus. The rule has exactly one correct form.

This is the same pre-filter vs post-filter distinction from S3 and S4, but the stakes have changed: there, post-filtering cost you recall; here, it costs you a disclosure. The engineering is identical — which is why getting it right in S3 was worth the effort.

patternLabel-based ACLs on the chunk

Every chunk carries an array of permission labels inherited from its document. The user’s session resolves to a label set, and retrieval intersects them in the WHERE clause — a GIN-indexed array overlap in Postgres. Simple, fast, auditable, and it composes with your other metadata filters at no extra cost.

patternLate-binding permissions

Resolve the user’s labels at query time from your identity system rather than copying group membership into the index. Otherwise a revocation only takes effect after the next re-index — which is precisely the window in which a departing employee’s access matters most.

trapDerived data leaks the source

The trap everyone finds late. Summaries, contextual-retrieval prefixes (S2), extracted graph edges (S5), and cached answers are all derived from restricted documents and inherit their restrictions. A graph edge that says two companies share an auditor may encode a fact from a document the user cannot read. Propagate ACLs to every derived artefact, or do not derive across permission boundaries at all.

Deletion in a vector index, and the honest version of “right to erasure”

Soft-delete with a filtered index is the right default: WHERE deleted_at IS NULL costs nothing and makes revocation instantaneous. But for a genuine erasure obligation, soft delete is not enough — the text still exists, and so do its embeddings, which are lossy but not meaningless. A defensible erasure therefore has three parts: hard-delete the rows and their vectors, purge derived artefacts (graph edges, summaries, caches, traces), and log the erasure. Design the third part first, because the audit will ask for evidence, not assurances.

2.3 · Guardrails and graceful degradation

Pass 2 · mechanism

treats guardrails as a first-class part of scaling, and RAG has one threat that a plain LLM application does not: the retrieved documents are untrusted input that lands inside your prompt. If any document in your corpus can be authored or influenced by someone outside your trust boundary — a news feed, an uploaded PDF, a scraped filing exhibit — then indirect prompt injection is a live path, not a theoretical one.

inputBefore retrieval

PII detection and redaction on the query, prompt-injection screening, and a scope check — “is this a question this product should answer at all?”. Cheap classifiers; run them in parallel with routing, not in series, so they cost nothing on the critical path.

contextBetween retrieval and generation

The RAG-specific one. Treat retrieved text as data: delimit it clearly, state in the static prefix that instructions inside the context are to be ignored, and screen chunks for injection patterns at ingest time rather than query time — it is the same check, done once per document instead of once per query.

outputAfter generation

Citation verification (does every cited chunk ID exist in the retrieved set?), a numeric sanity check against the context for financial figures, and a PII/compliance scan. Citation verification is the highest-value one and is pure string matching — no model required.

degradeWhen a component is down

Define the fallback for each: reranker down → serve top-k from fusion with a warning; dense index down → BM25-only, flagged as degraded; generator down → return the retrieved passages themselves, which are genuinely useful on their own. A RAG system has the rare property that partial results have real value. Use it.

The degradation ladder is a product decision, not an ops decision

“Return ranked source passages with a banner saying the summariser is unavailable” is a better outcome than a 503, and in a fintech tool it may be a perfectly acceptable one — analysts read primary sources anyway. Agree that ladder with whoever owns the product before the incident, because at 3am the on-call engineer will otherwise choose the error page.

2.4 · Cost, latency, and the caches that decide both

Pass 2 · mechanism

You already know from the earlier serving stage that retrieved context inflates prefill and that prefix caching only helps for the static block. RAG adds four more caches, and knowing which one you are missing is usually worth more than any model change.

Scale-to-zero and RAG: the interaction your Modal config creates

Your deployment runs min_containers=0, which is excellent economics and a specific hazard here. A cold start pays container startup plus weight load and arrives with an empty prefix cache, so the first request after idle can be an order of magnitude slower than steady state — and RAG traffic is often bursty (a filing drops, twelve analysts ask at once). Three options, in increasing cost: accept it and set the client timeout honestly; keep one warm container during market hours only; or serve a retrieval-only degraded response while the container warms. The wrong option is to discover the behaviour in production and conclude that RAG is slow.

2.5 · Buy versus build — the decision tree for the platform itself

frames this as the platform question, and it is the highest-leverage decision in the whole stage because it is made once and constrains everything after. Run each component through the tree separately — the right answer is almost always a mix, and “build everything” and “buy everything” are both usually wrong.

For this component: buy, or build? Follow “no ↓”. First “yes” exits right. Bottom-left is the default.

Start: one component — parsing, embedding, vector store, reranking, orchestration, evaluation, or the whole platform.

Does the data have a residency, sovereignty, or contractual restriction that forbids it leaving your boundary?

Self-host, decision closed. For MNPI, embargoed research, or client-confidential material this is not a trade-off to optimise. Note the practical consequence: it forces self-hosted embedding too, since embedding an entire corpus through a third-party API sends the corpus through a third party.

no ↓

Is this component a genuine differentiator — the thing your product is actually better at?

Build it. For a fintech RAG product this is usually the domain-specific layer: filing-structure parsing, the financial entity graph, your evaluation set. Almost never the vector store. Be strict here — most teams believe three components are differentiators and are right about one.

no ↓

Would running it yourself add a system your team does not already operate?

Buy, or fold it into what you already run. This is the argument that made pgvector the S3 default: you already operate Postgres, so vectors cost you zero new operational surface. A dedicated vector database is a second stateful system with its own backups, upgrades, scaling behaviour and on-call — worth it above roughly 50–100M vectors, hard to justify below.

no ↓

Is the managed option’s cost, at your actual volume, less than roughly a third of an engineer’s time?

Buy. Do the arithmetic honestly and include the maintenance tail, not just the build. A managed parser at a few hundred dollars a month against three engineer-weeks of PDF debugging is not a close call — and PDF parsing is the canonical example of a problem that is infinitely deep and worth zero as a capability.

no ↓

Is the component still changing fast enough that today’s best choice will be wrong within a year?

Buy, but behind your own interface. Rerankers and embedding models are the live examples. Wrap them in the Protocol you defined in S3 so that swapping the implementation is a config change. The interface is the real deliverable; the vendor behind it is temporary.

no ↓

Default: self-host it with open-source components, behind a narrow interface. For your situation this is also the career-aligned answer — operating the open-source stack is what builds the NVIDIA-adjacent depth you are aiming at, and the interface discipline means you can still swap in a managed service for any component the day the arithmetic changes. The thing to avoid is not self-hosting; it is self-hosting without the interface, which is how a stack becomes unswappable.

Reality check

≈25%

3.1 · Worked example — the full monthly bill for the fintech system

Everything from S2–S6, priced together on your actual stack: L4 on Modal at roughly $0.45/hour, 12,000 documents growing 4% monthly, self-hosted embedding and reranking, Postgres you already run.

ONE-OFF · BUILD THE INDEX

parsing 12,000 filings (Docling cascade, ~2.4 s/doc, batched)8 GPU-h $3.60

contextual retrieval prefixes (S2)$220

embedding 421,875 chunks × 1024-dim (self-hosted BGE on L4)6.5 GPU-h $2.93

entity extraction for the graph (S5, Design A)$105

golden set: 200 questions, verified (S6)1.5 eng-days + $0.24

total to first production index≈ $332 + 1.5 eng-days

RECURRING · STORAGE AND REFRESH

halfvec(1024) HNSW index, 421,875 rows (S3)1.6 GB RAM · in existing Postgres

text + metadata + tsvector~4.1 GB disk

monthly delta: 480 docs → 16,875 chunksparse+embed $0.35

monthly contextual prefixes on the delta$8.80

monthly graph re-extraction on the delta$4.19

ingestion subtotal$13.34/mo — a rounding error, as it should be

RECURRING · SERVING (the actual bill)

interactive at 20,000 queries/day, k=5see model below

online eval judge at 3% sample (S6)$131/mo

Langfuse + Prometheus/Grafana (self-hosted, existing)marginal

The lesson in that table is the ratio. Building the index costs about as much as three days of serving it. Every argument about ingestion cost — contextual retrieval, the graph, better parsing — is an argument about a few hundred dollars, once, on a corpus representing years of work. Serving is where the money actually goes, and within serving, k is the dominant term, because k multiplies prefill on every single request. Move the sliders and watch which one has leverage.

Two things that model makes concrete. First, the self-hosting argument is real but not infinite — at low volume a frontier API is cheaper than a GPU you keep warm, and the crossover arrives surprisingly early; scale-to-zero is what makes the self-hosted line competitive at the bottom end, which is exactly why your Modal configuration matters. Second, k₂ is the cheapest quality-versus-cost lever you own. Going from k=10 to k=5 halves prefill on every request, and S4 already showed you it usually improves quality by removing distractors. That is the rare change that is better and cheaper, and it should be the first thing you test.

3.2 · Grounding · three current sources

Where production RAG projects actually stall · 2026 practitioner reporting The consistent finding across production retrospectives is that the failures that kill projects are infrastructural rather than model-related: document parsing quality, index freshness, permission enforcement, and cost predictability. Parsing in particular is repeatedly described as the largest single sink of engineering time, which is the empirical case for the “buy the parser” exit in the §2.5 tree.
databricks.com — long-context and RAG in production

Indirect prompt injection through retrieved content · OWASP LLM Top 10 The threat model treats retrieved documents as untrusted input and injection through them as a distinct, high-severity category from direct prompt injection, precisely because the attacker never talks to your system — they only need to influence a document you will later index. The recommended controls match §2.3: screen at ingest, delimit context explicitly, and validate outputs rather than relying on instructions to hold.
owasp.org — Top 10 for LLM applications

Managed RAG platforms · the 2026 buy-side options The managed landscape has settled into full-stack platforms (Vertex AI Search, Amazon Bedrock Knowledge Bases, Azure AI Search) and component services you can adopt piecemeal (managed parsing, embedding, reranking). NVIDIA’s NeMo Retriever occupies the middle ground — self-hosted NIM containers for embedding, reranking and extraction with managed-quality models — which is the combination that fits a data-residency constraint plus a small team.
docs.nvidia.com/nemo/retriever Bedrock Knowledge Bases

Apply to my stack — the capstone

≈10%

This is the whole stage in one artefact: the fintech RAG system, every component chosen by a decision tree you have already run, every number from a calculator you have already moved. Nothing below is a default — each box carries the reason it is there.

Every box traces to a guard clause. The two most consequential decisions on the diagram are the least visible: the ACL predicate inside the retrieval query, and the chunk_ids on every trace — the first makes the system deployable, the second makes it improvable.

Capstone · the build order, and the operational surface that makes it real

The order matters more than any individual choice. Build it in this sequence and each step is testable before the next depends on it; build it in any other order and you will be debugging four unknowns at once.

#	Ship	Justified by	Done when
1	Golden set — 200 questions, chunk-level ground truth, adversarial slice	S6 §2.3	BM25-only recall on the synthetic slice is below 0.9 (else the set is measuring vocabulary echo)
2	`migrations/003_chunks.sql` — halfvec, tsvector, HNSW, GIN on acl_labels	S3 tree	a filtered ANN query with `iterative_scan` returns a full k under a 2%-selective ACL
3	Ingestion: Docling cascade → Item-aware recursive 512 → contextual prefix → embed	S2 tree	parse-quality tripwires fire on a deliberately broken PDF; re-running the job changes nothing
4	`retrieval/hybrid_rrf.sql` + rerank to 5 with a score floor	S4 tree	recall@50 ≥ 0.85 and nDCG@5 beats dense-only by a margin larger than the noise band
5	Retrieval gate in CI + Langfuse spans carrying chunk_ids	S6 §4	a deliberately worsened chunker fails the build in under 90 seconds
6	Generation: static prefix → context → query, citations enforced, abstention path	S1 §2.4	abstention rate on the adversarial slice ≥ 0.9; prefix cache hit rate is what you predicted
7	Router (regex → model) and the ≤1-retry grader	S5 rungs 2–3	blended latency matches the §3.2 widget at your measured hard-query share
8	Entity graph + Cypher tool, on relational routes only	S5 rung 5	the ten relational questions you could not answer in S5 now resolve, with citations
9	Operational surface: staleness metric, DLQ alarm, degradation ladder, 3% online eval	S7 §2.1–2.4	you can answer “is the index current, and what broke last night?” without opening a notebook

The operational file that ties it together — ops/health.sql: four queries that answer the questions an on-call engineer and a compliance officer will actually ask. This is the smallest honest definition of “in production”.

-- ops/health.sql — export as Prometheus gauges; alert on the first two

-- 1. FRESHNESS: the failure no quality metric will ever catch
SELECT source,
       EXTRACT(EPOCH FROM now - max(indexed_at)) AS staleness_seconds,
       count(*) FILTER (WHERE indexed_at > filed_at + interval '2 hours') AS late_docs
FROM chunks GROUP BY source;

-- 2. PARSE HEALTH: a document that parsed to nothing is silent data loss
SELECT parser_version,
       count(*) FILTER (WHERE length(text) < 200)::float / count(*) AS suspicious_ratio,
       count(DISTINCT doc_id) FILTER (WHERE table_count = 0 AND form_type IN ('10-K','10-Q')) AS no_tables
FROM chunks GROUP BY parser_version;   -- a 10-K with zero tables did not parse

-- 3. INDEX DRIFT: are all rows on the same model? (class-4 change, half-applied)
SELECT embed_version, chunker_version, count(*) FROM chunks
GROUP BY 1,2 ORDER BY 3 DESC;   -- more than one row here during a swap only

-- 4. ACL COVERAGE: the query the audit will ask for
SELECT count(*) FROM chunks WHERE acl_labels IS NULL OR cardinality(acl_labels) = 0;
-- must be 0. an unlabelled chunk is either invisible to everyone or visible to everyone,
-- and which one it is depends on an operator precedence detail in your WHERE clause.

Where the capstone maps onto your career stack

Open source, and what you operate: Postgres + pgvector, Docling, BGE-M3 and BGE-reranker on vLLM, Neo4j Community, Langfuse, Prometheus/Grafana, FastAPI, Modal. Every one of these is a system you can explain end to end in an interview, which is the actual deliverable of this stage. NVIDIA path: the swap targets are already isolated behind interfaces — NeMo Retriever NIMs for embedding, reranking and extraction; cuVS for GPU-accelerated index build if the corpus grows two orders of magnitude; TensorRT-LLM under vLLM if generation throughput becomes the constraint. Because S3 defined an Embedder Protocol and S5 defined tools behind schemas, each of those is a configuration change rather than a rewrite. Managed equivalents: Bedrock Knowledge Bases or Vertex AI Search replace boxes 2–4 wholesale; Neptune Analytics or Spanner Graph replace Neo4j; Bedrock or Vertex evaluation services replace the judge. Knowing precisely which boxes each would replace — and what you would lose — is the answer to “why did you build this yourself?”, and it is a much stronger answer than either “we always self-host” or “we always buy”.

Capstone exercise · the document that outlives the code Write a two-page architecture decision record for this system. For each of the nine build steps, record: the choice, the guard clause that produced it, the measurement that would reverse it, and the cost of reversing. Then do the part that makes it valuable — fill in the numbers from your own corpus: your recall@50, your hard-query share, your parse failure rate, your p95, your monthly bill. A stage of study produces understanding; this document produces a system someone else can operate, and it is the artefact that turns seven sessions of reading into something you can hand to a team or walk an interviewer through. Revisit it after the next embedding model ships and see how many decisions the trees still defend — that number is the real measure of how well the architecture was reasoned.

Term ledger · defined in this tab

the production wall: the source material’s name for the gap between a working prototype and a deployable system; almost entirely data engineering, not modelling.
content_hash: Hash of extracted text used as an idempotency key, so unchanged documents skip the pipeline entirely.
incremental refresh: Processing only the delta since the last run rather than rebuilding, gated on content_hash.
blue/green index: Building a new index version alongside the live one and swapping an alias after evaluation. The safe path for an embedding-model change.
soft delete: Marking rows unreachable via a filter rather than removing them. Instant revocation; insufficient for a true erasure obligation.
label-based ACL: Permission labels stored on each chunk and intersected with the user’s labels inside the retrieval query, never after it.
late-binding permissions: Resolving a user’s access at query time from the identity system rather than freezing it into the index at ingest.
derived-data leakage: Summaries, contextual prefixes, graph edges and caches inheriting the restrictions of the documents they came from — and usually not being labelled with them.
indirect prompt injection: Instructions hidden in a retrieved document rather than in the user's message. RAG-specific; mitigated at ingest and by output validation.
graceful degradation ladder: The pre-agreed fallback for each component failure. RAG can serve real value from partial results, which most applications cannot.
semantic cache: Serving a stored answer when a new query is semantically near a previous one. Dangerous where answers are time-sensitive.
dead-letter queue: Where documents that failed the pipeline go, so that failure is visible rather than silent.
staleness metric: Time since the newest indexed document per source. The monitor that catches confidently-wrong-about-the-past answers.
buy vs build boundary: Decided per component, not per platform: residency, differentiation, operational surface, cost against engineer-time, and rate of change.

Stage complete

You began the stage able to define an embedding and ended it able to defend, with numbers, every component of a production retrieval system — including the ones you chose not to build. The trees are the durable part: models will turn over, but “does the answer exist in one passage?” and “what does reversing this cost?” will still be the right questions next year.

RAG is an open-book exam. The retriever decides what's on the desk.

Every RAG system is two loops

RAG is a choice, not a default

Prompt / few-shot

RAG

Long context

Fine-tune

What you'll be able to defend at the end

Design

Retrieve

Escalate

Operate

How the seven sessions build

RAG & semantic search concepts

Ingestion — loading & preparing documents

Embeddings for retrieval & vector stores

Retrieval that actually works

Advanced architectures — agentic & graph RAG

RAG evaluation

Production RAG + capstone

How each session tab is built

The real-world problem, in plain words

Dense retrieval, and the reason a second model has to re-read everything

Why this session exists

Core concepts

2.1 · Dense retrieval — search as nearest-neighbour lookup

Asymmetry

Domain shift

There is always an answer

2.2 · Bi-encoder vs cross-encoder — the central trade in all of retrieval

2.3 · The online loop as a funnel

Query in — 1,000,000 candidates

2.4 · Grounded generation and the citation contract

2.5 · The decision tree: should this even be RAG?

Term ledger · defined in this tab

Reality check

On paper: why you cannot cross-encode your corpus

In the wild — three cited data points, 2026

Apply to my stack — lab

Ingestion: the failures nobody sees, because nothing throws an exception

Why this session exists

Core concepts

2.1 · Parsing — three families, and why you will use two of them

Heuristic / geometric

Layout models

Vision-language models

Tables: the part that decides whether fintech RAG works

1 · keep it as a table

2 · summarise it

3 · extract it out

2.2 · Cleaning — small, boring, and worth more than a model upgrade

Strip the furniture

Deduplicate boilerplate

Expand the jargon

Generate hypothetical questions

2.3 · Chunking — six strategies on one axis

A · Fixed-size (character or token) splitting

Overlap geometry — what the sliding window actually buys

Two 2026 techniques that changed the ceiling

Prepend the chunk's place in the world

Embed first, split second

2.4 · Metadata — the cheapest retrieval feature you will ever ship

2.5 · The decision tree: which chunking strategy?

Term ledger · defined in this tab

Reality check

On paper: what one chunking decision costs across the whole corpus

In the wild — three cited data points, 2026

Apply to my stack — lab

The index: two irreversible choices and a graph you have to understand

Why this session exists

Core concepts

2.1 · Choosing an embedding model without trusting the leaderboard

Matryoshka: buy fewer dimensions without retraining

Qwen3-Embedding / BGE-M3 / Nemotron

Gemini / Cohere / Voyage / OpenAI

NeMo Retriever embedding NIM

2.2 · Approximate nearest neighbour — HNSW and IVF

The four knobs you will actually turn

M

ef_construction