GenAI Foundations
Stage 01 · Line A — the genaipros path · concepts before code
This stage has one job: make the inside of a language model stop being a black box. By the end you should be able to trace a single sentence all the way through — how it gets chopped up, how it becomes numbers, how those numbers look at each other, and how one new word comes out the far end — and explain each step to someone who has never heard of any of it.
The spine of the stage. Everything you learn hangs somewhere on this line. The dashed return path — feeding the model's own output back in as input — is the single fact that explains why inference is expensive and why a KV cache exists.
Each session is a self-contained lecture. You do not need the previous one to follow the current one — but they do stack in a deliberate order, from the outside of the model inward, and then back out to how you talk to it.
S1 gives you the map: what kinds of language model exist and why. S2 zooms into the very first thing that happens to your text — it gets cut into pieces. That cut determines cost, speed, and which languages the model is good at.
Once text is numbers, S3 opens the engine: attention, what it actually computes, and why the cost of generating token 5,000 is not the same as generating token 5. That's where the KV cache earns its existence.
Knowing how the loop works changes how you write for it. S5 is prompt engineering as an engineering discipline, ending in the decision every team eventually faces: prompt, retrieve, or fine-tune?
The only code-along in the stage: a tiny GPT from scratch, sitting between S3 and S5. Nothing makes attention permanent like having typed it — and the parameter count you work out at the end explains why mixture-of-experts splits the layer it does.
All five sessions are built. S2 is open by default — it is the one to read next.
What language models are
Representation vs generative. Encoder-only, decoder-only, encoder–decoder. Open vs proprietary. How LLMs are trained, at altitude. the source material.
Tokens & embeddings
Tokenizers and how they're trained, vocabulary size trade-offs, token vs sentence embeddings, why tokens are money. the source material.
Inside the transformer
Attention in full, multi-head, why the KV cache exists, decoding and sampling strategies. the source material.
Build a tiny GPT — code-along
From scratch, small enough to fit in your head. ~60 lines of PyTorch, a free GPU, half an hour. the source material.
Prompt engineering
Zero-shot, few-shot, chain-of-thought, structured output, system prompts. Closes with the prompt vs RAG vs fine-tune decision tree. the source material.
Stages either side of this one
| Complete | Deploying models — engines vs orchestration vs platforms. You already know how to run a model. This stage explains what you were running. |
| Here | GenAI foundations — the model from the inside. |
| Next | Inference — where the KV cache, batching, and quantization stop being intuitions and become numbers you can predict. |
| Say this | And you get |
|---|---|
| next | The next session's tab, added to this same file. Nothing earlier is touched. |
| re-teach | The current tab rebuilt concepts-only: slower, more visuals, fewer words. |
| go deeper on X | One concept expanded to full trade-off depth, inside the tab it belongs to. |
| more visual | Text walls converted into diagrams. |
| ground it | Fresh web sources and current real-world examples pulled in. |
| fix file | The file rebuilt with every tab preserved, if the tabs or scripts break. |
A colour convention used everywhere in this notebook
Every diagram in every session uses these two colours for these two jobs. If a box is teal it produces numbers you compare; if it's rose it produces words you read. S1 explains where the split came from — and where, in 2026, it has started to blur.
Seventy years of people trying to turn words into numbers without throwing away the meaning. This session is the whole map — every term defined, nothing assumed.
Store the word cat on a disk and you have three bytes: 99, 97, 116. Store dog and you get 100, 111, 103. Nothing about those two triples says the words are related. Now store catastrophe — it starts with the exact same three bytes as cat and has nothing to do with a cat.
That is the entire problem. Text is unstructured: it carries enormous meaning for a human reader and almost none for a machine, because the meaning lives in relationships between words, not in the characters themselves. Every technique in this session is an attempt to close that gap — to find a numeric representation of language in which similar things end up near each other and different things end up far apart.
The gap this whole field exists to close. Note what is being asked for: not a dictionary, not a rulebook, but a geometry — a space where "close together" happens to mean "similar in meaning." Every model in this session is a different attempt at building that space.
One warning before we start. The word large in "large language model" is doing almost no work. There is no size threshold. A 300-million-parameter model that produces excellent search vectors and a 700-billion-parameter model that writes essays are both routinely called LLMs. Treat "LLM" as a loose family name and pay attention to the two things that actually matter: what job does it do, and what shape is it built in.
Each step below solved a real failure of the step before it. Click through them — the point is the sequence of failures, because modern architectures are still visibly shaped by which problem they were built to dodge.
Imagine summarising a shopping trip by handing someone the receipt with the order of items scrambled and the layout of the shop removed. You know what was bought and how much of each. You have no idea in what order, or why. That's bag-of-words: a document becomes a tally of its words, and the order goes in the bin.
Pass 2 · MechanismThree steps, and two new terms fall out of them.
[2, 0, 1, 3]. Each position in the list
means something specific and always means the same thing. A vector is how nearly all machine learning
represents a thing.
Bag-of-words in full. It is a genuine representation — fixed length, comparable, cheap — and it is the first time text became something a computer could do arithmetic on. The pink panel is the failure that every later step is trying to fix.
Two costs, one surprising upside.
And yet it is not dead. Exact-word matching is a real strength when someone searches for a product code or a legal citation, and no amount of learned semantics beats a literal match there. Production retrieval systems in 2026 routinely run a keyword scorer (BM25, a weighted descendant of word counting) alongside a neural retriever and merge the results — usually called hybrid search. Counting words survived by becoming half of something better.
Suppose that instead of a slot per word, you gave every word a position on a map. Not a map of places — a map of meaning, where things that mean similar things are near each other. King sits near queen. Tuesday sits near Wednesday and nowhere near saxophone. You have no idea what the map's north-south axis "means," and you don't need to. You only need distances to be honest.
Nobody writes them by hand. They are learned, by a technique introduced in 2013 called word2vec, and the trick is delightfully indirect: train a model on a task you don't care about, then throw away the model and keep a by-product.
Word2vec, 2013. Note there are no human labels anywhere: the "right answer" is harvested from raw text itself, which is why this scales to the whole internet. That property has a name — self-supervised learning — and it is why LLMs became possible at all.
The dimensions do not correspond to human concepts. It is tempting to imagine axis 12 is "royalness" and axis 40 is "plural" — a useful lie for a first pass, and false. In a trained model the meaning is smeared across all the dimensions at once, and no individual axis is interpretable. What is reliably true is only the geometry: distances and directions are meaningful even when coordinates aren't.
Word2vec embeddings are static. Each word gets exactly one vector, computed once, downloadable as a file. So the word bank gets a single point in space that has to serve both "I deposited the cheque at the bank" and "we sat on the river bank." It lands somewhere unhelpfully in between and is wrong for both.
Meaning depends on context. A representation that ignores context has a hard ceiling — and everything from here on is the story of getting context into the vector.
The same idea applies at different levels of granularity, and mixing them up is a common and expensive mistake.
Token / word embedding — one vector per token. This is what lives inside a language model as its input layer, and what word2vec produced.
Sentence embedding — one vector for a whole sentence or paragraph, so you can compare passages rather than words. This is the workhorse of semantic search and retrieval-augmented generation. When someone says "the embedding model" in a production context, they almost always mean this.
Document embedding — one vector for an entire document. Bag-of-words was, in its clumsy way, already a document embedding.
Embeddings are also not limited to text. Images, audio, and even users and products get embedded the same way, into the same kind of space — which is what makes "search a photo library with a text query" possible: put both in one shared space and measure distance.
Here is how translation worked before 2014. One network reads the whole English sentence and compresses everything it understood into a single fixed-size vector. A second network reads only that vector and writes the French. Imagine reading a paragraph, being allowed to write one index card, and then having to translate the paragraph from the card alone with the original taken away. Short sentences: fine. Long ones: the card runs out of room.
The 2014 fix was almost embarrassingly direct: stop throwing the source away. Instead of handing the decoder one summary vector, keep the encoder's per-word states available, and let the decoder decide, at each output word, which input words to weight most heavily.
Producing the French lama's? Put most of the weight on the English llamas. The bottleneck disappears, because the decoder can reach back to any source word at any time.
Attention arrives in 2014 as a patch on recurrent networks, not as a replacement for them. It fixed quality. It did not fix speed — and the speed problem is what triggered the next move.
The recurrence is inherently serial. Word 10's hidden state cannot be computed until word 9's exists. A GPU is a machine with tens of thousands of cores that all want work simultaneously; handing it a strictly sequential dependency chain wastes almost all of them. Training on internet-scale text was therefore impossibly slow. The bottleneck had moved from quality to hardware utilisation — which turns out to be the constraint that has shaped every architecture since.
The 2017 paper's title was the whole argument: Attention Is All You Need. If attention already lets any position look at any other position, why keep the slow sequential machinery whose only job was carrying information along the sequence? Drop it. Let every word look at every other word directly, all at once.
The analogy: the RNN is a game of telephone down a line of people, where information at position 1 reaches position 10 only by being whispered through everyone in between. The Transformer puts everyone in one room and lets each person listen to all the others simultaneously. Same conversation, one round instead of ten — and crucially, all the listening happens in parallel, which is exactly the shape of work a GPU wants.
The original design had two halves, an encoder stack and a decoder stack. Click a labelled part below to see what it does and why it's there.
Six clickable regions. Keyboard: tab to a box and press Enter.
Self-attention lets every token see every other token. With n tokens that is n × n pairs of comparisons. Double the input length and you quadruple the attention work — cost grows quadratically. At 1,000 tokens nobody notices. At 100,000 tokens it dominates everything.
That single fact is the reason the 2026 architecture landscape looks the way it does. Nearly every "innovation" in a recent model card — sliding-window attention, grouped-query attention, multi-head latent attention, sparse attention, linear-attention hybrids — is a different bargain struck against that quadratic term. Session 3 takes them apart. For now, just register the shape: the Transformer traded a serial bottleneck for a quadratic one, and got a hardware-friendly design in return.
In 2018 the field split the Transformer in half, twice, and got two model families that dominated the next several years. This is the most useful distinction in the whole section, so it gets its own colour scheme, used everywhere in this notebook.
Keep the encoder stack, delete the decoder. Nothing is masked, so every token sees the entire sentence in both directions at once — the B in BERT is "bidirectional."
Trained by masked language modelling: blank out roughly 15% of the tokens and make the model guess what was removed. To fill a blank well you must understand everything around it, so the model is forced to build rich contextual representations. Again: no human labels, the answer is the text itself.
What comes out: one vector per token, plus a special [CLS] token whose vector
is treated as a summary of the whole input. Vectors, not words. It cannot write you a sentence and was
never meant to.
What it's for: classification, clustering, semantic search, retrieval, reranking, deduplication — anything where you need to compare or sort text rather than produce it.
Keep the decoder stack, delete the encoder. Self-attention stays masked, so each position sees only what came before it.
Trained by next-token prediction: given everything so far, predict the next token. That's the entire objective. Run it over a large enough corpus and grammar, facts, style, arithmetic, and code structure all fall out as side effects of getting good at that one guess.
What comes out: a probability distribution over the whole vocabulary for the next token. Pick one, append it, feed the longer sequence back in, repeat. That loop is what you are watching when text streams onto a screen.
What it's for: writing, summarising, answering, reasoning, coding, tool use, agents.
The distinction is about the job, not the wiring
This is the part people get wrong. Representation vs generative describes what a model is trained and used for, not what shape it is. It happened to line up neatly with encoder-only vs decoder-only for several years. In 2026 it no longer does: many of the strongest embedding models are built on decoder backbones — a generative architecture, post-trained to emit one vector instead of a stream of words. See §3 for the current leaderboard. Hold on to the job distinction and treat the architecture correlation as historical.
The original 2017 Transformer kept both halves, and that design — sometimes called sequence-to-sequence or seq2seq — is still the natural fit when the output is a transformation of a specific input with tight alignment between them: translation, grammar correction, speech transcription, some summarisation. T5 and the BART family are the well-known examples.
It faded from the conversation for a practical reason rather than a technical one. A decoder-only model can do translation perfectly well by just being asked to, and one general model that does everything beats a fleet of specialised ones on operational grounds — one set of weights to serve, one endpoint, one thing to keep updated. Seq2seq quietly survives inside speech and translation systems where the alignment structure genuinely pays for itself.
GPT-1 in 2018 had 117 million parameters and was trained on about 7,000 sources plus web text. GPT-2 had 1.5 billion, GPT-3 had 175 billion. For several years the strategy was visibly "make it bigger."
That story is now more complicated in two ways. First, mixture-of-experts broke the link between total size and cost: a model can hold 700 billion parameters but only route each token through 40 billion of them, so you pay compute for the small number while getting the capacity of the large one. Most large open-weight models in 2026 are built this way. Second, and more importantly, when a widely cited 2026 survey compared ten open-weight architectures released in early 2026 he concluded that modelling performance is likely attributable not to the architecture design itself but rather to dataset quality and training recipes. Architecture now mostly buys you efficiency; data and post-training buy you capability.
Parameter count is therefore a poor proxy for quality and a decent proxy for memory cost. Use it for the second thing only.
Classical machine learning is one step: you have a task, you have labelled examples, you train a model for that task. It can do that one thing and nothing else.
Language models split this into a general education and then a specialisation. First an enormously expensive phase where the model reads a very large fraction of the written internet and learns what language is — grammar, facts, styles, the shape of an argument. Then a much cheaper phase that teaches it a job: follow instructions, be helpful, be safe, work through problems step by step.
-it or -instruct. It is almost always the one you want.
The material presents a clean two-step process. That was accurate for 2024 and is now the outline rather than the picture. Stage 4 — reinforcement learning against automatically checkable answers — is the change that produced "thinking" models.
Stage 4 deserves a plain-English explanation because it is the most consequential recent change and it is not in the material.
The current consensus pipeline is described plainly in the literature: as of 2026, LLM training follows a standard pipeline of pretraining, then supervised fine-tuning, then reinforcement learning via verifiable rewards, with a preference-optimisation step commonly sitting between the last two. One 2026 survey of the practice puts the consequence bluntly: post-training now accounts for the majority of a model's usable capability.
Pass 3 · Two consequences that will bite you laterTwo ways to get a model. Rent access to one running on someone else's hardware, reached over the network — you never see it. Or download the file and run it on hardware you control.
Buys you: no GPUs, no ops, no cold starts, top-tier capability, someone else's problem when it breaks.
Costs you: per-token pricing forever; your data leaves your perimeter; no fine-tuning beyond what the vendor exposes; the model can change or be deprecated underneath you; rate limits are theirs to set.
Buys you: data never leaves; fixed and predictable cost per GPU-hour; the exact version pinned forever; full freedom to quantize, fine-tune, and profile; no vendor in your critical path.
Costs you: you now own an inference stack. Memory sizing, batching, throughput, cold starts, upgrades, on-call. The skill is real and so is the toil.
A licence note worth internalising early: "open weights" and "you may use this commercially" are independent claims. Some strong open-weight models ship research-only or revenue-capped licences. Check the licence field on the model card before you build a business on the file.
Read it as guard clauses. Start at the top, answer each question, follow no ↓ until something exits yes →. If you fall all the way through, the box at the bottom left is your default — and the default is correct far more often than people expect.
The tree is ordered by how expensive the wrong answer is. Reaching for a generative model when a 100-million-parameter classifier would do is the single most common and most costly mistake in production language systems.
Note the default has moved. In 2024 a source could reasonably say closed models are simply more capable. In 2026 the aggregate gap between the best open-weight and best proprietary models is measured in single benchmark points, which changes where the burden of proof sits.
You need nothing but a pen. Three sentences:
Step 1 — vocabulary. Pool the unique tokens across all three, in a fixed order:
Step 2 — count. One number per slot, per sentence:
Step 3 — cosine similarity of A and B. Multiply the pairs, add them up, divide by the two lengths:
0.875 out of a maximum of 1. Reasonable — the sentences really are similar. Now the same calculation for A and C:
Sit with that number for a second
The method just declared "the cat sat on the mat" and "the mat sat on the cat" to be the same sentence. Not similar — identical, 1.000. Two different animals are on two different objects and the representation cannot tell. You have now personally derived, with six numbers, exactly why the field needed embeddings and then needed attention. Everything in §2 after bag-of-words exists to make this number come out below 1.
Cosine measures the angle between two vectors and ignores their length. That is usually what you want, because length tracks document size: a 2,000-word article and a one-paragraph summary of it point in nearly the same direction but have wildly different magnitudes. Straight-line distance would call them dissimilar; cosine correctly calls them close.
Practical note for later: if vectors are normalised to length 1 — which most embedding APIs do for you — cosine similarity and the dot product become the same operation. That is why vector databases advertise "dot product" and "cosine" as interchangeable options.
Two terms from §2, parameter and VRAM, meet each other. This arithmetic is the first question to ask about any model you are thinking of self-hosting, and it is one multiplication.
Take a small model with 8 billion total parameters, on a 24 GB accelerator:
"Everything else" is not a rounding error — it is the working memory for every request in flight, and it grows with how many users you serve and how long their conversations are. Going from 8 GB of headroom to 16 GB can be the difference between four concurrent conversations and forty. That is the entire commercial argument for quantization in one line of arithmetic.
The trap in the multiplication
Some model families advertise an effective parameter count that is smaller than the number of parameters you must actually load. Mixture-of-experts models are the obvious case: a model may route each token through 40 billion parameters while all 700 billion sit in memory waiting to be routed to.
Google is explicit about this for the small Gemma 4 variants, where extra embedding tables are stored per layer: the static weights need more memory to load than the effective parameter count would suggest. Always multiply by total parameters, never by the marketing number.
1 · The decoder-only design won, and then quietly diversified
A February 2026 field round-up walks through ten open-weight architectures released in a six-week window — Arcee Trinity Large, Kimi K2.5, StepFun Step 3.5 Flash, Qwen3-Coder-Next, GLM-5, MiniMax M2.5, Nanbeige 4.1 3B, Qwen3.5, Ant Group's Ling 2.5, and Cohere's Tiny Aya. §1's central claim holds completely: every one of them descends from the original GPT design.
But the surface has changed. Mixture-of-experts is the default at scale — GLM-5 activates about 40B of 744B parameters per token. The attention mechanism is now a design axis rather than a given: sliding-window attention in Trinity and Tiny Aya, DeepSeek-style multi-head latent attention in Kimi K2.5 and GLM-5, and Gated DeltaNet hybrids in the Qwen3.5 line. The survey's own conclusion is the load-bearing one for a beginner: the differences in capability trace back to training data and recipes more than to architectural choices.
magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight · Feb–Apr 2026
2 · Gemma 4 — every §1 concept visible in one model card
Released 2 April 2026, and useful here because you can point at each §1 term in turn. It is a decoder-only Transformer — the technical report says so in its first architecture sentence, and it lists pre-norm and post-norm RMSNorm together with QK-Norm as its stabilising choices. It ships as a family of five sizes rather than one model, spanning both dense and mixture-of-experts designs, with a 256K context window and both base and instruction-tuned variants of each size.
The small variants illustrate the effective-parameters trap from the arithmetic above exactly: E2B and E4B use per-layer embeddings, giving 2.3B and 4.5B effective parameters out of 5B and 8B total. And configurable thinking modes across the family are precisely the stage-4 post-training artifact described in §2.6 — a behaviour that was trained in, exposed as a switch.
ai.google.dev/gemma/docs/core · huggingface.co/google/gemma-4-E4B-it · arxiv.org/abs/2607.02770
3 · Representation models are thriving — but they aren't encoders any more
§1's teal-versus-pink split has half come apart, and it's worth seeing how. The job split is stronger than ever: retrieval, clustering, classification and reranking all still need a model that emits a vector, and every RAG system on earth runs one. What changed is the wiring underneath.
The models topping the Massive Text Embedding Benchmark in mid-2026 are largely built on decoder backbones rather than BERT-style encoders — Alibaba's Apache-2.0 Qwen3-Embedding-8B, Google's Gemini Embedding line, and Tencent's KaLM-Embedding-Gemma3-12B, which is built on a Gemma 3 backbone and leads the official multilingual board. Take a generative architecture, post-train it to emit one vector instead of a stream of tokens, and it beats the purpose-built encoders.
So: keep the distinction, but hold it at the level of what the model is for. "Encoder-only means representation" was a true generalisation in 2024 and is a historical note in 2026.
MTEB / MMTEB leaderboard snapshots, Mar–Jul 2026
| the source material (Sept 2024) | As of July 2026 |
|---|---|
| Phi-3-mini (3.8B) as the default small model to learn on | Still a fine teaching model, but the small-model class has moved on several generations. Current picks in the same size band: Gemma 4 E2B/E4B (Apache 2.0), the small Qwen3.5 variants, Nanbeige 4.1 3B, Cohere's Tiny Aya for multilingual work. |
| GPT-3.5 / GPT-4 as the reference frontier; closed models "tend to be more performant" | Both halves have aged. The frontier moved several generations, and the open-weight gap narrowed from generations to benchmark points. Treat "closed is better" as a hypothesis to test on your own data, not a default. |
| Mamba and RWKV as promising alternatives that may displace the Transformer | They did not displace it — they got absorbed into it. Linear-attention blocks now appear as hybrid layers inside otherwise conventional transformer stacks: Gated DeltaNet in the Qwen3-Next and Qwen3.5 line, Lightning Attention in Ling 2.5. The pattern is a few full-attention layers for precise recall, many cheap linear layers for everything else. |
| Training is two steps: pretraining, then fine-tuning | Now four, with the last one the interesting part: pretraining → supervised fine-tuning → preference optimisation → reinforcement learning with verifiable rewards. Reasoning behaviour comes from that last stage. |
| "2023, the Year of Generative AI" | 2025–26 is the agentic and reasoning era. The interesting work has moved from architecture to post-training, tool use, and long-horizon task completion. |
| Roughly 800,000 models on Hugging Face; a free 16 GB T4 as the entry-level GPU | Both numbers have moved substantially. Treat any specific count in a printed source as a timestamp, not a fact. |
Useful for reading vendor documentation without getting lost in branding.
| Concept | Open source | NVIDIA | AWS | GCP |
|---|---|---|---|---|
| Find weights | Hugging Face Hub | NGC catalog, Nemotron family | SageMaker JumpStart | Vertex AI Model Garden |
| Serve a decoder | vLLM, SGLang, llama.cpp, Ollama | TensorRT-LLM, Triton, NIM microservices | Bedrock (managed), SageMaker endpoints | Vertex AI endpoints, GKE + vLLM |
| Embedding model | sentence-transformers, vLLM embed mode | NeMo Retriever, NV-Embed | Bedrock Titan / Cohere embed | Vertex AI text-embedding, Gemini Embedding |
| Post-train it | TRL, Axolotl, Unsloth, PEFT | NeMo Framework, NeMo-Aligner | SageMaker training jobs | Vertex AI custom training, TPU |
| Vector store | FAISS, Qdrant, pgvector | cuVS / RAFT | OpenSearch, Aurora pgvector | Vertex AI Vector Search, AlloyDB |
You have been running a Gemma 4 E4B endpoint on an L4 through vLLM, with a profile switch that ablates batching, prefix caching and FP8. Every flag in that file is now a §1 concept wearing a command-line argument.
| What's in your config | The concept it's an instance of |
|---|---|
| MODEL_NAME = "google/gemma-4-E4B-it" | Read the string as three claims. gemma-4 — open-weight family, decoder-only Transformer. E4B — 4.5B effective parameters out of 8B total, because per-layer embedding tables are counted differently from always-on compute. -it — the instruction-tuned branch of the §2.6 diagram. Drop the suffix and you get the base model, which will complete your prompt instead of answering it. |
| --limit-mm-per-prompt {image:0, video:0, audio:0} |
§1 lists multimodality as one of the LLM application shapes. You are switching that shape off. Gemma 4 is multimodal from the ground up with dedicated vision and audio encoders; the small variants carry a roughly 150M-parameter vision tower. Text-only mode reclaims that VRAM for the thing you actually multiply in worked example 2. |
| head_size 256 → no FlashInfer | The cleanest possible illustration of why model cards matter operationally. An architecture choice made during training — an unusually large attention head dimension — closes off an entire optimised attention backend at serving time. Nothing in your code caused it; you inherited it from a decision made in a training run. Session 3 explains what a head is and why its size is a choice. |
| max_model_len = 10000 | Gemma 4's context window is 256K tokens. You serve 10K. Both numbers are correct: the context window is what the model supports, the served length is what your VRAM budget supports, because reserving room for long conversations costs memory whether or not anyone has one. This gap is the single most useful thing to understand before the inference stage. |
| quantization="fp8" kv_cache_dtype="fp8" |
Worked example 2, applied. The first flag halves the weight bytes. The second halves the per-request working memory — which is the KV cache, and which you'll be able to derive from first principles after Session 3. Right now the honest summary is: autoregressive generation means re-reading everything you've written so far, so you cache it, and the cache is not free. |
| enable_thinking (per request) | Not an architecture switch. Gemma 4 ships with configurable thinking across the family, which is a stage-4 post-training behaviour exposed as a chat-template flag. Same weights, same layers, different elicited behaviour. Worth internalising: the reasoning trace is made of real tokens that cost real money and real latency, even when you don't render them. |
| Fixed financial-analyst system prompt, always first |
This is §1's autoregressive property being exploited deliberately, and it's the sharpest connection in this session. Because generation is left-to-right and every token attends only to what precedes it, an identical prefix produces identical intermediate state every time — so it can be computed once and reused. Your prompt ordering is not a style choice; it is what makes a cache hit possible. Session 5 turns this into a discipline. |
One small thing to do before Session 2
Open the Gemma 4 E4B model card and write down five fields by hand. No code, five minutes:
max_model_len you actually serveDo this for one model and you will do it reflexively for every model afterwards. It is the single highest-leverage habit in this stage.
And the fintech routing repo
Your intelligent router — cheap model for "what is EBITDA", capable model for "analyse these risk factors" — is the §2.9 decision tree implemented as running code, evaluated per request instead of per project. The cost table sitting next to it is the same tree scored in dollars. Two threads carry forward from here: Session 2 turns "tokens" into the unit those dollars are actually denominated in, and Session 5 turns the fixed system prefix into a deliberate prompt architecture.
The first thing that happens to your text is that something cuts it into pieces, and that cut decides your speed, your bill, and which languages you're good at. We'll count tokens by hand, look at why a tokenizer is itself a trained model, and find out why the same sentence costs twice as much in one language as another. Say next when you're ready.
The model has never seen your text. Something cut it up first — and that cut is a trained artifact that quietly sets your bill, your latency, your context limit, and how good the model is at Portuguese.
Session 1 ended with a promise: text becomes numbers. This section is about the machine that does the becoming. It is easy to skip past — it looks like plumbing — and skipping it is why teams get surprised by three things that all have the same root cause.
Nothing about the prompt changed. The users changed — and some languages cost far more per character than others through the same model.
A model that writes flawless code confidently miscounts the r's in "strawberry." Not a reasoning failure. It never saw letters.
A "128K context" holds wildly different amounts of actual document depending on what language and format that document is in.
All three are the tokenizer. Once you can see it, they stop being surprises and become arithmetic.
The tokenizer runs twice per request: once to encode your prompt, once to decode each generated ID back into characters. It is a separate downloadable artifact from the weights — and every model ships its own.
Think of a phrasebook with a numbered list of entries. Entry 4,876 is the word "email." Entry 27,746 is the fragment "apolog." To send a message you look up each piece and write down its number; the recipient looks the numbers back up. Neither of you ever transmits letters. That numbered list is a vocabulary, and the whole system is a tokenizer.
email), a word fragment
(apolog, izing), a punctuation mark, or a run of whitespace.Take the sentence "Write an email apologizing for the tragic gardening mishap." A typical subword tokenizer produces something like this:
Sixteen tokens for a nine-word sentence. The ratio between words and tokens is not fixed — it depends entirely on how well this particular vocabulary happens to cover this particular text.
The model has no access to the letters inside a token. To it, garden is entry #12,043 —
an opaque integer with no visible spelling. Ask it how many letters are in the word and it must have
memorised that fact from text, because it cannot look. This is the real reason for the famous
letter-counting failures, and the same reason models historically struggled with arithmetic: if
870 is one token and 871 is two tokens (871), the
model has to learn number sense through a representation that scrambles it. Several tokenizers now split
every digit into its own token specifically to fix this.
Click each to see what it buys and what it costs.
The single trade-off underneath all four
Every tokenizer is buying shorter sequences with a bigger vocabulary, or the reverse. Bigger vocabulary → each token covers more characters → fewer tokens per document → less attention work and more text per context window. But the embedding table is vocabulary size × embedding dimension, so a bigger vocabulary means more parameters spent on lookup rather than reasoning — and for small models that table can dominate the parameter count. Subword tokenization won because it sits at a good point on that curve, not because it is principled.
Here is the thing most people don't realise: a tokenizer is itself trained, on data, before the model is. It is not a rule written by a linguist. It's a compression algorithm that reads a corpus and works out which chunks are worth having their own entry.
The dominant method is byte pair encoding (BPE), and the idea is greedy and simple: start with individual characters, then repeatedly find the most common adjacent pair anywhere in the corpus and glue it into a single new token. Do that 50,000 times and you have a vocabulary.
Pass 2 · Mechanism — a toy BPE run you can followCorpus: low low low lower lowest. Start from characters, then merge the most frequent
adjacent pair, over and over:
Two things fall out of this that explain almost everything you'll see in the wild:
## (so "tokens" becomes token ##s).The clearest trend of the last three years. Llama 2 shipped a 32,000-token SentencePiece vocabulary; Llama 3 moved to a tiktoken-style byte-level BPE at 128,256; the Gemma line went larger still. Bigger vocabularies mainly buy efficiency on code and non-English text.
Cost: a bigger embedding table, which matters most for the smallest models.
Code-focused tokenizers often give each digit its own token, so 600 becomes
6 0 0. Verbose, but it means the model sees a consistent representation of numbers instead
of an arbitrary one where 870 is one token and 871 is two.
Code tokenizers assign one token to each common indentation run — up to dozens of spaces. Without this, a Python file costs a fortune in leading whitespace. A pure-prose tokenizer will be dramatically worse at code for this reason alone.
Beginning/end of text, and — since chat models became the norm — turn markers for user, assistant, and system. Also domain markers: fill-in-the-middle tokens for code completion, filename and repository tokens, citation delimiters for scientific models.
When you send a chat model a list of messages with roles, nothing magic happens. Those messages get
flattened into one string using the model's chat template, which wraps each turn in the special
tokens that model was post-trained with — something structurally like
<|user|> … <|end|> <|assistant|>.
The model learned during post-training that text after the assistant marker is its turn. Use the wrong template and you get a model that is technically working and subtly worse at everything: it may keep writing past where it should stop, answer as the user, or ignore the system message. The template ships alongside the tokenizer, which is why "load the tokenizer that came with the model" is not optional advice.
A token ID is just an index; there's no meaning in the number 4,876 being larger than 385. So the very first thing inside the model is a lookup table: row 4,876 of a big matrix holds a vector of numbers that is the model's representation of that token. That row is the token embedding, and Session 1 already told you where such vectors come from — they're learned.
But here is the piece that ties Session 1's loose end. Those looked-up vectors are static: every occurrence of "bank" starts from the identical row. The model's whole job over its stack of layers is to turn those static vectors into contextual ones, where the "bank" in "river bank" has drifted somewhere different from the "bank" in "savings bank." Static goes in the bottom; contextual comes out the top.
Every token gets both a static starting vector and a contextual finishing vector. Which of the two you want depends on the job — and that distinction is exactly what separates an embedding model from a generative one in practice.
Session 1 gave the outline. Here is the mechanism, because the same trick reappears when embedding models are trained today.
Skip-gram is how positive examples are generated. Slide a window over the text — say two words either side of a centre word — and emit a training pair for each neighbour. "not make a machine" with centre "make" yields (make, not), (make, a), (make, machine), and so on, each labelled 1 for "these are neighbours."
Negative sampling fixes an obvious problem: a dataset where every label is 1 is trivially solved by always answering 1. So for each real pair, throw in a few pairs
The general shape — pull genuine pairs together in the space, push random pairs apart — is called contrastive training, and it is still how modern sentence embedding models are trained. Only the scale and the pair-selection strategy changed.
A token embedding is per-token. But to search a document collection you need one vector per document, so you can compare them the way you compared bag-of-words vectors in Session 1. That single vector for a longer span is a text embedding (also sentence embedding, or just "an embedding" in casual production speech).
Pass 2 · Mechanism — pooling, and why pooling alone isn't enoughThe obvious approach: run the text through a model, get one contextual vector per token, and average them. This is called mean pooling and it works, in the sense that it produces a vector of the right shape which is somewhat meaningful.
It also underperforms badly, for a reason worth internalising: the model was never trained to make that average useful. Its objective was next-token prediction or masked-token filling — nobody ever rewarded it for putting semantically similar documents near each other. Purpose-built embedding models add a training stage that does exactly that, with contrastive pairs: query and its correct passage pulled together, query and wrong passages pushed apart.
[CLS] token; last-token pooling takes the final position, which
is what decoder-backbone embedding models use.Guard clauses again. Follow no ↓ until something exits yes →; the dark box bottom left is the default.
Guard 1 catches most real situations and people still get it wrong: swapping a tokenizer under a trained model doesn't degrade it, it destroys it. Row 4,876 means something specific to those weights.
Take one short system prompt and count tokens with the rules from §2.1: common words survive whole, rarer words split, punctuation is its own token, a leading space fuses onto the following token.
A useful rule of thumb for English prose: ~0.75 words per token, or about 4 characters per token. Now put that to work on a realistic workload — 10,000 requests a day, each with this fixed 12-token system prompt plus roughly 400 tokens of document and 100 tokens of answer:
The second-order effect is worse than the bill
Those extra tokens don't only cost money — they occupy the context window and they cost time. At 1.6× fertility, a "128K context" holds 1.6× less actual document, and every request does 1.6× more prefill work before the first token appears. A tokenizer mismatch shows up as a latency regression and a quality regression and a cost regression, which is why it's so often misdiagnosed as three separate problems.
1 · Vocabularies quadrupled, and it was mostly about other languages
The material's tokenizer tour tops out at GPT-4's roughly 100,000 entries, with Phi-3 and Llama 2 sharing a 32,000-token vocabulary. That range has shifted decisively upward. The driver is explicit in the literature: tokenizers were historically tuned for English and fell back to near-byte-level segmentation for everything else, which caps how much non-English text fits in a context. The response was to expand vocabularies sharply — Llama 3 to 128,256 entries and the Gemma line to over a quarter of a million.
The practical read: a model's vocabulary size is a rough proxy for how seriously its makers took multilingual and code workloads. Check it on the model card — it's the field you wrote down at the end of Session 1 without knowing why.
arXiv:2502.12560 · model cards for Llama 3 and the Gemma family
2 · The tokenizer premium on low-resource languages is now measured, not folklore
Practitioner writeups in 2026 describe the exact failure mode from the worked example above as a routine production incident: a multilingual support agent's costs jump with no code change because the user mix shifted, with prompts in some languages tokenizing at roughly 1.6× the per-character cost of English on an older vocabulary, and moving to a larger, newer vocabulary recovering a large fraction of that.
There is a matching research literature on fertility — tokens per word — as the metric to optimise when adapting a model to a new language, with the caveat that lower fertility does not automatically mean higher downstream accuracy. Cheaper is not the same as better; measure both.
futureagi.com/blog/what-is-tokenization-llms-2026 · arXiv:2601.13328 · arXiv:2510.13481
3 · The live research question is whether tokenizers should exist at all
This is genuinely new since the material. The most cited result is Meta's Byte Latent Transformer, which drops the vocabulary entirely and instead groups raw bytes into variable-length patches, with the boundaries chosen by a small entropy model. An 8-billion-parameter BLT trained at comparable compute matched a Llama 3 8B baseline while handling typos, code, and low-resource languages more gracefully. A related line, T-FREE, replaces the vocabulary with hashed character-trigram embeddings and reports the embedding table shrinking by roughly 85%.
Why it isn't in production everywhere yet: both require architecture changes, so neither is a drop-in swap for an existing model. Worth tracking rather than adopting. The reason to know about it now is that it clarifies what a tokenizer is — a compression heuristic we tolerate, not a law of nature.
arXiv:2412.09871 (BLT) · arXiv:2406.19223 (T-FREE)
| the source material (Sept 2024) | As of July 2026 |
|---|---|
| Tokenizer tour ends at GPT-4 (~100K) and Phi-3 / Llama 2 (32K) | 32K is now small. 128K–256K is the norm for models that care about multilingual and code performance. The direction of travel is one-way. |
| Subword tokenization presented as settled | Still true in production, but byte-level and tokenizer-free architectures now match subword models at scale in research settings. The assumption is being actively tested. |
| all-mpnet-base-v2 (768-dim) as the go-to sentence embedding model | A fine teaching model. Current production picks are decoder-backbone models such as Qwen3-Embedding (Apache 2.0, several sizes) or hosted options like Gemini Embedding; several are now natively multimodal, embedding text and images into one shared space. |
| Averaging token embeddings as the way to get a text embedding | Still the fallback, still underperforms. Use a model trained for the job; the gap is large enough to be the difference between a working and a broken retrieval system. |
| Concept | Open source | NVIDIA | AWS | GCP |
|---|---|---|---|---|
| Tokenize | HF tokenizers, tiktoken, SentencePiece | NeMo Curator tokenizers | Bedrock token counting APIs | Vertex countTokens |
| Train a tokenizer | HF tokenizers trainers, SentencePiece | NeMo Framework | SageMaker training job | Vertex custom training |
| Serve embeddings | sentence-transformers, TEI, vLLM embed mode | NeMo Retriever, NIM embedding microservices | Bedrock embeddings, SageMaker endpoint | Vertex text-embedding endpoint |
| Watch token cost | Langfuse, Phoenix, Prometheus | Triton metrics | CloudWatch + Bedrock usage | Cloud Monitoring + Vertex usage |
| What you have | What this session explains about it |
|---|---|
| tokens/sec in your benchmarks (41.5 → 934.4 across batch sizes) |
This is the metric that most needs an asterisk. A token is not a fixed amount of language, so tokens/sec is only comparable within one model. Phi-2 at 41.5 tok/s and a Gemma at 41.5 tok/s are not producing the same amount of English per second, because their vocabularies pack different amounts of text into a token. If you ever compare throughput across models, convert to characters or words per second first, or the comparison is meaningless. |
| Cost tables: $324 → $45/month at 100 tokens/request average |
Every number in that table is denominated in a unit the tokenizer defines. "100 tokens per request" is an assumption about your traffic that a change in user language or document format can invalidate without anyone touching the code — see the +59% in §3. Worth adding characters-per-token as a tracked metric next to cost, so a fertility shift shows up as a cause rather than a mystery. |
| Prefix caching on a fixed financial-analyst system prompt |
The cache matches on token IDs, not on strings. Two prompts that look identical to you but differ by a single space may tokenize differently and miss the cache entirely. This is why the fixed prefix has to be byte-for-byte identical every time — assembled from a constant, never rebuilt with string formatting that might vary. Your 91% cache hit rate is a measurement of exactly this discipline holding. |
| max_model_len = 10000 | Ten thousand tokens, not words. In English prose that's roughly 7,500 words; in a language your tokenizer covers poorly it could be under 5,000; in JSON or heavily indented code, less again. If long inputs are being truncated, this arithmetic is the first place to look. |
| Gemma 4's large vocabulary | The Gemma line has consistently shipped one of the largest vocabularies of any open model. That buys good fertility across 140+ languages and on code — real value for a multilingual workload — and costs a large embedding table, which is part of why an "effective 4.5B" model loads 8B of parameters. The trade-off from §2.2 made concrete in the model you're serving. |
| Intelligent routing by prompt complexity |
Your router classifies a query before choosing a model. Note that classification is exactly the §2.9 embedding-model job — and doing it with a small embedding model plus a classifier is roughly a thousand times cheaper than asking a generative model to judge complexity. If the router currently uses an LLM call, that's a measurable win sitting in plain sight. |
One small thing to do before Session 3
We open the black box in the middle of the diagram above. Attention computed properly, with queries, keys and values on a three-token toy you'll work through by hand — and then the payoff: why generating token 5,000 would cost 5,000× more than token 1 if nobody cached anything, and what exactly gets cached.
Attention, computed properly — three vectors, one softmax, one weighted sum. Then the payoff: the exact reason a KV cache has to exist, derived rather than asserted.
Everything strange about serving an LLM comes from one fact you can see on any chat screen if you watch carefully: the text arrives a piece at a time. That isn't a streaming effect layered on for show. It is literally how the thing works.
Ask for a 500-token answer and the model runs 500 complete passes through its entire stack of layers. Each pass takes the whole conversation so far, does all its arithmetic, and emits exactly one new token. Then software appends that token to the input and runs the whole thing again.
You can already see the problem before knowing any of the mechanism. Hold onto that: by the end of §2.6 you'll be able to say precisely which intermediate values are worth keeping and why the others aren't.
Step through the pass. Each stage below is a real component you'll see named in a model's config file.
Session 1 said attention is "a weighted blend of other positions." Here is where the weights come from, and the standard analogy is a good one because it's nearly literal.
Picture a library card catalogue. You arrive with a query — what you're looking for. Every source on the shelf has a key — a label describing what it's about — and a value — its actual contents. You compare your query against every key, which tells you how relevant each source is, and then you take a bit from each source's contents, weighted by relevance. Very relevant sources contribute a lot; irrelevant ones contribute almost nothing.
Now the twist that makes it self-attention: every token plays all three roles at once. Each token issues a query about what it needs from context, advertises a key saying what it offers, and holds a value that is what it actually contributes.
Click through the diagram. Attention is genuinely only a dot product, a softmax, and a weighted sum.
Keyboard: tab to a box and press Enter.
This is the section the whole stage has been building toward. Read it slowly.
Pass 1 · IntuitionGo back to §1's diagram. Every forward pass re-reads the entire sequence. So on pass 500, the model recomputes the keys and values for tokens 1 through 499 — and here is the crucial observation: those keys and values are identical to what they were on pass 499.
Why identical? Because of masking. Token 12 can only attend to tokens 1–12. Nothing that happens at position 500 can reach backwards and change token 12's representation. Its key and value vectors were finalised the moment it was processed and are frozen forever after.
So: compute them once, keep them, and on the next pass only compute the K and V for the one genuinely new token. That store is the KV cache. Not an optimisation someone thought up — a direct consequence of causal masking.
The asymmetry in the name is the tell. K and V are what a token offers to future positions, so they must persist. Q is what a token wants right now, so it's disposable. If you can explain that sentence, you understand the KV cache.
Caching splits generation into two phases with completely different performance characteristics. Every serving metric you'll ever look at is really about one of these two.
All prompt tokens go through the model in parallel — they already exist, so there's no sequencing constraint. This is one huge matrix multiplication that saturates the GPU.
Compute-bound. Limited by raw arithmetic throughput. Cost scales with prompt length.
The metric: time to first token (TTFT). A long prompt means a long wait before anything appears.
One token per pass, forever. The arithmetic per pass is tiny, but the model must read all its weights and the whole cache from memory to do it.
Memory-bandwidth-bound. Limited by how fast bytes move, not by arithmetic. The GPU's compute units are mostly idle.
The metric: inter-token latency (ITL). This is why batching helps so much — you amortise one weight read across many sequences.
Two consequences worth stating explicitly, because they explain a lot of otherwise confusing behaviour:
The cache is bytes on the GPU, sitting next to the weights, competing for the same VRAM. Its size:
Every term in that formula is an architectural decision someone made, and three of them are levers that a model designer can pull to make long context affordable. That is the entire motivation for the attention zoo in modern model cards:
Multi-head attention gives every head its own K and V. Multi-query attention shares one K/V across all heads — a huge saving, sometimes too aggressive. Grouped-query attention (GQA) is the compromise that won: heads are split into groups, each group sharing one K/V set. Cutting 32 KV heads to 8 cuts the cache by 4×.
Multi-head latent attention (MLA), from DeepSeek, goes further: store a compressed latent vector and reconstruct K and V on the fly. Now used in Kimi K2.5, GLM-5, and Ling 2.5.
If a layer only ever attends to the last 4,096 tokens, it only needs to cache 4,096 tokens — no matter how long the conversation gets. Models interleave a few full-attention layers (for genuine long-range recall) with many windowed ones. Gemma 3 used a 5:1 local-to-global ratio; Arcee Trinity and Olmo 3 use 3:1.
Sparse attention generalises this: a small "indexer" picks which tokens matter before doing the expensive part.
Store the cache at 8 bits instead of 16. Exactly halves it, for a small and usually acceptable quality cost. This is a serving-time flag, not an architecture choice — the one lever on this list you control without changing models.
Gated DeltaNet, Lightning Attention, and relatives replace attention in most layers with a fixed-size recurrent state that does not grow with sequence length. Qwen3-Next and Qwen3.5 mix these with full-attention layers at a 3:1 ratio; Ling 2.5 reports roughly 3.5× the throughput of a same-size conventional model at 32K context.
The catch: linear states retrieve less precisely than real attention, which is why a few genuine attention layers always remain.
The single most useful reframe in this session
Nearly every attention variant you will ever read about in a model card is answering one of two questions: how do I do less than n² work? or how do I store a smaller KV cache? Once you can sort a new technique into one of those two buckets, model cards stop being intimidating. There is no third bucket.
The forward pass ends with a probability for every token in the vocabulary. It does not choose. Choosing is a separate step, done in software, entirely under your control — and it's the difference between a model that sounds like a legal document and one that sounds unhinged.
Pass 2 · Mechanism — the four knobsDeterministic: the same prompt gives the same output every time. Equivalent to temperature 0. Excellent for extraction, classification, and anything you need to test. Tends toward flat, repetitive prose over long outputs.
Divide every logit by T before the softmax. T below 1 sharpens — the top token gets even more probability. T above 1 flattens — unlikely tokens get a real chance. It does not add knowledge or creativity; it only redistributes probability mass.
Rough guide: 0 for extraction, 0.2–0.4 for factual answers, 0.7–1.0 for drafting, above 1.0 for brainstorming and accepting nonsense.
Discard everything outside the top k tokens, then sample from what's left. Prevents catastrophic picks from the long tail. Blunt, because a fixed k is wrong in both directions: too permissive when the model is confident, too restrictive when it genuinely isn't.
Keep adding tokens, highest first, until their probabilities sum to p (say 0.9), then sample from that set. Adaptive: when the model is confident the set is tiny, when it's uncertain the set is large. This is why top-p is the more common default.
Note that guard 1 exits to something the material doesn't cover: constrained decoding, where the sampler is forbidden from picking any token that would break a supplied grammar. Session 5 returns to it — it's the difference between JSON that usually parses and JSON that always does.
Real models use vectors of 128 dimensions per head. We'll use two, so the arithmetic fits on a napkin. Sequence: "the cat sat". We are processing position 3, sat, and we want its output vector.
Step 1 — score. Dot product of the query with every key:
Step 2 — scale. Divide by √d where d is the head dimension (here 2, so √2 ≈ 1.414). This exists to stop the scores growing with dimension and pushing softmax into a corner where gradients vanish:
Step 3 — softmax. Exponentiate, then divide by the total:
Step 4 — combine. Weighted sum of the value vectors:
That's it. That is attention.
A dot product, a division, a softmax, a weighted sum. Real models do this with 128-dimensional vectors, across dozens of heads, across dozens of layers, on thousands of positions simultaneously — but the arithmetic in each head is exactly what you just did. Notice also that position 3 gave itself only 22% of the weight and 44% to "the": attention weights are learned behaviour, not intuition, and heads routinely do things that look strange in isolation.
Use the formula from §2.6 on a plausible small model: 32 layers, 8 KV heads (so it uses GQA), head dimension 128, stored in BF16 at 2 bytes per value.
Forty-one gigabytes — on a card that has 24 GB in total, most of which is already holding the weights. This is not an edge case; it is the central constraint of LLM serving, and it explains three things at once:
That last line is worth dwelling on. Grouped-query attention — a change to how the model was trained — is doing more for your serving economics than every flag you can set combined. It is also why the GQA-versus-MLA line in a model card is a deployment fact, not trivia.
1 · The KV cache became the thing architectures are designed around
The material presents GQA as a recent tweak used by Llama 2 and 3. Two years on it's the floor, and the interesting work is above it. A survey of spring 2026 releases finds DeepSeek's multi-head latent attention adopted by Kimi K2.5, GLM-5 and Ling 2.5, and DeepSeek Sparse Attention in GLM-5, noting these are changes aimed at cutting inference cost when the context is long. Both are, in the terms of §2.6, ways of shrinking the same formula.
He also documents which models stayed conservative: MiniMax M2.5 and Nanbeige 4.1 ship plain grouped-query attention with no further efficiency tweak, and M2.5 is among the most-used open models on OpenRouter. Simpler is still viable — the exotic variants buy long-context economics, not general quality.
magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight
2 · Sliding-window attention is now standard, with a published ratio
The material mentions sparse and sliding-window attention as efficiency ideas that GPT-3 interleaved with full attention. That interleaving is now a headline architecture spec with a stated ratio. the survey describes sliding-window attention as restricting each token to a fixed window of recent tokens, which reduces the per-layer attention cost from quadratic in sequence length to roughly linear in the window size, and records the ratios: 5:1 local-to-global in Gemma 3 and Xiaomi MiMo, 3:1 with a 4,096-token window in Olmo 3 and Arcee Trinity.
Read a ratio like that as a direct statement about serving cost: a 3:1 model caches full-length K and V for only a quarter of its layers.
magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight
3 · Attention is being partially replaced from the inside
Session 1 flagged that Mamba and RWKV never displaced the transformer. What actually happened is more interesting and lands squarely in this section: linear-attention blocks moved inside transformer stacks. Qwen3-Next and its successors mix Gated DeltaNet blocks with gated attention blocks at a 3:1 ratio, which is what enables a native 262K context — the previous generation managed 32K natively. Ant Group's Ling 2.5 does the same with Lightning Attention and reports roughly 3.5× the throughput of Kimi K2 at the same parameter count on 32K sequences.
The pattern is consistent across all of them, and it's the practical takeaway: keep a few real attention layers for precise recall, replace the rest with something whose state doesn't grow. The survey is explicit that the trade-off is real — DeltaNet retrieves less precisely than full attention, which is exactly why one attention layer per group survives.
magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight · arXiv:2412.06464
| the source material (Sept 2024) | As of July 2026 |
|---|---|
| GQA as a recent improvement used by Llama 2 and 3 | Now the baseline everyone starts from. The frontier moved to MLA, sparse attention, and linear-attention hybrids — all still solving the same KV-cache problem. |
| Flash Attention and Flash Attention 2 | Flash Attention 3 is standard on Hopper-class hardware and later. Note that these are exact implementations — same maths, better memory movement — not approximations, so they cost nothing in quality. |
| RoPE as the positional method | Still dominant, with variants: partial RoPE applied to a subset of dimensions, and some models dropping positional embeddings entirely in their global attention layers (NoPE) on the grounds that causal masking already leaks order information. |
| KV caching presented as a speed optimisation | Accurate but understated. In 2026 it is the capacity constraint. How many users you can serve concurrently is a KV cache question before it is anything else. |
| Sampling covered as temperature, top-k, top-p | Still the core, plus constrained/grammar-based decoding for structured output and speculative decoding for latency — a small draft model proposes several tokens and the big model verifies them in one pass. |
| Concept | Open source | NVIDIA | AWS | GCP |
|---|---|---|---|---|
| Fast attention kernel | FlashAttention, FlexAttention, FlashInfer | cuDNN fused attention, TensorRT-LLM | Neuron SDK kernels (Inferentia) | XLA fused attention (TPU) |
| KV cache management | vLLM PagedAttention, SGLang RadixAttention | TensorRT-LLM paged KV | Bedrock (managed, opaque) | Vertex (managed) or GKE + vLLM |
| Prefix / prompt caching | --enable-prefix-caching, RadixAttention | TensorRT-LLM reuse blocks | Bedrock prompt caching | Vertex context caching |
| Quantized KV | --kv-cache-dtype fp8 | TensorRT-LLM FP8 KV | Neuron quantization | Vertex serving config |
| Speculative decoding | vLLM / SGLang spec decode | TensorRT-LLM Medusa, EAGLE | Bedrock latency-optimised | Vertex speculative decoding |
Your profile ablation was built to measure these levers. Now you can predict the results before running them.
| Profile / flag | What §2 says will happen |
|---|---|
| "worst": max_num_seqs=1 enforce_eager=True |
One sequence at a time means decode is running at its worst possible efficiency: the full weight matrix is read from memory to produce a single token. Decode is memory-bandwidth-bound, so you're paying the entire bandwidth cost for 1/256th of the possible output. This profile is a direct measurement of that waste, and it's why it floors at around 40 tokens/sec. |
| "baseline" → "best": continuous batching, 256 seqs |
The 22× throughput jump in your benchmarks is exactly the prefill/decode asymmetry. One weight read now serves many sequences, so the per-token cost collapses. Note what did not improve proportionally: time-to-first-token, because prefill was already saturating the GPU. |
| --enable-prefix-caching (the "prefix" profile) |
Your fixed financial-analyst system prompt produces identical K and V vectors on every request — identical tokens, identical positions, identical masking. Cache them once, skip that portion of prefill entirely for every subsequent request. That's the 80% saving in your cost table, and it is purely a prefill-phase win: TTFT drops, decode speed is unchanged. |
| kv_cache_dtype="fp8" | Worked example 2, applied. Halves the bytes_per_value term, so the same VRAM holds twice the concurrent conversations. On a 24 GB card serving a model that already needs 8–16 GB for weights, this is often the difference between the engine booting and OOMing on the KV allocation. |
| gpu_mem_util=0.92 (0.95 OOM'd) |
Now you know precisely what OOM'd. vLLM pre-allocates a KV cache pool from whatever VRAM remains after the weights. At 0.95 the pool was sized against too little headroom for activations and fragmentation. The comment in your config is a KV-cache sizing note. |
| max_model_len=10000 vs Gemma 4's 256K window |
The gap is entirely the KV cache. Advertising 256K would require reserving cache capacity for 256K-token conversations, which on an L4 means serving approximately one user. Your 10K is the answer to "what context length can I afford at 256 concurrent sequences?" — a capacity decision wearing the costume of a context-length setting. |
| head_size 256 → FlashInfer unsupported |
Session 1 flagged this without explaining it. Now: head_size is the head_dim term in the cache formula — the width of each attention head's vectors. Gemma 4 uses an unusually large 256 where most models use 64 or 128. Optimised attention kernels are written with specific head dimensions compiled in; 256 falls outside FlashInfer's supported set. An architecture choice made during training closing off a kernel at serving time. |
| enable_thinking per request | Reasoning tokens are decode-phase tokens. They occupy the KV cache exactly like visible output, cost the same per token, and add to context length. A thinking response with a 2,000-token trace costs 2,000 tokens of cache growth and 2,000 inter-token latencies before the user sees the answer begin. Budget for it explicitly rather than treating the flag as free. |
| PagedAttention ("near-zero KV growth" in your README) |
Worth restating precisely, because that line is easy to misread. PagedAttention doesn't make the cache smaller — the formula in §2.6 is unchanged. It stops the cache being wasted, by allocating it in small fixed pages instead of one contiguous slab per sequence sized to the maximum possible length. The saving is fragmentation, and it's what makes 256 concurrent sequences feasible at all. |
One small thing to do before Session 4
num_hidden_layers, num_key_value_heads and head_dim in
the Gemma 4 E4B config. Plug them into the §2.6 formula.max_num_seqs=256 is not actually reachable at full
context — the engine will be scheduling fewer. That number is a ceiling, not a promise, and knowing the
real one is the difference between tuning and guessing.The only code-along in the stage. About sixty lines of PyTorch that contain every idea from this session — masking, queries and keys, softmax, heads, residuals — small enough to hold in your head, and trainable on a free GPU in half an hour. Concepts recap first, then the code.
About sixty lines of PyTorch containing every idea from Session 3. Small enough to hold in your head, trainable on a free GPU in half an hour, and structurally identical to the model you're serving in production.
Session 3 gave you the mechanism and a worked example. This session removes the last hiding place. When
you have written wei = q @ k.transpose(-2, -1) yourself and watched a loss curve come down,
attention stops being a diagram you've understood and becomes a thing you know.
The model you build is genuinely tiny — around 10 million parameters, working at the level of individual characters rather than subword tokens, trained on a megabyte of Shakespeare. It will produce text that looks like Shakespeare from across a room and is nonsense up close. That is the correct outcome. The point is not the output.
Every item in the teal column is something you'll write today. Every item in the pink column is a number in a config file. That asymmetry is the whole argument for doing this.
Before you start
Open a notebook with a GPU attached — a free Colab T4 is plenty. You'll need PyTorch, which is preinstalled there. Budget 30–60 minutes. The lineage of this code runs through Andrej the classic Let's build GPT video and the nanoGPT repository; the source material's version is derived from it, and this session teaches the concepts first and the code second.
Seven components. Read this list first; it's the map for everything below, and each entry names the Session 1–3 concept it makes concrete.
Session 2's tokenizer, at its simplest possible setting: one token per character, vocabulary of 65. No BPE, no merges. It lets you see the entire encode/decode round trip in four lines.
Session 3's block size made literal. Note the detail that surprises people: one block of 8 characters yields 8 training examples, because every position predicts its successor.
A model with no attention at all — each character predicts the next from a lookup table alone. It trains, produces garbage, and gives you a loss number to beat. Everything after this is an argument for why attention was needed.
The single most clarifying trick in the whole build. Session 3's causal mask turns out to be a lower-triangular matrix of ones — and averaging over "everything before me" is just a matrix multiply.
Session 3's worked example, in code. Q, K, V projections; scaled dot product; mask the future; softmax; weighted sum of values. Twelve lines.
Heads in parallel, then the block (attention + feedforward + residuals + norm), then stack the block N times, add embeddings at the bottom and an LM head at the top. Done.
Concept first, then the code that implements it. Type it rather than pasting; the point is the typing.
Concept. Session 2 said a tokenizer maps text to integers via a fixed vocabulary. A character
tokenizer is the degenerate case: the vocabulary is simply every distinct character in the corpus. Nothing
is learned; the "training" is a call to sorted(set(text)). It's a bad choice for a real model
— sequences get long and the model must spend capacity learning to spell — and a perfect choice for
learning, because there is nothing hidden.
Concept. The model must be measured on text it has never seen, or you learn nothing about whether it generalises or has simply memorised. Ninety per cent train, ten per cent held out. The whole corpus becomes one long integer array — there are no "documents" here, just a stream.
Concept. Session 3's context window, here called block size. Pick a block of 8 characters and look at what it contains: given "F" predict "i"; given "Fi" predict "r"; given "Fir" predict "s" — eight separate training signals from eight characters. This is why next-token prediction scales so well: every position in every document is a labelled example, for free, with no annotation.
Concept. A GPU wants many independent pieces of work at once — the same parallelism argument that
made transformers win in Session 1. Grab several random starting points and stack their blocks into a
batch. The resulting tensor shape (B, T, C) — batch, time, channels — is the shape you will
see in every transformer codebase you ever read.
Concept. This is the part worth slowing down for, because it demystifies the mask completely.
We want each position to see only itself and what came before. Start with the simplest possible version of that: let each position be the plain average of all positions up to and including it. Written as loops that's obvious and slow. Written as a matrix multiply it's one line — build a lower-triangular matrix of ones, normalise each row to sum to 1, and multiply.
Now the pivot. Rewrite the same thing using softmax, by starting from zeros, setting everything above the diagonal to negative infinity, and softmaxing each row. Identical result — but the numbers going into the softmax are no longer forced to be zero:
The insight the whole session turns on
Those two versions are numerically identical, but the second has a
slot where the first has a constant. Fill wei with zeros and you get a uniform
average. Fill it with learned, data-dependent affinities and you get attention. Softmax
with a -inf mask is not an implementation detail bolted on afterwards — it is the
causal constraint from Session 3, expressed in a way that leaves room for the model to have opinions.
Concept. Session 3's worked example, verbatim. Project each token into a query, a key, and a value. Score every query against every key. Divide by √(head size) so the scores don't blow up with dimension. Mask the future. Softmax. Weighted sum of values. If you did the three-token arithmetic in S3, you have already run this code in your head.
Compare line by line with the §3 worked example in Session 3. q @ k.transpose
is the dot products. ** -0.5 is the ÷√d. masked_fill is the causal mask.
softmax is the exponentiate-and-normalise. wei @ v is the weighted sum. There is
nothing else in there.
Concept. One head learns one kind of relationship. Run several in parallel with independent projections, concatenate their outputs, and project back to the model dimension — now the layer can track syntax and coreference and topic simultaneously. Then the feedforward layer: after attention has gathered context, this does the per-position processing. It expands to 4× the model dimension and comes back, which is where most of the parameters live.
x matterConcept. A block is attention, then feedforward. But look at the two additions. Each sublayer's output is added to its input rather than replacing it — a residual connection. This gives gradients a clean path from the loss all the way back to layer 1 during training, without which deep stacks simply don't train. And each sublayer's input is normalised first (pre-norm), which Session 3 noted is the modern ordering.
Concept. Session 3's forward pass, top to bottom. Token embedding table plus a position embedding table (this toy uses simple learned positions rather than RoPE — same job, simpler). Stack of blocks. Final norm. LM head projecting to vocabulary size. That's a GPT.
Read generate once more
Six lines, and they are the entire autoregressive loop from Session 3 §1.
Note logits[:, -1, :] — the model computed a prediction at every position and we
throw all but the last away. Note idx = torch.cat(...) — the output is appended to the input
and the whole thing runs again. And note what is missing: there is no KV cache here. This loop
recomputes everything, every step. Add a cache and this becomes a serving engine; that is very nearly the
only difference.
Concept. Compute the loss, ask PyTorch for the gradients, nudge every parameter downhill, repeat. The loss here is cross-entropy — how surprised the model was by the character that actually came next. A useful sanity anchor: a model that has learned nothing predicts uniformly across 65 characters, and ln(65) ≈ 4.17. If your first loss isn't near 4.17, something is wired wrong.
What you'll see: correctly formatted Shakespearean dialogue with speaker names, line breaks, and plausible English-looking words that are mostly not words. The model learned the shape of the text perfectly and its meaning not at all — a remarkably clean demonstration that next-token prediction picks up structure long before it picks up semantics, and that semantics is what the remaining four orders of magnitude of scale buy.
Having built one, the tempting conclusion is that you could train a real one. Usually you shouldn't. Here is when you should.
Guard 4 is the one teams skip. "The model doesn't know our product catalogue" feels like a training problem and is almost always a retrieval problem — cheaper, faster, and updatable without a training run.
You typed the architecture. Now work out how big it is, by hand, from the config
n_embd=384, n_head=6, n_layer=6, block_size=256, vocab_size=65. Every number below comes from
a line of code above.
Look at the ratio inside the block
Attention: 589,824. Feedforward: 1,179,648. The feedforward layer is exactly twice the size of
the attention layer — and that ratio holds in essentially every transformer, because it falls
out of the 4× expansion in FeedForward. Roughly two-thirds of a transformer's parameters are
in the part that does no mixing between tokens at all.
Which immediately explains something from Session 1 that may have felt arbitrary: mixture-of-experts splits the feedforward layer, not attention. That's where the parameters are. A model can hold eight copies of the feedforward layer and route each token to one of them, getting eight times the capacity for roughly the same compute per token — and it leaves attention completely untouched, because attention was never the expensive part parameter-wise.
1 · Training a GPT-2-quality model from scratch now costs about fifty dollars
The economics here have changed faster than anything else in this stage. the classic nanochat has become the canonical solo-developer starting point: a single script that runs the entire pipeline — tokenizer, pretraining, mid-training, supervised fine-tuning, optional RL on grade-school maths, eval, and a chat UI. Practitioner writeups in 2026 put it at roughly $48 on an eight-GPU H100 node, or about $15 on spot instances, in under two hours of wall clock.
Read that as an education budget, not a product budget. The same writeups are blunt that for almost every real use case the answer is still to fine-tune an open-weight model — pretraining makes sense only for a genuinely novel domain with billions of clean tokens no open model has seen, which is guard 2 of the tree above.
github.com/the classic walkthrough/nanochat · codersera.com/blog/self-training-small-llm-complete-guide-2026
2 · The nanoGPT speedrun is where architecture tweaks get stress-tested in public
Keller Jordan's modded-nanogpt turned "train this exact model to this exact loss" into a competitive benchmark: reach 3.28 validation loss on FineWeb using eight H100s, as fast as possible. The baseline — the well-known GPT-2 reproduction in llm.c — took 45 minutes. The repository now reports reaching the same target in under 90 seconds and on under 400 million tokens, against the baseline's 10 billion.
This matters for a learner because the leaderboard is an itemised list of what actually helps. As of April 2026 the record run is credited to the Muon optimiser, Flash Attention 3, an FP8 head, learnable cross-stream attention, and multi-token prediction — several of which you met in Session 3. It's the fastest way to see which ideas survive contact with a stopwatch.
github.com/KellerJordan/modded-nanogpt
3 · The 2026 default architecture, if you were building this for real
The toy above uses ReLU, LayerNorm, learned absolute positions, and plain multi-head attention — faithful to the 2017 design and easiest to read. Nobody ships that any more. The current template, and the one you should recognise in any config file, is decoder-only with RoPE, RMSNorm, SwiGLU, and grouped-query attention.
Each swap is a small, well-understood upgrade you can now explain: RoPE for relative position that generalises past the trained length (S3); RMSNorm because it's cheaper than LayerNorm and works as well; SwiGLU as a gated activation that outperforms ReLU at equal parameter count; and GQA because it cuts the KV cache by the group factor (S3 §2.6). Swapping these four into the code above is a genuinely good exercise — each is a handful of lines.
codersera.com/blog/self-training-small-llm-complete-guide-2026 · model cards across the Llama, Qwen, and Gemma families
| the source material | As of July 2026 |
|---|---|
| ReLU, LayerNorm, learned absolute positions, MHA | Correct for teaching, superseded in practice by SwiGLU, RMSNorm, RoPE, GQA. Swapping them in is a good follow-up exercise. |
| Free Colab T4 as the assumed hardware | Still works for this model. For anything larger, spot H100 hours and per-hour GPU rental have made real pretraining experiments accessible in a way they weren't in 2024. |
| GPT-2 (125M–1.5B) as the reference "real" model | Now the small end of small. The comparison that lands in 2026 is against 2B–8B open-weight models that run on a laptop — same architecture, four orders of magnitude more data. |
| Three-stage training: pretrain → SFT → DPO | Add a fourth: RL with verifiable rewards after preference optimisation. That's where reasoning comes from, as Session 1 §2.6 covered. |
| In your stack | What you can now see in it |
|---|---|
| Gemma 4 E4B's layer stack | Print the model config and you will recognise every field: num_hidden_layers is your
n_layer, hidden_size is n_embd, num_attention_heads
is n_head, intermediate_size is the 4× expansion in
FeedForward. Different numbers, same fields. |
| head_size 256 | In your code, head_size = n_embd // n_head — a derived quantity. Gemma sets it
explicitly and large. Now you can see why a kernel would care: head_size is the inner
dimension of the q @ k.transpose matrix multiply, and optimised kernels are compiled for
specific values of it. |
| Why your model is 8B total but "4.5B effective" |
The §3 parameter count explains where a model's mass sits: embeddings at the bottom, then blocks dominated 2:1 by feedforward. Gemma's per-layer embeddings add extra tables that are pure lookup rather than always-on compute — real parameters that must be loaded, doing no arithmetic per token. Hence the two numbers. |
| The FP8 quantization flag | You now know what's being quantized: those nn.Linear weight matrices, which are
essentially the whole model. Two-thirds of them are the feedforward layers. Quantization is turning
16-bit floats into 8-bit ones in exactly those tensors. |
Your generate loopvs vLLM's |
The most useful comparison in this session. Part 9's generate is correct and
unusably slow — it recomputes the entire sequence per token, one request at a time. vLLM is that same
loop plus: a KV cache, paged allocation so the cache doesn't fragment, continuous batching so new
requests join mid-flight, and prefix caching. Every one of your profile flags is a modification to
those six lines. |
| Considering a fine-tune later | The tree in §2.9 is the pre-check. Before any fine-tune, confirm the model is missing a skill or format rather than facts — facts go in a retrieval store. That distinction will save you more time than any hyperparameter. |
Optional stretch, if you enjoyed this
block_size
and watch what each approach does.generate. Time 500 tokens with and without. That measurement is
the most direct possible confirmation of Session 3 §2.6.n_head for the key and value projections only — you will have implemented
grouped-query attention.Back out of the model and up to the interface. Zero-shot, few-shot, chain-of-thought, structured output, system prompts — treated as engineering rather than folklore, with the caching consequences of each. Closes the stage with the decision every team eventually faces: prompt, retrieve, or fine-tune.
The weights are frozen. The architecture is decided. The only thing you control at runtime is the sequence of tokens you hand the model — which makes prompt design the highest-leverage engineering surface in the entire system.
Sessions 1–4 built up one mechanical picture: the model computes a probability for every next token, conditioned on every token that came before. Prompting is the act of choosing those preceding tokens so that the distribution you want becomes the likeliest one.
That framing kills most prompt mysticism immediately. "Please" doesn't work because the model has feelings; polite, well-structured requests appear in training data alongside careful, well-structured answers, so conditioning on one raises the probability of the other. Everything below is a technique for moving probability mass, and each one has a cost in tokens you can now calculate.
§2.9 turns this into a proper decision tree. For now, note the asymmetry: prompting is the only option you can test a hundred variants of before lunch.
A prompt is not one thing; it's an assembly of components, each doing a distinct job. Treating it as a single blob of text is why prompts become unmaintainable — you can't debug a paragraph, but you can debug six labelled parts.
Pass 2 · Mechanism — the six componentsClick each part of the prompt below to see what it does and when to include it.
Keyboard: tab to a band and press Enter.
Describing a task in words is one way to specify it. Showing two or three completed instances is another, and often far more precise — the way handing someone a filled-in form beats explaining the form. Few-shot prompting is that: worked pairs, right there in the context.
Pass 2 · MechanismWhy does it work at all? Because the model is a next-token predictor conditioned on everything before. Once the context contains three examples that all follow the pattern input → tight JSON object, the single likeliest continuation after the fourth input is a tight JSON object. You have not taught it anything; you have made the desired shape overwhelmingly probable.
This explains a finding that surprises people: examples teach format and style far more reliably than they teach judgement. Showing three examples of your exact output schema works beautifully. Showing three examples of correct medical triage does not make the model a clinician. Use examples for the shape of the answer, not the substance of it.
Pass 3 · The costs nobody budgets forAsk someone a hard arithmetic question and demand an instant answer and they'll often get it wrong. Let them work on paper and they get it right. Chain-of-thought is exactly that: instruct the model to reason step by step before committing to an answer.
Pass 2 · Mechanism — why it works, mechanicallyThis is worth getting precisely right, because the usual explanation ("it thinks harder") is wrong.
Recall Session 3: one forward pass produces exactly one token, and the amount of computation in a forward pass is fixed by the architecture. It cannot vary with question difficulty. So if a problem needs more computation than fits in one pass, there is precisely one way to get it: use more passes.
Chain-of-thought does that. Every reasoning token is another full forward pass, and — crucially — each one lands in the context where subsequent passes can attend to it. The model is writing intermediate results onto a scratchpad that it can then read. That's not a metaphor; the scratchpad is the KV cache.
More tokens literally means more computation. That is the whole trick, and it's why reasoning is expensive: you are buying compute by the token.
In 2024, chain-of-thought was something you asked for. In 2026 it is frequently something the model was
trained to do, in the stage-4 reinforcement learning from Session 1 §2.6 — which is exactly what
your enable_thinking flag toggles. Three consequences:
Self-consistency. Run the same reasoning prompt several times at non-zero temperature and take the majority answer. Reasoning paths that reach the right answer tend to agree; wrong ones tend to be wrong in scattered, different ways. Costs n× as much for a real accuracy gain on hard problems — a straightforward compute-for-accuracy trade.
Prompt chaining. Instead of one prompt doing five things, use five prompts each doing one, feeding into each other. Each step is separately testable, separately cacheable, and can use a different model — a cheap one to classify, an expensive one to write. This is the same instinct behind your complexity router, applied within a single task rather than across tasks.
You need JSON. You ask for JSON. You get JSON 97% of the time, and 3% of the time you get JSON wrapped in a friendly sentence, or a markdown code fence, or a trailing comma. At 10,000 requests a day that's 300 parse failures daily. Prompting cannot fix this, because prompting only shifts probabilities and 3% is what's left.
Pass 2 · Mechanism — constrained decodingThe fix comes from Session 3, not from this section. Recall that decoding is a separate software step that picks one token from the probability distribution. Nothing forces that step to consider all 256,000 candidates.
Constrained decoding attaches a grammar to the sampler. At every step it computes which
tokens could legally come next given the schema and what's been emitted so far, sets the probability of all
the others to zero, and samples from what remains. If the schema says the next character must be
" or }, no other token is reachable — not unlikely, unreachable.
Free, works everywhere, no engine support needed. Gets you most of the way. Cannot ever reach 100%, because it's shifting a distribution rather than restricting one.
Always pair it with a parse-and-retry. Always.
Structurally invalid output becomes impossible. Supported by most modern serving engines via a JSON schema or a formal grammar.
Caveat worth knowing: it guarantees the shape, never the content. A schema-valid object can still be completely wrong.
This is where the whole stage converges. Session 3 established that a shared prefix produces identical keys and values, so it can be computed once and reused. That fact has a direct implication for how you write a prompt, and it is the single most valuable practical thing in this session.
The rule in one line: most static first, most variable last. System prompt, then examples, then schema, then retrieved documents, then the user's question.
The mistake that silently destroys a 90% cache hit rate
Put anything variable at the very top and the entire cacheable prefix evaporates. The usual culprits, all of which look completely harmless in a code review:
"Current time: 14:32:07. You are a financial analyst…" — a timestamp, changing every
second.f"Session {uuid}. You are a financial analyst…" — a request ID.If you need a timestamp, put it after the static block, next to the user's question. It costs nothing there and everything at the top.
There's a broader shift behind this, and it has a name now. The field increasingly talks about context engineering rather than prompt engineering — Anthropic frames it as deciding which tokens should occupy the context window at every inference call, including everything that arrives from tools, retrieval and memory rather than from the prompt itself. The distinction matters most for agents, where a single task may run for dozens of steps and the context at step 47 is mostly residue from steps 1 to 46. Prompt engineering is a sentence; context engineering is the budget for the whole window.
The tree this whole stage has been pointing at. Guard clauses; follow no ↓; the dark box bottom left is the default and it is right far more often than teams expect.
One rule of thumb that survives contact with reality: retrieval for what the model should know, fine-tuning for how it should behave. Fine-tuning facts into weights is expensive, hard to update, and produces confident staleness.
Same workload as Session 2: 10,000 requests a day. A prompt with a fixed 800-token static block, a 1,200-token retrieved document, and a 50-token user question.
Now the counterexample, which is the part worth remembering:
Why this is the right note to end the stage on
Nothing in that calculation required knowing anything about prompt writing. It required knowing that generation is autoregressive (S1), that the cache is keyed on token IDs (S2), that identical prefixes produce identical keys and values (S3), and that prefill is compute-bound while decode is bandwidth-bound (S3). The prompt-engineering "tip" is a consequence of the mechanism. That is the whole argument for having learned the mechanism.
1 · Prompt engineering became context engineering
The material treats a prompt as a text field you compose. The 2026 framing is broader, driven by agents: the model's context on any given call is assembled from a system prompt, tool definitions, retrieved documents, conversation history, and memory carried between sessions — most of which no human typed. Anthropic's definition of context engineering is the practice of curating and maintaining the best possible set of tokens in the window during inference, and the discipline has grown its own techniques, including rule-based pruning of the context as an agent loop runs.
Practical read for you: your fixed system prefix is context engineering in miniature. The moment you add retrieval, memory, or tools, the question stops being "what should I write" and becomes "what should occupy this window, in what order, and what gets evicted first."
anthropic.com/engineering/effective-context-engineering-for-ai-agents · sourcegraph.com/blog/context-engineering
2 · Current lab guidance inverts the material's few-shot reflex
§6 introduces one-shot and few-shot early and enthusiastically. Anthropic's 2026 prompting guidance treats examples as an escalation rather than a starting point — add few-shot examples only when a clear instruction hasn't produced what you need — alongside two other recommendations worth adopting directly: give the model explicit permission to say it is uncertain rather than guess, which measurably reduces confident fabrication; and in long contexts, place the most critical information at the beginning or the end rather than buried in the middle.
That last one has a mechanical explanation you can now supply yourself: attention is a weighted average over thousands of positions, and material in the middle of a very long context competes with everything around it for weight.
claude.com/blog/best-practices-for-prompt-engineering
3 · Prompts are now version-controlled artifacts with tests
The largest cultural change. Platform guidance in 2026 assumes the prompt lives inside a product or an agent loop rather than a chat window, and treats it accordingly: OpenAI's guidance recommends keeping prompts in code under version control and pairing every change with tests and evaluations.
Concretely, that means: prompts in files, not string literals scattered through the codebase; a fixed eval set of 50–200 real inputs with known-good outputs; every prompt change scored against it before merge. Without this you are not engineering prompts, you are adjusting them and hoping — and because prompt changes have no compiler and no type system, hoping fails silently.
OpenAI prompting guidance · refontelearning.com/blog/prompt-engineering-types-2026
| the source material (Sept 2024) | As of July 2026 |
|---|---|
| Chain-of-thought as a prompting technique you request | Frequently trained in via RL and exposed as a mode switch. On a reasoning model, asking for it can be redundant or actively counterproductive. |
| Few-shot introduced early as a core move | Lab guidance now says start zero-shot and escalate only if needed. Examples cost tokens on every request forever and bias in ways you didn't intend. |
| Grammar / constrained sampling as an advanced curiosity | Standard in every serious serving engine and the correct answer whenever a machine parses the output. Prompting for JSON is the fallback, not the solution. |
| Temperature / top-p tables per use case | Still sound. Add: reasoning models often specify their own recommended sampling settings, and those are not boilerplate. |
| Prompt as a single composed string | Now one input among several — tools, retrieval, memory, prior agent steps. Context engineering is the umbrella term and the ordering of everything in the window is itself a decision. |
| No treatment of caching consequences | The largest omission. In 2026, prompt ordering is a cost decision before it is a quality decision — see §2.7. |
| Concept | Open source | NVIDIA | AWS | GCP |
|---|---|---|---|---|
| Structured output | Outlines, XGrammar, vLLM guided decoding | TensorRT-LLM guided decoding | Bedrock tool use / JSON mode | Vertex controlled generation |
| Prompt versioning | Langfuse, PromptLayer, files in git | NeMo Guardrails configs | Bedrock prompt management | Vertex prompt management |
| Evals | promptfoo, DeepEval, Phoenix, lm-eval-harness | NeMo Evaluator | Bedrock model evaluation | Vertex AI evaluation service |
| Prompt caching | --enable-prefix-caching | TensorRT-LLM KV reuse | Bedrock prompt caching | Vertex context caching |
| Guardrails | Guardrails-AI, LLM Guard | NeMo Guardrails | Bedrock Guardrails | Vertex safety filters |
| In your stack | What to do about it |
|---|---|
| Fixed financial-analyst system prompt, ordered first |
You already got the most important thing right, and now you know exactly why: it produces a stable cacheable prefix. Two things to verify. First, that it's a module-level constant, not built per request. Second, that nothing variable — no timestamp, no user ID, no session UUID — precedes it. Your 91% cache hit rate says this is holding; a drop in that number is the alarm. |
| 91% cache hit rate, 80% cost saving |
Worth promoting from a metric to an alert. Cache hit rate is a direct measurement of prompt hygiene, and it degrades silently — someone adds one harmless-looking dynamic field and nothing breaks, the bill just goes up. An alert at, say, below 70% catches the regression on the day it ships rather than at the end of the month. |
| Intelligent routing by prompt complexity |
This is §2.9's tree running per request, which is a good design. One refinement from this session: route on whether reasoning is needed as well as on model size. "What is EBITDA" needs neither a big model nor a thinking trace; "analyse these risk factors and rank them" may need the trace more than it needs more parameters. Two dimensions, not one. |
| enable_thinking per request | Exactly the right shape of control. Add a token budget to go with it — a reasoning trace is
unbounded by default and a runaway one can consume your entire max_model_len. Cap it,
measure the distribution of trace lengths, and route accordingly. |
| Cost tracking per token | You have the denominator. The missing numerator is quality. Without a fixed eval set, "we cut cost 86%" and "we degraded the answers" are indistinguishable. Fifty real financial queries with known-good outputs, scored on every profile change, converts your ablation from a performance benchmark into a proper cost/quality frontier — which is a substantially more valuable artifact. |
| Structured output from the analyst prompt |
If anything downstream parses the response, move from asking for JSON to enforcing it. vLLM supports guided decoding against a JSON schema; it removes an entire class of failure and costs nothing at inference time. |
| Prompts living in Python string literals |
The one structural change worth making. Move them to versioned files, give each a version identifier, log which version produced each response, and score changes against the eval set before merge. This is the 2026 consensus and it's also what makes your Langfuse tracing genuinely useful — prompt versioning is a first-class feature there and it's currently only half-wired. |
Check these honestly. Anything you can't do out loud, without notes, is worth a re-teach or a go deeper on X before moving on.
Everything you derived here as an intuition gets formalised as numbers you can predict: KV cache sizing, continuous batching, PagedAttention, quantization trade-offs, speculative decoding, and the throughput/latency frontier. You arrive already knowing why each of those exists, which is the hard part. Commands still available: re-teach, go deeper on X, more visual, ground it, fix file.