Every retrieval system eventually rediscovers an idea Donald Michie wrote down in 1968. He called it a memo function: if a computation is a pure function of its inputs, you never run it twice — store the result under a key built from the arguments, and the second call becomes a lookup, not a calculation. IBM’s developer team recently published a clean primer on cache-augmented generation (CAG), framing it as an improved version of retrieval-augmented generation (RAG). Strip the packaging and it is Michie’s memo function pointed at a language model: the tokens you never re-read are the cheapest tokens you will ever serve.

That is worth understanding structurally, not just operationally, because caching is where the cost of AI actually lives in 2026 — and where a working studio either saves real money or quietly bleeds it.

What it is: CAG sits at two levels, and it helps to see both. At the shallow level — the one IBM describes — you put a key-value store in front of your retrieval step. A query arrives, you check the cache; on a hit you return the stored answer, on a miss you query the external knowledge base and write the result back. At the deeper level, CAG means preloading a fixed corpus directly into the model’s attention state so there is no retrieval step at all at inference time. Both are the same instinct at different depths: identify the part of the work that does not change between requests, compute it once, and stop paying for it.

Why it works: The mechanism rests on two structures a computational designer will recognise. The first is the cost curve of attention itself. As our PAZ concept panel on transformer attention puts it, every token attends to every other token in one matrix multiplication, and the cost is O(n²·d) — quadratic in sequence length. That is why a frozen 40-page BEP re-tokenised on every query is not a rounding error; it is the dominant term. Caching removes the constant, unchanging prefix from the quadratic. The second structure is geometric, and it is where a semantic cache earns its name. A vector store answers “which stored key is nearest to this query embedding?” — and “which point is nearest to where” is exactly the question a Voronoi tessellation answers. As the PAZ Voronoi panel notes, one diagram serves every trade that has to decide nearest-neighbour membership. A cache hit is simply landing inside a Voronoi cell in embedding space that you have already answered for. Miss means you fell into an empty cell and must compute its answer, then plant a new seed.

The hard numbers come from the frontier, not the primer. PAZ covered this thread when DeepSeek shipped V4 — read that piece for the full mechanism, but the load-bearing figure is this: at a one-million-token context, DeepSeek’s V4-Pro (1.6 trillion parameters, 49 billion activated) reported spending roughly 27 percent of the single-token inference FLOPs and only 10 percent of the key-value cache its V3.2 predecessor needed; against grouped-query attention in bfloat16, the cache footprint dropped to around 2 percent. The bottleneck at long context is not compute, it is the bandwidth between HBM and the matmul engines as the KV cache grows linearly per layer. CAG and KV-cache compression attack the same enemy from opposite sides — one avoids re-reading, the other shrinks what is read.

←TODAY: In 2026 a cached answer can cost 12.5× less than a recomputed one — the exact write-vs-read gap ($3.75 vs $0.30 per million tokens) behind Anthropic’s March cache-TTL swing. →3012: The office that survives keeps the derivation of every cached answer, not the cache file, which will not outlive its plugin. Fulcrum: A cache is only correct while its invalidation rule holds — the key you can reconstruct is worth more than the value you stored under it.

Origins: The lineage is older than the language-model era and cleaner than the marketing. László Bélády proved the optimal cache-replacement bound at IBM in 1966 — the MIN algorithm, evict the entry whose next use is furthest away — and every real eviction policy since is an approximation of that oracle. Michie’s memo functions followed in 1968. The production tooling IBM recommends carries its own authorship: Memcached was built by Brad Fitzpatrick for LiveJournal in 2003; Redis was written by Salvatore Sanfilippo in 2009. RAG itself is a 2020 result — Patrick Lewis and colleagues at what was then Facebook AI Research named the pattern. CAG is not a new invention so much as the moment retrieval got expensive enough that memoization became the interesting layer. Naming that lineage is not pedantry; it is how a 25-year-old rebuilds the idea when the current SDK goes dark.

In practice: A Swiss studio should reach for CAG the instant it can separate the frozen from the live. The SIA norm text, the IFC4 schema, the office’s own BEP and LOIN definitions, the standard tender boilerplate — these are stable for months. That is your cache corpus: preload it, key it, stop re-billing it. The live half — this project’s tender addenda, the current clash report, yesterday’s site minutes — stays on classic RAG, because it changes and a stale cached answer there is a defect, not a saving. The failure mode IBM lists first is the real one: cache invalidation. In a highly dynamic environment a cache silently serves last week’s LOIN and nobody notices until a coordination review.

Atelier: For an office adopting AI this year, the split between frozen and live context is a governance decision, not a plumbing detail — whoever owns the cache owns which version of the norm the model quotes. Your Monday move: audit one AI workflow and tag every document it feeds the model as either FROZEN (norm text, schema, BEP) or LIVE (this project’s changing files), then cache only the FROZEN set with an explicit expiry you chose on purpose.

Hack: Hash the frozen half of your prompt and let the cache answer before the model does. The invariant that makes this safe is that the stored value is a pure function of a stable key — so build the key from the context, never from the free-text question, and set the TTL yourself rather than trusting a default. Anthropic’s own users learned that lesson the expensive way: a GitHub-documented server-side regression from a one-hour to a five-minute cache TTL around 6–8 March 2026 drove a measured $949.08 overpayment on Sonnet across roughly 120,000 calls. Pick the expiry that matches how often your norm actually changes.

import redis, hashlib
r = redis.Redis()
key = "cag:" + hashlib.sha256(frozen_context.encode()).hexdigest()
if not (hit := r.get(key)):
    r.setex(key, 3600, hit := retrieve(frozen_context))  # 1h TTL, chosen not defaulted

The line that matters is the third: the key is the SHA-256 of the frozen context, so an identical BEP maps to an identical cell every time, and a changed BEP maps to a new one automatically. That is the whole geometry — a deterministic partition of your inputs into cells you have answered and cells you have not.

Treat the invalidation rule as the deliverable, not the cache. A cached answer whose key you cannot reconstruct is exactly the parametric geometry with no derivation: a beautiful result you cannot defend in review. Write down, next to each cached corpus, the one condition under which it must expire — and put that rule under version control today, before the corpus grows large enough that nobody remembers why an answer is stale.

Source: developer.ibm.com