Every office in Zurich now feeds files to a language model and gets quietly surprised by what comes back. It miscounts a schedule. It mangles a 22-character GlobalId into something almost right. It swears a long quantity take-off is complete when it silently dropped the last third. The folklore explanation — “the AI is bad at numbers” — is unfalsifiable, and therefore useless on a Monday. There is a real mechanism underneath, and once you have it, the folklore collapses into engineering.

Here it is, and it is the whole lesson: a model does not read characters. It reads tokens — chunks minted by an algorithm called byte-pair encoding, chosen for compression, not for meaning. Everything else in this piece is a consequence of that one fact.

Where the token comes from

The clearest free account of this is Andrej Karpathy’s “Deep Dive into LLMs like ChatGPT” (5 February 2025, three and a half hours, over nine million views). Karpathy — an OpenAI founding member, later Tesla’s AI director, now at Eureka Labs — walks the mechanism from the bits up. Packaging that lineage into three and a half free hours that nine million people watched is itself real engineering; the framing below is his, the IFC numbers are ours.

Text enters a network as a one-dimensional sequence from a finite vocabulary. Raw bytes give you 256 symbols and very long sequences. Byte-pair encoding shortens them: find the most frequent adjacent pair, mint a new symbol for it, rewrite every occurrence, repeat. Each merge trades vocabulary size for sequence length — and Karpathy is blunt that sequence length is “a very finite and precious resource.” GPT-4’s vocabulary is 100,277 symbols; its successor encoding, o200k_base, carries 200,019. The training corpus in the lecture, FineWeb, is 15 trillion tokens decanted from 96 Common Crawl snapshots — a pipeline with thresholds in it, not a substance.

Why IFC is close to the worst case

English prose compresses beautifully because BPE has seen “the building” a billion times. Coordinate digits and GlobalIds are near-random; there are no frequent pairs to merge. I hand-wrote five IFC STEP lines and ran them through tiktoken 0.14.0 (o200k_base): 368 characters became 158 tokens — 2.33 characters per token, a mean of 31.6 tokens per line. One sentence of ordinary office prose came to 5.10 characters per token. The format the profession standardised on for interoperability is roughly 2.2× more expensive to show a model than plain English. Nobody told the architects.

Scale it: 10,000 IFC lines is about 316,000 tokens; 100,000 lines is 3.16 million. A 1M-token context window — currently the window for Claude Opus 5, Sonnet 5 and Fable 5.1 — therefore holds on the order of 30,000 IFC lines, a small building and no more. (Do not port that ratio across vendors: Anthropic’s own docs put 1M tokens at roughly 2.5M Unicode characters on their tokenizer, not OpenAI’s.) That single number is the whole “why does it choke on my model” answer.

←TODAY: A 1M-token window holds about 30,000 IFC lines — one small building, then it truncates in silence.
→3012: The model reads the building directly; the file becomes a fossil we translate for it.
Fulcrum: The token is the unit of cost today and the unit of forgetting tomorrow — count it while it still counts.

Why it miscounts, and why it cannot spell

There is a fixed, small budget of computation per token — it does not grow because your question got harder. A model forced to answer immediately crams the problem into one forward pass and justifies the guess afterward. Ask it to count and it estimates a crowd from one glance. Karpathy’s own phrase is that models need tokens to think: laying out intermediate steps spreads the same computation across many cheap tokens. Spelling fails for the same reason plus tokenization — the model never sees the letters, only the chunks. (The floating is 9.11 bigger than 9.9 failure is a genuine anecdote about unevenness, not a mechanism — treat it as such.)

The deeper distinction, and the most portable one for a Büro: the weights are a vague recollection of something read long ago; the context window is working memory, directly in front of the network. A hallucination is not lying — it is the model imitating the confident shape of every “who is X” answer when it has no recollection to draw on. Paste the text and it stops recollecting and starts reading.

The Tool

The Tool: openai/tiktoken — MIT-licensed, 19,145 stars, actively maintained. It is the encoder OpenAI ships in production, installs in one command, and answers the one question your office actually has: how much of my file fits, and what does the model see? Karpathy’s own karpathy/minbpe (MIT, 10,702 stars) is the second afternoon — dormant-but-stable, four readable files where you watch merges being minted rather than just counted.

Setup:

python -m venv .venv
.venv\Scripts\python.exe -m pip install tiktoken   # macOS/Linux: .venv/bin/python -m pip install tiktoken

# proof of life, straight from the tiktoken README:
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

First steps:

1. Export any storey from Archicad to IFC — call it model.ifc, sitting next to your script.
2. Run the Hack block below. You will get a character count, a token count, and a split of one class name.
3. Change the encoding line to tiktoken.encoding_for_model("gpt-4") (that maps to cl100k_base) and re-run: the ratio barely moves. Say which encoding you used every single time you quote a number.
4. Divide your token count by your context window. If it is over one, you now know it will truncate — before a wrong answer tells you.

Atelier: For a Zurich studio wiring a model into its Archicad automation, the shift is smaller than the hype and larger than a gimmick: treat the context window as working memory you fill on purpose, not a search box. The one move for Monday — stop asking the model what SIA 380/1 says, and paste the clause. Recollection becomes reading, and the answer stops being confident folklore. The same discipline is why a grounded Archicad copilot like PAZGPT beats a raw chat window: it feeds the tool the working memory instead of trusting the recollection.

Hack: Weigh your own IFC before you paste it anywhere. Read the file with errors="ignore" so a stray byte never stops you, print characters against tokens, then split one class name and look hard at the result.

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")   # -> o200k_base
ifc = open("model.ifc", encoding="utf-8", errors="ignore").read()
print(len(ifc), "chars ->", len(enc.encode(ifc)), "tokens")
print([enc.decode([t]) for t in enc.encode("IFCWALLSTANDARDCASE")])

That last line prints ['IF', 'CW', 'ALL', 'STANDARD', 'CASE'] — and that is the whole emotional beat: the word wall is not in there at all. The model never saw a wall — it saw five chunks chosen to compress Common Crawl. When you need a count, a spelling, a string slice, or arithmetic over a schedule, ask for that program instead of an answer. The interpreter counts; the model copy-pastes.

From where my desk sits, looking back, the offices that aged badly were the ones that outsourced memory to a vendor’s ever-growing context window and forgot it was rented by the token. Context length is something someone sells you; a GlobalId you can still read in a text editor is something you own. Meter the token now, and you keep the choice later about which future you build toward.

Learn-it:

• The tool: github.com/openai/tiktoken — the production encoder, README and all.
• The lecture: Karpathy — Deep Dive into LLMs like ChatGPT, tokenization from 07:47.
• Zero-install, for the colleague who won’t open a terminal: tiktokenizer.vercel.app — paste an IFC line, watch it split.
• The readable implementation: github.com/karpathy/minbpe — BPE in four files you can actually read.
• See the network itself: bbycroft.net/llm — Brendan Bycroft’s 3D walkthrough of the layers.
• The root standard: buildingSMART — Industry Foundation Classes, the format we just weighed.