If you only learn one piece of machine-learning mathematics this decade, make it attention. Not because it is fashionable — because it is the operator underneath every tool now arriving on an architect’s desk, from the LLM that drafts your competition narrative to the scan-to-BIM pipeline that turns a phone walk-around into an as-built. Understand attention once, and the whole 2026 AI landscape stops being a wall of acronyms and becomes a single idea, repeated.

←TODAY: In June 2026 the same fifteen-page architecture from 2017 still underwrites GPT-5, Claude, AlphaFold, and the Point Transformer your scan-to-BIM vendor is quietly shipping. →3012: By the Zurich-3012 horizon, “attention over tokens” is as basic to a designer’s literacy as the dot product — taught in first year, not chased in a course. Fulcrum: The reason one operator could eat an entire discipline is that it replaced memory with parallel lookup — and that trade is visible only if you can see both where it came from and where it is going.

What it is: Attention is a way for a model to let every piece of an input look at every other piece and decide, on the spot, what is relevant. Each input token x is projected by three learned matrices into a Query Q (“what am I looking for?”), a Key K (“what do I offer?”) and a Value V (“what do I carry?”). The output for a token is a weighted sum of all the Values, where each weight is the softmax-normalised dot product of that token’s Query with another token’s Key, scaled by √d so the gradients stay well-behaved. That is the whole machine. Every token sees every other token in one matrix multiplication — no recurrence, no scanning left to right.

Why it works: Older sequence models compressed a whole sentence into one fixed context vector and then tried to decode from it — a memory bottleneck that lost information the moment the input got long. Attention deletes the bottleneck: instead of remembering, the model looks things up, in parallel, every layer. The cost is honest — O(n²·d), because n tokens each attend to n others — which is exactly why frontier long-context models pour engineering into sparse, linear, and flash-attention variants. The operator never changes; only the bookkeeping around it scales. Three concrete anchors make this real: positional encodings (a small sin/cos trick) hand the otherwise order-blind operator a sense of sequence; Graph Attention Networks let a finite-element node listen to its mechanically-relevant neighbours rather than just its topological ones; and AlphaFold 2’s Evoformer runs attention over residue pairs to recover 3D protein geometry to near-experimental accuracy — a fifty-year biology problem closed by reweighting.

Origins: Attention began as a translation fix. In 2014 Bahdanau, Cho and Bengio noticed that squeezing a source sentence into a single vector lost too much, so they let the decoder peek back at every encoder state through learned weights; “soft alignment” became standard in neural machine translation inside eighteen months. The reorganisation came on 12 June 2017 — nine years ago this week — when Vaswani and seven co-authors at Google Brain posted Attention Is All You Need to arXiv, deleting recurrence and convolution and keeping only stacked self-attention. Within five years that one diagram underwrote BERT (2018, the largest Google Search jump in a decade), GPT-3 (2020), the Vision Transformer (2020, which cut an image into 16×16 patches and proved convolutions were no longer mandatory), Stable Diffusion (2022, cross-attention coupling text into image), and Point Transformer V3 (2024, self-attention over LiDAR clouds). PAZ’s own concept panels — Attention — Historia and En Ingeniería — keep this lineage on the shelf precisely so we do not re-explain it every time the news cites it.

In practice: A Swiss studio reaches for attention whenever the input is a set of irregular, variable-length tokens whose relationships matter more than their order: a LiDAR point cloud for scan-to-BIM, accelerometer streams for structural-health monitoring, an IFC element graph for clash detection, an energy-demand forecast where weekday and weekend tokens reweight each other. The AlphaFold template — attention predicting geometry from relationships — is already being copied for façade thermal behaviour and daylight redistribution across irregular floor plates. The honest trade-off worth stating plainly: attention is a relevance engine, not a truth engine — as recent reporting on long-context failure modes notes, its effectiveness degrades as sequences grow, so a model that “pays attention” to your whole BEP can still confidently weight the wrong clause. Read the output; do not deploy it.

Atelier: In our atelier we teach attention the way we teach a hanging chain — by building the smallest honest version and watching it behave. Eight rooms become eight tokens; one query lights up the studios. The point is not to ship a model but to feel, with your own hands, the move every frontier system makes billions of times a second.

Hack: This Hack teaches you to run one real attention head over a floor plan and watch which rooms a query lights up. The domain is AI/ML; the medium is runnable code. Eight rooms, four features each, the canonical QKV operation — the same operator an LLM runs, only the tokens change.

import torch, torch.nn.functional as F

# 8 rooms x 4 features: [area_m2, daylight_h, adjacency_to_core, prog_tag]
rooms = torch.tensor([
    [42,6.5,0.2,0],[38,6.8,0.3,0],[18,2,0.9,1],[12,0,1,2],
    [22,4,0.5,1],[55,7.2,0.4,0],[16,1.5,1,2],[30,3,0.6,3]],
    dtype=torch.float32)
d = 4; torch.manual_seed(7)
W_q, W_k, W_v = (torch.randn(d, d) for _ in range(3))
Q, K, V = rooms @ W_q, rooms @ W_k, rooms @ W_v
attn = F.softmax((Q @ K.T) / d**0.5, dim=-1)        # 8x8 attention
q = torch.tensor([45.,7.,0.2,0.]) @ W_q             # "quiet, south, studio"
print(F.softmax((q @ K.T) / d**0.5, dim=-1).round(decimals=2))  # mass on S1/S2/S3

Bonus pass: replace that final softmax with a masked variant whose -inf entries forbid attention across fire-compartment boundaries. It is the exact trick a transformer uses for causal masking in language — repurposed here as a code-compliance constraint baked into the geometry.

Move: Paste the snippet into a notebook this week, swap the eight rooms for your current project’s program, and read which tokens your query actually weights. That ten-minute exercise will teach you more about how your AI tools “think” than any cheatsheet download — and it is the first move we make in every PAZ atelier session on machine learning for AEC.