Start with the honest admission underneath the whole AI-for-AEC stack: we do not have a reliable instrument for measuring what these systems actually do. On the latest episode of Quanta Magazine’s The Joy of Why, Steven Strogatz sat down with Melanie Mitchell — cognitive scientist and computer scientist at the Santa Fe Institute — who argues that large language models are a form of “alien intelligence”: trained on human language, yet reasoning through non-human mechanisms we lack adequate methods to probe. Mitchell first appeared on the show around 2021, before ChatGPT’s arrival in November 2022 reset the field overnight; she returns to lay out six principles for assessing machine cognition as a science rather than a leaderboard — borrowing the experimental methodology psychologists already use on other alien minds: babies, dogs, dolphins.

Read that as a systems diagram and the bottleneck jumps out. Every office wiring an LLM into its CAD, quantity-takeoff or specification workflow is trusting a component whose internal state is unobservable and whose failure mode is silent. That is not a metaphor. It is the exact class of dependency that looks fine in every demo and breaks on the one input nobody rehearsed.

Mitchell reaches back to the early 1900s for the cautionary tale: Clever Hans, the horse that appeared to do arithmetic by tapping its hoof, but was actually reading involuntary cues from its handler. The horse passed the benchmark. It just wasn’t doing the thing the benchmark claimed to measure. When a frontier model recently cracked one of Paul Erdős’s open problems — the unit distance problem, solved by fusing two areas of mathematics that hadn’t been combined before — Strogatz called it something that looked like creativity. Mitchell’s sharper point: the model was “trained on everything,” including all of Strogatz’s own YouTube lectures, so the tidy human distinction between in-distribution and out-of-distribution stops being measurable. You cannot say a system transferred a skill when you cannot say what it hadn’t already seen.

←TODAY: In 2026 we deploy models we cannot audit against benchmarks we cannot trust, and call the gap “performance.” →3012: The offices still standing are the ones that logged where the model was guessing, not just where it was right. Fulcrum: A benchmark score measures the test; only a dependency graph measures the risk.

For the AEC desk this is not abstract. Your building is the out-of-distribution case. The specific site, the specific Wettbewerb brief, the specific structural exception your engineer flagged — that is precisely the input least represented in any training corpus, and precisely where a Clever-Hans model will produce fluent, confident, wrong output. Mike Frank at Stanford made the developmental-psychology version of this argument first: study how a child generalises from little data before assuming a machine trained on everything has generalised at all. That comparative lens is not new — Mitchell notes psychologists have probed non-human minds this way for at least 100 years. PAZ has kept a marker on this thread since the beginning — the field’s own canon, Artificial Intelligence: A Modern Approach, defines intelligence as search, planning and probabilistic reasoning, a far wider frame than “the LLM sounded right.”

Atelier: The Büro adopting AI this quarter faces a governance question dressed as a productivity question — you cannot supervise what you cannot see failing. The Monday move: pick one live task where you already use an LLM (spec drafting, code-compliance lookup, a Grasshopper C# snippet) and add a two-line “distribution check” to the workflow that flags when the query sits far from anything routine, routing those cases to a human before they reach a drawing.

Hack: Score how far your prompt sits from what the model has actually seen. Instead of trusting a single confident answer, measure the query’s distance from your own corpus of known-good, in-distribution prompts — the fraction of embedding dimensions that fall within a couple of standard deviations of your training mean. A low fraction is your Clever-Hans alarm: the model is out past its evidence, and the answer needs eyes.

import numpy as np
def in_distribution(query_vec, corpus_mean, corpus_std, k=2.0):
    z = np.abs(query_vec - corpus_mean) / corpus_std
    return float((z < k).mean())  # share of dims within k-sigma of training

Wire that fraction into a threshold and you have converted an unmeasurable black box into a routed one: high-confidence-in-distribution runs unattended, everything else goes to a person. That is the whole discipline — not trusting the machine less, but knowing which of its answers you are actually trusting. Draw that routing line into your workflow before the model draws it for you.

Source: quantamagazine.org