Every neural network you have heard of this year learns with one trick. It runs the chain rule backwards through a graph of numbers and asks each number one question: if I nudged you a little, how much would the error move? That trick is called backpropagation. You can build it in an afternoon in plain Python, with nothing to install, then point it at a facade and ask how far it leans.

The idea is older than most office servers. Seppo Linnainmaa described reverse-mode automatic differentiation in his 1970 master’s thesis in Helsinki, published in BIT in 1976. Rumelhart et al. made it the engine of neural networks in “Learning representations by back-propagating errors”, Nature 323, October 1986. What is new is how small and readable the teaching version has become.

The Tool: micrograd, by Andrej Karpathy, on GitHub since 13 April 2020 under the MIT licence. According to its README, it is “a tiny Autograd engine” that “implements backpropagation (reverse-mode autodiff) over a dynamically built DAG”, and “potentially useful for educational purposes”. The engine is 94 lines, and its tests check every gradient against PyTorch “as a reference”. Writing a teaching engine this clear is real engineering: it turns a 1986 Nature paper into something a designer can read on one screen. Below, we rebuild micrograd’s core idea in 51 lines and put it to work on a wall.

←TODAY: In 2026 the loop that trains a GPT fits in a file you can read over one coffee.
→3012: In Zurich-3012 the plain-text script still opens. The sealed black boxes of its era did not.
Fulcrum: A model you can read is a model you can check, and repair.

Setup: no install, only Python 3. The editor ran both files on Python 3.10 and on Python 3.9, the version inside Rhino 8’s ScriptEditor, with identical output.

mkdir tiny-grad && cd tiny-grad
python --version
# save tiny_grad.py and lean.py into this folder, then:
python lean.py
# optional: read the original (engine.py is 94 lines)
git clone https://github.com/karpathy/micrograd

First steps:

1. Save the engine below as tiny_grad.py.
2. In the same folder, run this check, which is small enough to verify by hand:

   from tiny_grad import Value
   a, b = Value(2.0), Value(3.0)
   c = a * b + a          # c = 2*3 + 2 = 8
   c.backward()
   print(c.data, a.grad, b.grad)   # 8.0 4.0 2.0

   It prints 8.0 4.0 2.0.
3. Why is a.grad 4 and not 3? The variable a is used twice, in a*b and in +a, so its two contributions add up: 3 from the multiply plus 1 from the add. That explains the most important symbol in the engine.

The code, explained: first the engine, then the wall.

# tiny_grad.py - a small autograd engine, micrograd-style
class Value:
    def __init__(self, data, parents=(), op=""):
        self.data = data          # the number itself
        self.grad = 0.0           # d(loss)/d(this number), filled by backward()
        self._parents = parents   # the Values this one was made from
        self._backward = lambda: None
        self.op = op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad += out.grad          # d(a+b)/da = 1
            other.grad += out.grad         # d(a+b)/db = 1
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad += other.data * out.grad   # d(a*b)/da = b
            other.grad += self.data * out.grad   # d(a*b)/db = a
        out._backward = _backward
        return out

    def __pow__(self, n):                        # n is a plain number
        out = Value(self.data ** n, (self,), f"**{n}")
        def _backward():
            self.grad += n * self.data ** (n - 1) * out.grad
        out._backward = _backward
        return out

    def __neg__(self): return self * -1
    def __sub__(self, other): return self + (-other)
    def __radd__(self, other): return self + other
    def __rmul__(self, other): return self * other

    def backward(self):
        order, seen = [], set()
        def visit(v):                  # topological sort: parents before children
            if v not in seen:
                seen.add(v)
                for p in v._parents:
                    visit(p)
                order.append(v)
        visit(self)
        self.grad = 1.0                # d(loss)/d(loss) = 1
        for v in reversed(order):      # walk from the loss back to the inputs
            v._backward()

Lines 2–8 — the number that remembers: a Value is a class, a template for objects that hold data and know how to behave. This one wraps a plain float (data), adds a grad slot that starts at zero, and records in _parents which Values it was made from. That is how the graph, numbers connected by the operations that produced them, builds itself while you do ordinary arithmetic.

Lines 10–26 — local derivatives: each operation knows only its own slope. For a + b, the derivative with respect to each input is 1, so the gradient passes straight through. For a * b, the derivative with respect to a is b, and the other way round. Multiplying by out.grad is the chain rule: the slope of a chain of functions is the product of the slopes along it. Now look at +=. The classic beginner bug is writing =: a number used twice, like a above, then has its second path overwrite the first, and the gradient is silently wrong. If your first-steps check does not print 4.0 for a.grad, look for an =.

Lines 28–38 — power and sugar: __pow__ applies the school rule: the derivative of xⁿ is n·xⁿ⁻¹. The four one-liners let Python’s operators work in both directions. __radd__ matters because sum() starts from the plain number 0, and 0 + Value only works if Value can be added from the right.

Lines 40–51 — the backward walk: a topological sort is an ordering in which every number comes after the numbers it was built from. visit produces it by recursion, a function calling itself on the parents first. The loss gets a gradient of 1.0, because it changes one-for-one with itself, and the loop walks the order in reverse, each node pushing its gradient to its parents. By the time a node runs its _backward, everything built on top of it has already added its share.

Now the AEC task. A wall, six laser readings, one question.

# lean.py - how far does this wall lean? fit offset = a*height + b
from tiny_grad import Value

heights = [0.5, 2.0, 4.0, 6.5, 9.0, 11.5]       # m above ground
offsets = [3.0, 7.0, 10.0, 16.0, 21.0, 26.0]    # mm out of plumb (laser)

a, b = Value(0.0), Value(0.0)                   # lean (mm/m) and base offset (mm)
lr = 0.004                                      # learning rate: step size

for step in range(2001):
    loss = sum(((a * h + b) - o) ** 2 for h, o in zip(heights, offsets)) * (1 / len(heights))
    a.grad, b.grad = 0.0, 0.0                   # forget last step's gradients
    loss.backward()                             # fill a.grad and b.grad
    a.data -= lr * a.grad                       # walk downhill
    b.data -= lr * b.grad
    if step % 500 == 0:
        print(f"step {step:4d}  loss {loss.data:8.3f}  a {a.data:5.2f} mm/m  b {b.data:5.2f} mm")

print(f"lean: {a.data:.2f} mm per metre, {a.data * 11.5:.1f} mm over the 11.5 m wall")

Lines 4–5 — the data: six offsets out of plumb, measured at heights up an 11.5 m wall. These numbers are illustrative, made up for this tutorial, not a real survey.

Lines 7–11 — the model and the loss: we guess a straight line, offset = a·height + b, where a is the lean in mm per metre and b the offset at the base. The loss is the mean squared error: how wrong the current line is, as one number. The learning rate lr is the size of each step downhill.

Lines 12–15 — the four-line training step: zero the gradients, because with += they would otherwise pile up from step to step. The PyTorch “Autograd mechanics” notes describe the same accumulation, which is why PyTorch users call optimizer.zero_grad(). Then backward() fills a.grad and b.grad. A gradient is how much the loss changes if you nudge that number; its sign points uphill, so subtracting it walks downhill. The editor’s real output:

step    0  loss  255.167  a  0.86 mm/m  b  0.11 mm
step  500  loss    0.237  a  2.14 mm/m  b  1.70 mm
step 1000  loss    0.146  a  2.09 mm/m  b  2.09 mm
step 1500  loss    0.139  a  2.08 mm/m  b  2.20 mm
step 2000  loss    0.139  a  2.08 mm/m  b  2.23 mm
lean: 2.08 mm per metre, 23.9 mm over the 11.5 m wall

The cross-check: NumPy’s exact least-squares fit, np.polyfit(heights, offsets, 1), gives a = 2.075 mm/m and b = 2.247 mm. Gradient descent walked to the same answer the closed-form formula gives. That is the point of building it yourself: you can trust the engine because you can check it.

Atelier: picture a 12-person Zürich studio surveying an old facade before a refurbishment, with laser offsets per storey arriving as a spreadsheet nobody fully reads. The loop that trains a GPT is the same loop that fits that lean, and it is short enough to live in the project folder. Monday move: save tiny_grad.py and lean.py next to the survey data, swap in the real per-storey readings, and agree as a team that no lean figure leaves the office until it matches np.polyfit.

If you work in Grasshopper, you already own an optimiser: Galapagos, David Rutten’s evolutionary solver. Rutten wrote in 2011 that the solver’s job is “to find the highest peak in this landscape”, and Galapagos does it by breeding and sampling candidate answers, with no slope needed. Gradient descent is a different tool, not a better one: it reads the slope where it stands. The trade-off is plain: gradient descent needs a differentiable model and a sane step size, while an evolutionary solver buys its freedom with many more evaluations. Slope-reading is why differentiable form-finding exists, from Rafael Pastrana’s JAX FDM (“Crafted with care at Princeton University”) to Pastrana et al.’s 2023 Computer-Aided Design paper on constrained form-finding with automatic differentiation, which has an ETH Zurich co-author.

Hack: Push the step size until the descent overshoots, and predict it before you run. Change one line in lean.py:

lr = 0.03                                       # was 0.004

Expect the steps to be too big. Real output:

step    0  loss  255.167  a  6.47 mm/m  b  0.83 mm
step    1  loss  822.216  a -5.18 mm/m  b -0.56 mm
step    2  loss 2656.461  a 15.77 mm/m  b  2.04 mm
step    3  loss 8589.489  a -21.93 mm/m  b -2.53 mm

Then Python stops with OverflowError: (34, 'Result too large'). Each step jumps over the valley and lands higher on the far side, so a flips sign every time. The wall did not move. The maths did. For the record, the editor measured lr 0.01 and 0.02 converging to 2.08 mm/m, while lr 0.001 is too timid and still sits at 2.14 mm/m after 2,000 steps.

From where I sit, the files that survived were the plain-text ones. Two small .py files will still open long after today’s licences have lapsed.

Learn-it:

• The original engine: karpathy/micrograd on GitHub
• The lecture: The spelled-out intro to neural networks and backpropagation: building micrograd, part of Neural Networks: Zero to Hero
• The industrial version: A Gentle Introduction to torch.autograd
• The root concept: Backpropagation (Wikipedia) and Chain rule (Wikipedia)
• PAZ note: the engine needs no packages and runs on Python 3.9, the same version Rhino 8’s ScriptEditor uses, so it makes a good first file to open there.

Save the two files today, break the learning rate once on purpose, and keep the folder.