Every reinforcement-learning agent learns the same way: it does something that might be wrong, and finds out. That is not a bug in the method — it is the method. Which means every RL system you will ever meet is quietly answering one accounting question nobody puts in the slide deck: who absorbs the cost of the mistake the agent has to make in order to learn anything.

Richard Sutton and Andrew Barto work both ends of that axis in their own book, Reinforcement Learning: An Introduction (the second-edition draft is free at incompleteideas.net; MIT Press printed the finished one in 2018). At the cheap end sits the 10-armed bandit on page 48 — 2000 toy problems, and a wrong pull costs you exactly one sub-optimal payout on one of them. That is why every RL course starts there: the tuition is measured in pennies. The purely greedy learner lands about 1.0 reward per step against a possible 1.55 and finds the best arm only a third of the time; the ε=0.1 explorer does better but pays a standing 9% tax forever for keeping the door open.

The expensive end is Example 6.6, Cliff Walking (p172), and it is the whole story in four numbers. A gridworld, reward −1 on every step, and one strip marked The Cliff: step in and you eat −100 and get teleported back to Start. Run Sarsa and Q-learning on the identical grid, same ε-greedy 10% wobble. Q-learning learns the values of the optimal policy — the path hugging the edge — and then, because of that 10% wobble, keeps falling off it. Sarsa learns the longer, boring route along the top and quietly outperforms it in practice. Sutton and Barto say it plainly: the algorithm that learns the optimal policy performs worse online than the one that doesn’t.

←TODAY: A 2015 textbook draft, read in 2026 against a live BOPTEST repo, still describes exactly why RL cools data centres but not a Rohbau. →3012: The controllers that survive are the ones whose reward a 25-year-old can still read and audit. Fulcrum: The cliff and the elevator are the same lesson — you only see it once you stop asking whether the agent is smart and start asking what its wrong move costs, and who pays.

The difference is one symbol, and it changes who gets hurt

Here is the mechanism, because it is smaller than the hype. Sarsa (introduced in section 6.4) bootstraps from the action it will actually take next — the update is literally the quintuple that gives it its name — so the cliff’s −100 leaks backwards into the squares beside the edge, and the agent learns to give them a wide berth. Q-learning (Watkins, 1989) bootstraps from the best action it could take, max over the next state. It evaluates a greedy policy it is not following, so it never prices in its own 10% chance of a clumsy step. On-policy learns the driver it has; off-policy learns the driver it wishes it had.

The Tool: To see this on a real building instead of a grid, the project worth an afternoon is BOPTEST — the Building Optimization Testing framework from David Blum, Javier Arroyo, Michael Wetter, Lieve Helsen and colleagues at LBNL, KU Leuven, PNNL, NREL and SINTEF, published in the Journal of Building Performance Simulation in 2021. It is the cliff with a safety net: containerised Modelica building emulators sitting behind one common HTTP API, so you can throw a controller at a realistic thermal plant without touching a real boiler. The companion repo ibpsa/project1-boptest-gym wraps it in a Gym-style interface so an RL agent can talk to it directly.

Setup:

git clone https://github.com/ibpsa/project1-boptest
cd project1-boptest
# needs Docker + docker compose
TESTCASE=bestest_hydronic_heat_pump docker compose up -d
# in another shell: prove it answers
curl http://127.0.0.1:80/name
# -> {"name":"bestest_hydronic_heat_pump"}

First steps:

1. Bring one test case up as above and hit /name — if it answers, the emulator is live and you have a building you cannot break.
2. POST /advance with an empty control input to let the built-in baseline controller run, and read the returned measurements — that is your do-nothing reference.
3. Clone ibpsa/project1-boptest-gym, point its url at http://127.0.0.1:80, and run the example agent for a short horizon. You now have Cliff Walking with radiators: the agent explores, the KPIs (energy, thermal discomfort) tell you what each wrong move cost.
4. Change the reward from energy-only to energy-plus-discomfort and watch the behaviour flip. Reward it on kWh alone and it discovers that an empty, freezing building is extremely efficient.

Atelier: The transferable move for a Swiss Büro is not “adopt RL”. It is the one-page memo you write before anyone proposes a learning controller for a building: (1) state the reward in a single sentence, (2) ask whether the building can measure that reward from the sensors it already has, and (3) name what a wrong action costs and who absorbs it. Crites and Barto’s elevator study is the warning — their reward was the sum of squared passenger waiting times, and they admit on page 303 that a real building never knows how many people are waiting, only how long since the button was pressed. The building cannot measure its own reward. Your Monday move: take the next BMS optimisation someone is excited about and fill in those three lines. If line 2 is “no”, you have your answer before you spend a franc.

Hack: Watch the one symbol that turns a cautious controller into a reckless one. Paste these three lines and read them slowly — the whole Sarsa-versus-Q-learning argument lives in the second token of the target.

target = r + gamma * Q[s2, a2]      # Sarsa: bootstrap from the step you'll ACTUALLY take
target = r + gamma * Q[s2].max()    # Q-learning: bootstrap from the step you WISH you'd take
Q[s, a] += alpha * (target - Q[s, a])   # same update rule, opposite temperament

Swap Q[s2, a2] for Q[s2].max() and you have quietly told the agent to stop pricing in its own mistakes. On a gridworld that is a falling stick figure. On a chiller, that is a comfort complaint with a name attached.

Why it runs in the data centre and not on your site

DeepMind told this lesson twice, and the gap between the tellings is the point. In July 2016 it was a recommendation system — 40% less cooling energy, 15% lower PUE, and a human operator decided whether to apply each suggestion. In August 2018, “Safety-first AI for autonomous data centre cooling” took direct control for around 30% average savings. What changed was not a smarter learner. It was a fence: an action only every five minutes, eight distinct safety mechanisms, uncertainty estimation that throws away low-confidence moves, and — the sentence to remember — operators always in control and able to exit AI mode at any time. That is cliff-walking engineering: build the long safe path in on purpose.

RL is deployed where the plant is continuous, densely instrumented, cheap to simulate, and a wrong action costs a degree of comfort you can claw back next interval. That describes a data centre and an elevator bank. It does not describe a Rohbau, where the exploratory action costs a person, the state is unobservable, the episode runs exactly once, and there is no reset button. The 2024 Sinergym evaluation by Manjavacas and colleagues (arXiv:2401.05737) is the honest counterweight even inside the friendly cases: SAC and TD3 look good in-distribution, but generalising to conditions they were never trained on is still an open problem.

Learn-it:

• The book, free and complete: Sutton & Barto, Reinforcement Learning: An Introduction — read Example 6.6 first.
• The framework: ibpsa/project1-boptest and its RL wrapper ibpsa/project1-boptest-gym.
• The paper behind it: Blum et al., BOPTEST, J. Building Performance Simulation 2021.
• The honest counterweight: Manjavacas et al., DRL for HVAC control, arXiv:2401.05737.
• If the neural-net half is new: Nielsen, Neural Networks and Deep Learning — intuition-first, free.

Clone BOPTEST, run the do-nothing baseline once, and only then let an agent explore. The order matters: you want to know what a wrong move costs on a building you cannot break, long before you write that same line against one you can.