MIRN

1. The measurement problem

“How much did the robot disturb this crowd?” is a causal question wearing the clothes of a descriptive one. The quantity anybody actually wants is a difference between two worlds: the room with the robot in it, and the same room, the same people, the same day, without. Only one of those worlds gets filmed.

So the field measures the next best thing. Forecast where each pedestrian was going, compare against where they went, and attribute the gap to the robot. It is reasonable, it is standard, and MIRN exists because it is also very difficult to check — there is nothing to check it against.

2. Invent the room, and the answer comes free

The move is to give up realism to buy ground truth. Because the crowd is a model I wrote, the same room can be run twice: identical people, identical starting positions, identical random wobble, once with a robot and once without. Everything is held fixed except the treatment, so the gap between a person’s two paths is the robot’s effect on them. Nothing is predicted and nothing is estimated.

That construction is not novel and MIRN claims no credit for it. What is unusual is the use it is put to: the paired difference is not the result, it is the answer key. The question on screen is what a method that cannot see the second run reports against it.

3. What the console found

At the settings a first-time reader lands on, the crowd really moved 0.352 m. The forecast method — the one you would be forced to use in a corridor — reported 0.285 m. Then the same method was pointed at a run where the crowd cannot see the robot at all, so the true effect is exactly zero, and it reported 0.334 m.

It reports more disturbance for a robot that did nothing than for one that did something.

Sweeping the forecast horizon makes the shape clearer. The reported number climbs from 0.005 m to 0.367 m while the true effect sits still at 0.317 m, and at every horizon on that sweep what the method reports on the real run is between 0.90× and 1.23× what it reports on a run with no effect at all. The failure is not that it inflates the answer. It is that the two are unrelated — a ruler reading about the same whatever happened is not reporting on what happened, so neither its alarms nor its silences carry information.

4. No number without its zero

The single design rule the whole console is built around: every number on screen sits one interaction away from what it assumes and from what it would read if the answer were nothing. The zero reference is rendered beside the value, measured at the same settings, never collapsed behind a toggle and never quoted from a neighbouring cell. Raw metres never appear alone — each is shown against a body-scale anchor, a zero-effect floor, or the ordinary run-to-run band.

This is enforced rather than intended. The readout component takes the reading and its zero as a pair, so a tile with a bare number does not compile, and a test renders every column in the catalogue and fails on a hard-coded metre value anywhere in the copy. A number whose caption outlived the settings that produced it is the exact error the site is arguing against.

1. State the assumption. Every readout carries, in plain English, the thing it takes for granted.
2. Print the zero. What this same ruler reports on a world where the true effect is nothing, measured now rather than remembered.
3. Give it scale. A body width, a detection floor, or the ordinary gap between two runs of the same room.
4. Refuse the conclusion. The crowd is invented, so the page says what would have to be measured instead of implying it was.

5. Five surfaces, no reading order

The console is the landing page: a settings panel, a scrubable arena, seven headline readouts with a picker for the rest, a sweep curve, and a ledger. Four more pages hang off it, none of them a step in a sequence — every one is reachable from every other at any time, and a test reads the page list from the build config and fails if any page fails to link the others.

6. The referee drill and the method card

The drill puts the reader in the corridor’s actual position. Eight rooms, each run twice under the paired construction so the honest answer is known, but the second run is withheld and the only figure on the card is one a real corridor could hand you. Five of the eight are built so that figure points the wrong way; three are built so it happens to point the right way, because a drill that always misled would teach that the method is backwards rather than what it is, which is uncorrelated. Which readings are even eligible to appear on a withheld card is proved per reading — the withheld run is swapped for a decoy from an unrelated crowd, and any column that notices the swap fails.

The method card scores a method instead of a reader. Answer five multiple-choice questions about how your metric is computed and a closed table maps it onto one of four families the bench implements; or hand it the function itself, which runs in a Worker with no DOM, no network and no storage, receiving only the arm with the robot in it so the control run cannot reach it through any parameter. You leave with a false-positive rate: how often a ruler of that shape fires on rooms where the robot moved nobody. It runs every family across one shared set of rooms and then refuses to rank them, because the crowd is invented and a leaderboard would be a claim about corridors.

7. Two implementations, one oracle

Every formula the browser quotes has a reference implementation in Python. A CLI writes the oracle’s answers to committed fixtures and the TypeScript has to reproduce them, with tolerances declared inside the fixture by the oracle author — so loosening one is a visible diff in a committed file rather than an invisible edit in a test. Changing a formula is a two-file commit minimum; changing one side alone is a red test, not a judgement call.

Three floating-point traps are pinned by tests rather than by comments, all learned the hard way. NumPy sums pairwise, so the browser reimplements that instead of folding left. Its quantile convention is one of nine and picking another silently shifts every detection floor on the site. And Math.hypot is banned inside the measurement directory because V8’s is more accurate than NumPy’s naive form and therefore disagrees in the last bits — a test greps the directory’s own source, because a comment is not a build error.

Determinism is structural rather than disciplinary. Both arms of a pair are driven by one addressable noise tape — one tape, two consumers, nothing to keep in step — so determinism tests compare bytes rather than approximations. The sweep runs in a Worker that returns numbers and never trajectories, so selecting a ledger row re-simulates that run from its key, and a test asserts the rebuild is identical bit for bit on both arms.

8. The gate that blocks everything else

CausalAgents (Roelofs et al.) found that standard trajectory forecasters move their error by 25–38% when agents that provably could not have influenced a scene are deleted from it. A ruler that responds to bystanders who did nothing is measuring the scene, not the robot. The paired estimator computes each person’s divergence from their own two paths and nothing else, so it structurally cannot do that, and a placebo test in each language is what keeps it that way. If either is red, nothing on the page can be trusted and no other work proceeds.

That gate deliberately runs on an analytic fixture rather than on the crowd. Delete a bystander from the dynamical sim and the estimate moves by up to 37% — correctly, because removing anyone rewires the interaction chain and the robot then takes a different path. Pointing the gate at the crowd would make it permanently red for a good reason, and the natural repair is to loosen the tolerance until it passes, which leaves a test that asserts nothing.

9. Calibration, and the number that looked measured

Both crowds are fitted to nothing at all, which is the largest thing standing between this bench and an instrument. So one page reads a recording of real people walking — in the browser, going nowhere, with every identifier discarded before anything is measured — and asks how far the invented crowd’s distribution of walking speeds sits from theirs.

The interesting part turned out to be which settings a recording actually decides. A parameter counts as pinned only if moving it across its whole range changes the fit by more than the invented crowd differs from itself. Feed the page a recording written out of the simulator at a known pace and it recovers that pace exactly and reports it pinned. It reports the wobble as not pinned, because it isn’t — recordings made at very different wobbles both fit best near nought, and the grid’s winner there is an artefact of where the grid was finest. The first version reported that winner as a fitted parameter: a number that looked measured and was not, which is the exact failure the rest of the project argues against.

That page shows no disturbance number and cannot compute one. Not restraint — arithmetic. A corridor cannot be filmed twice, so real trajectories have exactly one arm and the quantity the site exists to measure is simply not in them. A figure claiming otherwise would be the same confound wearing real data, which makes it worse rather than better. The two kinds of number are kept apart by living on different pages behind different workers, asserted over the page’s source rather than its output.

Why it matters

The transferable claim is not about pedestrians. It is that a paired comparison — the same situation, run twice, differing in one thing — settles a question a single observation cannot, and settles it by construction rather than by a cleverer estimate. Where the counterfactual is unavailable, a simulator is one of the few places you can hold up a ruler and check it against an answer you already know.

And what it is not. The crowd is a model I wrote. A toy crowd is exactly the environment that would make the underlying research question circular — decide how people respond to robots, then measure how people respond to robots. So the bench can demonstrate that a measurement can be confounded and why the paired design removes the confound. It cannot say how large that error is for real robots, which method wins in the field, or that any published result is wrong. Nothing here is a research finding, and the console says so above every number it prints.