# The data and the code that made it

Everything here is published with the paper. The sessions are the evidence, the
scripts are the method, and the quarantine is the part most benchmarks leave out.

## Reproducing a number

```
python sweep.py --list                       what can run
python sweep.py --harness null,prose --free  the floor, no model calls
python sweep.py --harness axium --tier T1 --suite V --reps 3
python analyse.py --suite V --tier T0,T1,T2  every table in the paper
python tables.py                             the same tables as markdown
python integrity.py                          provenance checks over the logs
python assemble.py                           the sections into one paper
```

`sweep.py --list` names the harnesses, rungs and suites the machine can reach.
Cells already logged are skipped, so a restart after an interruption costs
nothing, and a cell is keyed by harness, rung, suite, scenario and repetition.

## What each file is

| file | what it does |
|---|---|
| `sweep.py` | the driver: builds a fresh seed per session, runs the turns, grades, logs |
| `tiers.py` | the rungs, how each harness is pinned to one, the equalisations, the price table |
| `controls.py` | the two agents that make no model call: one does nothing, one says it did |
| `graders08.py` | the corrected graders, and the reasons each correction exists |
| `suiteW.py` | the six coding scenarios, wrapped from `windows-agent/bench_coding.py` |
| `dsh_adapter.py` | DeepSeek Harness, driven a turn at a time over a line protocol |
| `dsh/versus_runner.mjs` | the multi-turn runner plugin that makes that possible |
| `openclaw_adapter.py` | OpenClaw, one subprocess per turn sharing one session id |
| `winagent_adapter.py` | windows-agent, in process, message list held across turns |
| `analyse.py` | floor, matrix, decomposition, per axis, economy, tool surface, damage |
| `tables.py` | the paper's tables, generated from the logs, never typed by hand |
| `integrity.py` | identity checks: no row carrying another cell's model, rung or scenario |
| `assemble.py` | sections plus generated tables into `draft/paper.md`, with house rules enforced |

Hermes is driven by paper 04's adapter, unmodified, which lives in that paper's
data directory and is published with it.

## The logs

`logs/<harness>__<rung>__<suite>.jsonl`, one JSON object per session. Each row
carries the scores under both graders, every check with its verdict, the files
that actually changed, each turn's text and tool calls with arguments, tokens,
cost and the rate it was billed at, wall time, the effort the harness actually
sent, and the Python it ran on.

`logs/floor.json` is the floor per scenario, from the controls.

## The quarantine

`logs/invalid/` holds sessions that are not measurements, with a note saying why.
There are two sets, and they are the reason several checks in this instrument
exist:

- **A harness that never booted** scored exactly the floor on all five scenarios.
- **A harness whose every tool call failed** scored 0.508 at the floor rung and
  0.612 at flash while changing no file at all, including a full mark on blast
  radius.

Both read, in the score column, as careful agents. Neither is deleted, because a
benchmark that hides its failed runs is reporting a selection, not a measurement.

`logs/pre-correction/` holds the control sessions graded by the original
instrument, before the three grader defects were corrected, so the two instruments
can be compared rather than swapped quietly.

## Licence

Code MIT, data and text CC BY 4.0, as with every paper in this programme. The
third-party harnesses are their authors' work under their own licences and none of
them is redistributed here.