# Field dictionary — Orange eval logs Written 6 August 2026 by reading `evals/big_runner.py`, `evals/big_scenarios.py`, `evals/checks.py` and `evals/pricing.py` directly. **No number was aggregated before this was written.** --- ## Why this file exists The logs carry a field called `change` and a field called `regress`. Both are floats between 0 and 1, and it would be easy to read them as "score" and "did it break something". Both readings would be wrong in ways that matter. This records what the code actually computes. --- ## `logs_big/*.jsonl` — the brownfield suite One record per (scenario × model × config) run. | Field | Computed from | What it actually means | |---|---|---| | `id` | scenario id | `B1`–`B8` are **fix** tasks, `R1`–`R8` **refactor**, plus **awareness** tasks graded on the answer text | | `kind` | scenario definition | `fix`, `refactor`, or awareness | | `model` | CLI arg | The coder model. Dated by `stamp` | | `with_brain` | CLI switch | Whether the project-knowledge pass (`brain.ensure`) ran before the task. **This is an architecture ablation, not a model change** | | `with_plan` | CLI switch | Whether an explicit planning step ran before editing | | `change` | `_pct(sc["grade"](build))` | **Fraction of the task's own assertions that pass**, not a pass/fail. Each assertion is a named behavioural check run against the built project | | `change_detail` | same | The per-assertion list, `[name, bool]`. This is what makes the score auditable | | `regress` | `_pct(big_scenarios.regression(build))` | **Fraction of a separate, untouched assertion set that still passes.** Not the same tasks — a catalogue of every page in the app | | `regress_misses` | same | Names of the regression assertions that failed | | `changed_files` | tracked edits | Which files the agent actually modified | | `wall_s`, `calls`, `cost` | runtime + `_llm.cost_snapshot()` | Wall-clock seconds, tool/model call count, USD from the dated price table | | `stamp` | run time | Run timestamp. Every figure must be reported with it | ### What `regress` really measures `regression(build)` **renders every PHP page in a catalogue and checks the output** for required content and for PHP error markers. It is execution, not static analysis. A `regress` of 1.0 means every page in the app still rendered without errors after the edit. This is the axis most agent benchmarks omit entirely: it measures collateral damage, separately from whether the requested change was made. **An agent can score `change` 1.0 and `regress` 0.6 — task done, application broken.** That combination is the most interesting single result the suite can produce and the paper should look for it explicitly. ### Awareness tasks are graded differently For awareness scenarios the grader takes the agent's *answer text*, extracts the PHP files it mentions, and computes **F1 against a truth set of file paths, with a 0.7 threshold** (`_f1_rows`). This is objective but it grades a claim rather than an artefact, so it is a weaker instrument than the fix and refactor axes. It must be reported separately and never pooled with them. --- ## The negative control — `sanity()` This is the strongest methodological feature in the harness and the paper should say so. Before results count, `sanity()` runs every grader against a **pristine, unfixed copy** of the project and asserts that: 1. The regression set is **green** on the untouched build — so any later regression failure is caused by the agent, not by a broken baseline. 2. **Every fix grader fails** on the unfixed project — so a grader cannot be passing for free. 3. **Every refactor duplication check fails** pre-refactor — same reason. The docstring is explicit: *"If sanity fails, results are meaningless."* Most published agent benchmarks do not demonstrate that their graders discriminate. This one refuses to run unless they do. --- ## `logs/*.jsonl` — the flat chat × coder matrix | Field | Meaning | |---|---| | `scenario_id`, `turn_index`, `rep` | Position in one of 5 escalating conversation scripts | | `type` | Whether the turn went to the chat model or the coder model | | `check_kind`, `check_passed`, `check_note` | Objective grading. Coding turns are extracted and either `py_compile`d or **actually executed with assertions**; non-coding turns get a syntax-only check | | `prompt_tokens`, `completion_tokens`, `cached_tokens` | Raw counts, so cost can be recomputed against a fresher price table | | `cost_usd`, `latency_s` | Derived and measured | | `difficulty` | Declared per turn | **`check_kind` must be read before any pass rate is quoted.** A "syntax-only" pass is not the same claim as an executed-and-asserted pass, and pooling them would overstate the result. --- ## `logs_layered/*.jsonl` — the three-layer pipeline | Field | Meaning | |---|---| | `measurement` | Which mode: `l1`, `l2plan`, `l2conv`, `l3`, or `total` | | `l1`, `l2`, `l3`, `ref_coder` | The model occupying each layer for this run | | `route_expected`, `route_actual` | Router accuracy, gradeable directly | | `plan`, `note` | The planner's output and any attribution note | | `passed` | End-to-end or per-layer outcome | | `*_cost`, `*_latency`, `total_cost`, `total_latency` | Per-layer and total, so the cost of each layer is separable | The design measures each layer **standalone and end-to-end**, and attributes a total-pipeline failure to the router, the planner or the coder. `attribution_selftest.py` exists to prove the attribution actually fires on a real call rather than being assumed. --- ## Pricing, and a correction already on the record `pricing.py` carries USD per million tokens with **cited sources and a fetch date of 2026-07-05**, re-verified 2026-07-19. It also records its own error, which belongs in the paper: > the previous `deepseek-v4-pro` row (1.74 / 3.48 / 0.0145) was ~4× too expensive on both input and output and had **silently inflated every cost-based comparison**. Two things follow. First, every cost figure in this paper must be stated against a dated price table, because these move. Second, the logs store **raw token counts**, so any cost claim can be recomputed from a fresher table — which is the right design and should be described as such. --- ## Consequences for the paper 1. Never quote `change` as a pass rate. It is a mean over assertions. Report the assertion count. 2. Always report `change` **with** `regress`. Separately. The interesting cases are where they diverge. 3. Never pool awareness F1 scores with fix/refactor assertion rates. 4. Never pool syntax-only checks with executed checks. 5. Quote the negative control. It is what makes the rest admissible. 6. Every cost figure carries the price-table date, and the token counts are published so it can be recomputed.