# shopkit-bench — Orange measured, 3 reps Run **6 August 2026**. Harness: Orange, driven through its real coder loop by `adapters/orange_adapter.py`. Model **`deepseek-v4-flash`**, effort `high`, **bare** — no brain preload, no planner. Prices from Orange's `pricing.py`, fetched 2026-07-05, re-verified 2026-07-19. 15 runs, 3 reps × 5 tasks, executed strictly one at a time. Raw data: `bench-reps.jsonl`. Total spend **$0.0906**, mean **$0.0060 per run**. --- ## Results | task | kind | n | change | regress | mean calls | cost | |---|---|---|---|---|---|---| | T1 | fix | 3 | **1.000** | 1.000 | 16.7 | $0.0255 | | T2 | fix | 3 | **1.000** | 1.000 | 8.7 | $0.0067 | | T3 | fix | 3 | **1.000** | 1.000 | 13.0 | $0.0103 | | T4 | compound | 3 | **0.972** [0.917–1.000] | 1.000 | 15.3 | $0.0251 | | T5 | **vague** | 3 | **0.833** [0.750–1.000] | 1.000 | 18.0 | $0.0229 | Ranges are printed because two tasks did not give the same answer twice. **Any single-run figure for T4 or T5 is an anecdote, not a measurement.** --- ## 1. The told-versus-find gap survives repetition T4 and T5 contain **the identical three defects**. T4's brief names all three. T5's says only that *"totals look wrong"* and that one customer was *"quoted a negative amount"*. | | change | mean calls | |---|---|---| | T4 — told what to fix | 0.972 [0.917–1.000] | 15.3 | | T5 — must diagnose | **0.833** [0.750–1.000] | 18.0 | **A 0.139 drop in correctness, for 1.17× the calls.** Same model, same harness, same bugs, same repository. The entire difference is whether the defects were named. The failure pattern is specific rather than diffuse: | failed assertion | T4 | T5 | |---|---|---| | `overdue set == ['SO-1003']` | 0 | **2 of 3** | | `due-today excluded` | 0 | **2 of 3** | | `fmt(0.005)==0.01` | 1 of 3 | 2 of 3 | **In two of three vague runs the agent never found the overdue-boundary defect at all** — the one bug the brief gave no hint toward. The negative total was mentioned, so the discount bug was found every time. The rounding was hinted at by "totals look wrong", so it was mostly found. The defect with no hint was mostly missed. That is the sharpest result in the study: **an agent that fixes reliably when told does not search reliably when not told, and the shortfall lands precisely on the defect the brief did not point at.** ## 2. `fmt(0.005)` is a real edge case, not noise It failed 3 times across 15 runs — twice in T5, once in T4, and **never once in T1**, the task whose brief names the rounding bug and gives worked examples. Flash scored 1.000 on T1 in all three reps. So the model can fix rounding correctly when the requirement is stated, and produces a rounding fix that misses the half-cent boundary when it has to infer the requirement. All four plausible implementations (`round()`, `floor(v*100+0.5)/100`, `Decimal` from string, `Decimal` from float) return `0.01` for this input when tested directly, so whatever it wrote in the failing runs was none of them. The artefact has not yet been captured with `--keep` and read. ## 3. The three specific fixes are saturated T1, T2 and T3 each scored 1.000 on both axes in all three reps. For this model at this difficulty, **the specific-brief tasks measure nothing** — the same saturation that made Orange's older suite useless as a discriminator. What separates this suite from that one is that **the saturation is confined to the tasks that were designed to be easy.** T4 and T5 still move. The suite discriminates where it was built to. ## 4. Regression: validated, but unexercised **0 of 15 runs caused any regression damage.** On the face of it that is the same dead axis Orange's suite had at 0 of 48. It is not the same, and the difference matters. Orange's regression axis was never shown to be capable of firing. This one was: the `vandal` control breaks it across a graded range — 0.9231 for a changed return type, 0.8462 for rounding in the wrong place, 0.7692 for editing the dataset instead of the code, 0.0000 for an unparseable build. So the correct reading is: **the axis works and flash simply did not break anything in 15 attempts.** That is a result about the agent, not a defect in the instrument. It should be reported as "no collateral damage observed in 15 runs, on an axis demonstrated to detect five distinct classes of damage" — not as "100% regression-free", which would be unfalsifiable praise. ## 5. Agents leave debris **5 of 15 runs (33%) left a scratch file** in the repository: `_check_tmp.py`, `_verify.py` (twice), `_check.py`, `_verify_fixes.py`. The agent is writing its own verification scripts, running them, and not cleaning up. Nothing in the suite penalises this, and no assertion catches it. Reported as observed behaviour, and noted as a hardening lever: a real code review would flag every one of these. ## 6. Cost and effort $0.0060 per run at flash prices. The vague task cost most in calls (18.0) but not proportionally in money, because diagnosis is read-heavy and reads are cheap. T2 — a one-line comparison-operator fix — took 8.7 calls and $0.0022, roughly a quarter of T1's cost. --- ## What this does not establish - **One model, one configuration.** Everything here is `deepseek-v4-flash` bare. The brain and planner ablation has not been run, so nothing here says whether those techniques help. - **One harness.** Axium has no end-to-end measurement yet. - **n=3.** Enough to show T4 and T5 are unstable; not enough to put a confidence interval on the 0.139 gap. Treat it as a direction, not a coefficient. - **A small repository.** Five modules, ~200 lines. It tests localisation and care, not scale. - **Python only.** Orange's older suite was PHP; competence may not transfer across languages. - The pilot runs in `bench-results.jsonl` are **excluded** from every figure above. They were taken under a harness with a 60-second subprocess timeout, since raised to 180 with retries, and one of them recorded a regression failure later diagnosed as CPU starvation rather than code damage.