# Brownfield suite — corrected analysis Phase 0 step 2, 6 August 2026. Source: `orange/evals/logs_big/*.jsonl`, run window **2026-07-20T21:02 to 22:03**. Coder models `deepseek-v4-flash` and `deepseek-v4-pro`. Prices from `pricing.py`, fetched 2026-07-05 and re-verified 2026-07-19. Deduped data: `brownfield-deduped.csv` (72 rows). Field definitions: `field-dictionary.md`. --- ## A data-integrity finding, before any result **The raw logs contain 4 superseded reruns and naive aggregation reports the wrong number.** Four task ids appear twice in `deepseek-v4-pro.jsonl` — A2, B2, B7 and R2. The later stamp is the corrected value in each case: | id | first run | rerun | why | |---|---|---|---| | B7 | 0.667 @ 21:09:44 | **1.000** @ 21:46:40 | grader bug: a text-slice comparison that always compared the identical template fragment, fixed to an id-set comparison | | R2 | 0.667 @ 21:03:20 | **1.000** @ 21:14:01 | rerun after the same fix | | A2, B2 | 1.000 | 1.000 | reruns, no change | A first pass over these logs without deduplication produced `change` 0.976 for pro-bare. The correct figure is **1.000**. The difference is entirely superseded records made with a grader that was later found broken and fixed. **Rule for the paper: dedupe by (config, task id) keeping the latest stamp, and say so.** Anyone re-analysing these logs without that step gets a wrong and slightly worse-looking number. One residue remains and is flagged rather than hidden: **B7 in the `brain+plan` config still reads 0.667 on the same "pages differ" assertion.** That configuration was never re-run after the grader fix. The author's own results table records it as 100%, having corrected it by hand. This analysis keeps the logged 0.667 and marks it a known grader artefact — the honest position is that the true value is very probably 1.000 but was not re-measured. --- ## Corrected results All three configs, 24 tasks each: 8 fix, 8 refactor, 8 awareness. ### Coding tasks — fix and refactor, 16 per config | Config | n | change | regress | cost | perfect | |---|---|---|---|---|---| | `deepseek-v4-flash` bare | 16 | **1.000** | 1.000 | **$0.0716** | 16/16 | | `deepseek-v4-pro` bare | 16 | **1.000** | 1.000 | $0.1856 | 16/16 | | `deepseek-v4-pro` brain+plan | 16 | 0.979¹ | 1.000 | $0.2031 | 15/16 | ¹ the single miss is the B7 grader artefact above. ### Awareness tasks — 8 per config, graded by F1 ≥ 0.7 against a truth set of file paths | Config | change | notable failure | |---|---|---| | `deepseek-v4-flash` bare | 0.917 | A3 invented a file; A7 f1 0.64, **precision 0.47** | | `deepseek-v4-pro` bare | **1.000** | — | | `deepseek-v4-pro` brain+plan | 0.958 | **A8 hallucinated callers** | ### Whole suite | Config | total cost | wall | |---|---|---| | flash bare | **$0.0853** | 15 min | | pro bare | $0.2249 | 18 min | | pro brain+plan | $0.2374 | 19 min | These reconcile with the author's own results table in `docs/ORANGE_BIGPROJECT_BENCH.md` on every cell. --- ## Findings ### 1. The expensive coder bought nothing on this suite Flash and pro both scored **16/16 on every coding task** — all 8 planted bugs fixed, all 8 cross-file refactors completed, zero broken pages. Flash did it for **$0.0716 against $0.1856, 2.6× cheaper**, and finished the whole suite in 15 minutes against 18. This is a result against the harness's own multi-model architecture, which exists to route hard work to the expensive slot. On this evidence, at this scale, the routing bought nothing on correctness. ### 2. Adding context made honesty worse The `brain` preload builds a repo map and injects it before the task. On awareness tasks, **pro with brain+plan scored 0.958 against pro bare's 1.000**, and the single failure is A8, an honesty trap: the assertion is literally `no hallucinated callers`. The repo map appears to have tempted the model into naming plausible-but-unverified callers. **More context made the model claim more than it had checked.** This is one run and must be replicated before it is load-bearing, but it points the same way as the flash awareness failures, and it is the most interesting result in the dataset. ### 3. Flash's weakness is not coding, it is claiming Flash matched pro on every coding task and lost only on awareness: A3 invented a file that does not exist, A7 reached **precision 0.47** — more than half the files it named as relevant were not. Flash writes correct code and unreliable claims about code. That distinction matters for harness design: it argues for routing by *task type* (claims versus edits) rather than by difficulty. ### 4. The regression axis is fully saturated and currently useless **0 of 48 graded runs caused any regression damage.** Every page of the seeded application rendered without error after every edit in every config. The axis I expected to be the most informative measures nothing on this suite. It is not worthless — it is a genuine safety net that would catch a future harness regression — but as a discriminator between models or techniques it is dead, and the paper must say so rather than report "100% regression-free" as though it were an achievement. ### 5. The coding suite is saturated overall 48 of 48 coding tasks passed across all three configurations. **A benchmark on which everything passes cannot rank anything.** The author's own documentation reaches the same conclusion and lists hardening levers: compound interacting bugs, vaguer briefs that require locating before fixing, a 200+ file seed tier, and treating latency, cost and tool-calls as first-class axes. This is the justification for Phase 3. The existing suite cannot answer the paper's question, and a harder one has to be built. --- ## What this does not establish - **Nothing about Axium.** It has never been benchmarked. Every number here is Orange's. - **Nothing about brain or plan separately.** The design is not a 2×2 factorial: only three cells exist (flash bare, pro bare, pro brain+plan). Brain-only and plan-only were never run, so no effect can be attributed to either component individually. - **Nothing with statistical power.** One repetition per task per config. n=8 per family. The 0.958-versus-1.000 differences are single-task swings, not measured effects. - **Nothing at production scale.** The seed project is ~45 files. Real projects are larger and messier, which is exactly where the author argues the expensive model earns its premium — untested either way. --- ## Next Phase 1: technique inventory from both codebases. Then Phase 3: a harder, harness-agnostic suite that can run against Axium as well, built to the hardening levers above, because the existing one is saturated.