# 05 · What an agent actually costs

**Status: started 7 August 2026. Data extracted and analysed; no prose written yet.**

Seven dated versions of one production agent, 30 tasks each, one benchmark, one model
(`deepseek-v4-pro`), run 1–2 August 2026. **210 task-executions.** The record was made while
engineering the agent, not for publication, which is the reason it contains the regressions.

---

## The finding that decides the paper's shape

**The record is not comparable across versions as it stands, and the naive reading is wrong.**

v10 runs each task **3 times**; v3 through v9 run each **once**. Every run-level total therefore has
to be divided by task-executions before any two versions are set beside each other.

| | v9 | v10 | naive reading | corrected |
|---|---|---|---|---|
| cost | $0.3255 | $1.0714 | **3.3× more expensive** | $0.01085 → $0.01190 per execution, **+10%** |
| wall time | 10.5 min | 29.0 min | 2.8× slower | 20.9s → 19.3s per execution, **faster** |

A chart of run cost against version number would show a cost explosion at v10 that **did not
happen**. That chart is the one almost anyone would draw from this data.

## The corrected record

| ver | reps | score | $/execution | s/execution | tool calls/exec | Δ score |
|---|---|---|---|---|---|---|
| v3 | 1 | 87.1 | $0.00948 | 27.0 | 3.37 | — |
| v5 | 1 | 96.4 | $0.01183 | 29.4 | 4.67 | **+9.3** |
| v6 | 1 | 90.1 | $0.01309 | 26.7 | 5.30 | **−6.3** |
| v7 | 1 | 94.3 | **$0.00920** | **19.7** | 3.30 | +4.2 |
| v8 | 1 | 95.9 | $0.01189 | 25.9 | 4.03 | +1.6 |
| v9 | 1 | 95.6 | $0.01085 | 20.9 | 4.00 | −0.3 |
| v10 | 3 | **97.6** | $0.01190 | 19.3 | 1.45 | +2.0 |

**Cost per 100 points earned** — the number a buyer cares about:

| best | v7 | $0.00976 |
|---|---|---|
| worst | v6 | $0.01453 |

**v7 is the best-value version and nothing in the raw record says so.** It is not the highest
scorer; it is 49% cheaper per point than v6 and 25% cheaper than v10.

## What the paper can claim, with evidence already in hand

1. **Progress is not monotone.** 87.1 → 96.4 → **90.1** → 94.3 → 95.9 → **95.6** → 97.6. Two of six
   version transitions made the agent worse.
2. **v6 is a clean regression**: −6.3 points *and* +11% cost. Two tasks collapsed —
   `B24 Update installed software` (−88 points) and `B18 Inspect the Run keys` (−80).
3. **Score and cost decouple.** v8 scored 95.9 at $0.01189; v9 scored 95.6 at $0.01085. Three tenths
   of a point cost 10% more.
4. **One safety task regressed twice.** `B20 Block hive deletion` lost 40 points at v6 and again at
   v9. A safety check that keeps breaking is a different class of finding from a capability score.
5. **The failure mode is exhaustion, not error.** Across all 210 task-executions there are **four
   recorded errors and every one is a step limit** — twice "looped until it hit the step limit",
   twice "hit the step limit without finishing". Three of the four are package management. **The
   agent does not do the wrong thing; it fails to finish.**
6. **Saturation warning.** Only **2 of 30 tasks** never reached 100 in any version, both in
   `packages`. The suite is close to the ceiling and cannot rank much further — the same defect
   paper 04 documents in its own bench.

## What needs doing before this is publishable

- **Describe the 30 tasks precisely.** The proposal's own hard part. Numbers mean nothing without
  the task definitions; `data/benchmark-tasks.json` is published but needs prose.
- **Date the prices.** Cost is `estimated_cost_usd` from the runner. The rate table it used must be
  found, dated, and stated — the same discipline papers 02 and 03 apply to survey figures.
- ~~**Establish what changed between versions.**~~ **SOLVED.** The `windows-agent` git history (21
  commits) maps cleanly onto the run timestamps. See the timeline below; `data/timeline.py`
  regenerates it.
- **Check whether v4 exists.** The sequence skips it. No v4 run file and no commit names one — most
  likely an internal iteration never benchmarked. Say so rather than leaving a gap in the chart.
- **Decide on repetitions.** Only v10 has them, so only v10 has a variance estimate (`spread`).
  Re-running v7 and v9 at 3 reps would cost roughly $0.35 each and make the three most interesting
  versions comparable on reliability as well as score.
- **A negative control**, as in paper 04: confirm the graders fail on a do-nothing agent. Without it
  a 97.6 means less than it appears.

## Why this is the right second AI paper

Paper 04 asks whether a harness's architecture pays. This asks what a production agent costs across
its own engineering history — measured, with the regressions kept in. Paper 03 already published the
line that makes it the thesis: *a case study with no failures in it is marketing.*

The regressions are the asset. They are also the reason this is worth publishing: the industry
publishes capability claims, and the versions that got worse stay private.

## Provenance

Source: `C:\xampp\htdocs\windows-agent\bench\results` — 7 JSON runs plus transcripts, copied whole to
`data/runs/`. Extraction `data/extract.py`, analysis `data/analyse.py`, tidy record
`data/versions-by-task.csv` (210 rows). Nothing was modified in the source project.

---

## The timeline: what changed before each run

Reconstructed from commit timestamps against run start times. `data/timeline.py` regenerates it.

| ver | run started | score | $/exec | what preceded it |
|---|---|---|---|---|
| v3 | 08-01 19:22 | 87.1 | $0.00948 | rebuild as a package; safety gate fixed; benchmark suite added |
| v5 | 08-01 19:53 | 96.4 | $0.01183 | three fix commits from the first run's findings |
| v6 | 08-01 20:14 | **90.1** | $0.01309 | **no commits — run from uncommitted working-tree changes** |
| v7 | 08-01 20:16 | 94.3 | **$0.00920** | `41e8f35` *Stop misreading script structure as commands* |
| v8 | 08-01 20:30 | 95.9 | $0.01189 | `53e4980` *Stop rebuilding a missing capability by hand* |
| v9 | 08-02 01:48 | 95.6 | $0.01085 | coverage 33% → 66%, four bugs found by the new tests |
| v10 | 08-02 02:06 | **97.6** | $0.01190 | `7eb2081` *Set default applications for real: the UserChoice hash* |

### Three things this makes sayable that the scores alone do not

**1. The v6 regression never entered the repository.** There are no commits between v5 and v6: it was
run from an uncommitted working tree. Two minutes after v6 finished, the author committed
*"Stop misreading script structure as commands"*, and v7 — started **forty seconds later** — recovered
+4.2 points and became the cheapest version in the whole record. **The benchmark caught a bad change,
named it, and confirmed the fix, inside twenty minutes.** That is the case for having one at all, and
it is a better story than any score.

The two tasks that collapsed at v6 corroborate the diagnosis: `B24 Update installed software` (−88)
and `B18 Inspect the Run keys` (−80) are precisely where script structure gets confused with commands.

**2. Test coverage did not move the benchmark.** Before v9 the author took coverage from 33% to 66%
and fixed *four bugs the new tests found*. The benchmark score went **down 0.3**. Internal quality work
and external task performance are measuring different things, and this record shows it directly.

**3. The author documented the protocol change himself.** `6ec21f8 Add --repeat so the benchmark gives
a number worth trusting` is timestamped the same minute v10 started, and `edbda1f Stable baseline: 97.6
over 90 runs` records the 90. The normalisation this paper applies is therefore **corroborated by the
commit log**, not merely inferred from the JSON — which matters, because the whole cost argument rests
on it.