# shopkit-bench

A harness-agnostic agent benchmark. Built 6 August 2026 for paper 04, after Orange's existing brownfield suite was found saturated at 48/48 with **zero** regression damage across 48 runs — an axis that measured nothing.

## Design rules

1. **Grade the artefact by executing it.** Never grade the transcript. No LLM-as-judge for correctness.
2. **Two independent axes.** `change` = did the required behaviour appear. `regress` = did anything else break. Separate assertion sets, reported separately, never pooled.
3. **A negative control gates everything.** `sanity()` must pass before any result is reported.
4. **Every golden value is derived by executing `reference/`.** No expected number is hand-written anywhere in this suite.

## The seed application

`seed/` is `shopkit`, a small order and inventory toolkit in Python: `money`, `db`, `orders`, `reports`, `cli`, plus a five-product, five-order dataset. It runs with no server and no dependencies.

Three defects are planted, each marked `PLANTED BUG` in the seed and each fixed in `reference/`:

| | site | defect |
|---|---|---|
| T1 | `money.fmt` | `math.floor(v*100)/100` truncates instead of rounding — a cent is lost |
| T2 | `reports.is_overdue` | `due <= today` flags orders due *today* as already overdue |
| T3 | `orders.discount_amount` | stacked discounts are **summed**, so two 60% discounts give 120% and **a negative total** |

The reference differs from the seed at exactly these three sites and nowhere else.

## Tasks

| id | kind | brief style |
|---|---|---|
| T1 | fix | specific, names the symptom and the value |
| T2 | fix | specific, states the rule |
| T3 | fix | specific, names the order |
| T4 | compound | all three at once, "without breaking anything else" |
| T5 | **vague** | *"Customers are complaining that the totals on their invoices look wrong, and one customer was quoted a negative amount."* — must locate before fixing |

T5 is the hardening lever Orange's own documentation asked for: a brief that requires diagnosis rather than execution.

## The negative control

`python shopkit_bench.py sanity` asserts four things and refuses to report results otherwise:

1. The regression set is **green on the pristine seed** — so any later regression failure is the agent's doing, not a broken baseline.
2. **Every change-grader fails on the pristine seed** — a grader cannot pass for free.
3. **Every change-grader passes on the reference build** — the task is actually achievable.
4. The regression set is **green on the reference build** — the intended fix does not itself regress.

Verified 6 Aug 2026: `sanity: 0 problem(s)`, 13 regression assertions.

## The regression axis demonstrably bites

This is the part Orange's suite could not do. A `vandal` adapter makes plausible wrong edits — mistakes an agent could really make while attempting T1 — and the axis grades them across a range:

| vandal mode | what it did | `regress` | assertions broken |
|---|---|---|---|
| *(noop)* | nothing | **1.0000** | 0 |
| `returns_float` | fixed the rounding, changed the return type | 0.9231 | 1 |
| `kills_error_path` | tidied the CLI, lost the non-zero exit code | 0.9231 | 1 |
| `breaks_subtotal` | rounded in the wrong place | 0.8462 | 2 |
| `edits_the_data` | "fixed" the price in the dataset instead of the code | 0.7692 | 3 |
| `syntax_error` | left the build unparseable | **0.0000** | 13 |

A build that does not import scores 0 and records the exception in `build_error`; it does not crash the harness. That defect was found and fixed during construction.

## Adapters

A harness is plugged in by implementing one function:

```python
def run(build_path, brief):
    """Make edits under build_path to satisfy brief. Return a metadata dict."""
```

Shipped: `noop` (control), `vandal` (regression control). **Not yet written: `orange`, `axium`.**

## Usage

```
python shopkit_bench.py sanity
python shopkit_bench.py list
python shopkit_bench.py run --adapter noop
python shopkit_bench.py run --task T5 --adapter orange
```

`run` executes `sanity()` first and refuses to proceed if it fails.

## Known limitations

- **Five tasks, one repository, ~200 lines of seed code.** Smaller than Orange's ~45-file PHP suite and far smaller than a real project. It tests localisation and care, not scale.
- **Partial credit is real.** Some assertions are incidentally true on the buggy build — `noop` scores `change` 0.5 on T2 and 0.25 on T3. Sanity requires only that a grader is not *fully* satisfied on the pristine build. Report `change` as an assertion fraction, never as a pass rate.
- **Python only.** Orange's existing suite is PHP; a harness that is good at one may not be at the other.
- **No repetition yet.** Single runs measure nothing about variance.