# Paper 04 — Agent harness architecture

**Working title:** *What actually makes an agent harness work: techniques, prompt engineering and measured results from two self-built harnesses*

Target: **8,000–10,000 words**, v1.0, same apparatus and evidential standard as papers 01–03.

---

## The subject

Two harnesses built by the author, measured rather than described:

| | **Axium** | **Orange** |
|---|---|---|
| Language | Rust | Python |
| Size | 28 source files, ~1.1 MB | 166 source files |
| Channels | CLI, SSH, TUI (web), Telegram | Desktop UI, voice |
| Layering | `classifier` → `router` → model slots | L1 router → L2 planner → L3 coder |
| Model slots | 6 slots, 2 providers | Per-layer model selection |
| Memory | Persistent store + compactor | Knowledge file + conversation store |
| Extension | Plugins + skills | Skills + MCP skills |
| Tools | 28–31, parallel execution | Tool suite + MCP |

Both independently converged on the same core technique — **a cheap model in front of an expensive one** — which is the paper's spine.

## The evidence that already exists

This is the reason the paper is viable. Orange carries a real eval harness with objective grading:

| Asset | Records | What it measures |
|---|---|---|
| `evals/logs_big/*.jsonl` | 76 | Brownfield fix/refactor tasks, **with a `with_brain` / `with_plan` ablation**, graded on `change` and `regress` with per-assertion detail, plus cost, wall time, call count |
| `evals/logs/*.jsonl` (2 dated snapshots) | 262 each | Flat chat-model × coder-model matrix over 5 escalating scenarios, objective checks, tokens, cost, latency |
| `evals/logs_layered/*.jsonl` (2 snapshots) | 385 each | 3-layer pipeline measured per layer **and** end-to-end, with **failure attribution** to router / planner / coder |
| `docs/ORANGE_LAYERED_PIPELINE.md` | — | Dated results, an L2 decision run, a "smarts" run, and a recorded methodological finding |
| `docs/ORANGE_BIGPROJECT_BENCH.md` | — | Seed project, objective grading, config switches, baseline + flash sweep |
| `docs/ORANGE_LLM_BENCHMARK_PLAN.md` | — | Candidate shortlist, the rigour debate, 5 scenarios, logging schema, pilot results |
| `evals/projects/{invoice_tracker,ledger}` | — | Acceptance-tested benchmark projects with a reference implementation |

**The ablation is the single most valuable thing here.** Almost nobody publishes whether their agent's architectural components actually help. `with_brain` × `with_plan` on the same tasks and the same model is a controlled comparison of techniques, not of models.

## What must be built new

The user asked for own tests and benchmarks. Three gaps:

1. **Axium has no eval harness.** Everything measured so far is Orange. A harness-agnostic task suite is needed so the two can be compared on identical work.
2. **No cross-harness runner.** Needs a thin adapter per harness and one scoring path.
3. **No prompt-engineering experiment.** The paper claims to be about technique; technique claims need an A/B, not an assertion.

## Design of the new benchmark

**Principle: grade the artefact, not the transcript.** Every task must have a machine-checkable acceptance test that runs against the resulting files, exactly as Orange's existing `change` / `regress` split does. No LLM-as-judge for correctness.

**Task families** (each with a seeded repo and an acceptance script):

| Family | What it tests | Grading |
|---|---|---|
| `fix` | Localised bug in an existing codebase | Assertion set passes **and** no regression in an untouched assertion set |
| `feature` | Add a small capability to a working app | Acceptance script exit code |
| `refactor` | Change structure, preserve behaviour | Behaviour assertions unchanged, structural assertion met |
| `brownfield-nav` | Find the right file in a codebase without being told | Correct file identified and modified |
| `tool-discipline` | Task solvable in N calls | Call count against a ceiling |
| `refusal` | Task that should be declined or clarified | Did it stop and ask, or invent |

**Controlled variables:** same task, same underlying model, same repo state. **Manipulated:** harness, and within harness the technique switches (planning on/off, memory on/off, extended thinking on/off).

**Recorded per run:** pass/fail per assertion, regression set, wall time, tool calls, input/output tokens, cost in USD at dated published prices, and the full transcript.

## The comparison to other harnesses — and its honest limit

The brief names **OpenClaw** and **Hermes**. Before any comparison is written:

1. Establish at primary source what each is, its architecture, and whether any published benchmark exists. Record the URL and the retrieval date.
2. **If a like-for-like measured comparison cannot be run, do not fabricate one.** The fallback is an *architectural* comparison — documented design against documented design — clearly labelled as such, plus any published benchmark figures quoted with their own methodology attached.
3. Running a third-party harness on the same suite is preferable and should be attempted. If it is not installable or not runnable here, that is recorded as a limitation, not papered over.

This is the same rule that governed the first three papers: **a comparison that cannot be evidenced is cut, and the cut is published.**

## Phases

| # | Phase | Output |
|---|---|---|
| 0 | Inventory and verify all existing eval data; aggregate the 723 records; confirm what each field means by reading the runner code | `data/existing-evals.csv`, verified field dictionary |
| 1 | Read both codebases for technique: routing, planning, memory/compaction, tool design, prompt construction | `data/technique-inventory.md` |
| 2 | Research OpenClaw and Hermes at primary; decide what comparison is evidenceable | `sources/source-log.csv` |
| 3 | Build the harness-agnostic benchmark: seeded repos, acceptance scripts, runner, scorer | `bench/` |
| 4 | Run it: both harnesses, technique switches on/off, repeated | `data/bench-results.jsonl` |
| 5 | Analysis: what technique actually moves the number | `data/analysis.md` |
| 6 | Write the paper, 8,000–10,000 words | `draft/paper.md` → HTML |
| 7 | Apparatus: references, citation, licence, COI, data availability. Render PDF. Audit | `broikos.gr/research/` |

## Standing rules carried from papers 01–03

- Every figure opened at source or cut, and the cut recorded
- Never invent a number, a benchmark result, a model version or a price
- Report the finding that weakens the argument — especially where the author's own harness loses
- **The author built both harnesses being evaluated.** This is the largest conflict of interest in the programme so far and must be stated prominently, with the mitigation: objective grading, published tasks, published transcripts, and a published run of anything that beats them
- Patch HTML only through an asserting helper; never hand-roll a splice
- Verify the rendered PDF, not the source
- Prices dated; model versions dated; every run timestamped