# Paper 08: How much of an agent is the model **Working title.** *The exchange rate: what a harness buys you, per unit of model.* Planned 20 August 2026. Written before a single trial runs, and published in this form, so that the design cannot be reshaped around whatever the results turn out to be. --- ## The subject Paper 04 measured three independently built harnesses against **one model** and concluded that the harness decides less than everyone assumes. That sentence has a hole in it, and the hole is the whole of this paper: **if the harness decides less, what decides more, and by how much?** Nobody has published the crossed measurement. The field publishes harness comparisons at a fixed model (leaderboards, framework benchmarks) and model comparisons at a fixed harness (every model card, every eval suite). The two axes are never crossed, so the question every person building an agent actually asks has no measured answer: > I have a budget. Do I spend it on a better model or on a better harness? The claim worth owning is the **exchange rate**: how many tiers of model capability a good harness is worth, and where that stops being true. ### The four outcomes, all publishable | if the data shows | the finding is | |---|---| | model tier dominates, harness spread is noise | the harness is plumbing, and paper 04's conclusion generalises | | harness dominates at every tier | scaffolding is the product, and the model is a commodity input | | interaction: harness rescues weak models, adds nothing to strong ones | the harness is insurance, and its value curve is downward sloping | | interaction: harness only pays off with a strong model | scaffolding needs a model good enough to use it, which is the opposite of the market's pitch | The third and fourth are the interesting ones and they are opposites, which is what makes this worth measuring rather than asserting. --- ## The questions 1. **Main effect, model.** Holding the harness fixed, how much does score, damage, cost and restraint move across four capability tiers? 2. **Main effect, harness.** Holding the tier fixed, how much do six harnesses differ? 3. **The interaction.** Does the harness spread widen or narrow as the model gets better? 4. **The floor.** Is there a capability level below which no harness produces a working agent, and what fails first: the reasoning, or the tool protocol? 5. **The behavioural axes.** Do restraint, asking before destruction and continuity track model capability at all, or are they purely structural? Paper 04 says structural; that was at one tier. --- ## Design A two-factor factorial: **harness x capability tier**, repeated, on two benchmark suites, with a negative control on both. ### Factor A: the harnesses Six, of which two are the author's, one was measured in paper 04, and three are third party and new to this programme. Feasibility below is verified on this machine as of 20 August 2026, not assumed. | harness | built by | language | invocation | model override | adapter | |---|---|---|---|---|---| | **Axium** (Python port) | author | Python | in process | `--model`, `--continuation` | exists, `versus/adapters.py` | | **Orange** | author | Python | in process | `--orange-chat`, `--orange-coder` | exists, `versus/adapters.py` | | **Hermes** | NousResearch, MIT | Python | in process | `OPENAI_BASE_URL` plus model id | exists, paper 04 `data/hermes_adapter.py`, **clone is gone and must be re-fetched** | | **DeepSeek Harness (dsh)** | DeepSeek, MIT | Node | `npx @deepseek-ai/dsh --profile headless ""` | custom provider, base URL plus model | to write | | **windows-agent** (`pc-agent.py`) | author | Python, LangGraph | subprocess | `DEEPSEEK_MODEL` / `/model`, provider switchable | to write | | **OpenClaw** | Steinberger and contributors | Node | `openclaw agent --message ""` | `openclaw config set` on the `model` section | to write | **ZeroClaw is a stretch, not a commitment.** Rust, single binary, provider pluggable, and `cargo` is present, but it is channel oriented rather than task oriented and its single shot path needs proving before it earns a place. If it is not in the final set, it is named as not done, with the reason, in the paper. **Why these six.** Two harnesses whose internals the author can read, one independent Python harness already measured at one tier so paper 04 is directly extended, the newest harness in the market (dsh, developer preview since 13 August 2026, no independent measurements exist anywhere), the most adopted community harness (OpenClaw), and one deliberately simple agent built as a first line of defence, which is the control for "how much scaffolding do you actually need". ### Factor B: the capability tiers Every harness in the set speaks the OpenAI chat completions wire format or can be pointed at it, which is what makes a shared ladder possible at all. **Revised 20 August 2026, before any measured run, and the reason is recorded rather than tidied away.** The ladder was planned as four rungs with deliberation switched off and on at the same weights. Deliberation is not crossable: Hermes documents `reasoning_effort` for OpenRouter and Nous Portal only, not for an OpenAI compatible base URL pointed at DeepSeek, so a thinking rung would have been expressible for the two harnesses I wrote and not for the one I did not. **A factorial with holes in the third party column measures the author, not the field.** So deliberation becomes a side study on the subset that exposes the switch, and the primary ladder moves the weights only. | tier | model | deliberation | why it is on the ladder | |---|---|---|---| | **T0** | `llama3.1:8b`, served OpenAI compatible on localhost | none available | the floor. Free, reproducible by any reader, no account | | **T1** | `deepseek-v4-flash` | each harness's own default, recorded per cell | cheap weights | | **T2** | `deepseek-v4-pro` | each harness's own default, recorded per cell | flagship weights | | **T3** | `gpt-5.6-luna` | reasoning off, forced by the endpoint | a second vendor, above pro on nearly every public benchmark and a tenth of its top tier's price | | *S-off* | `deepseek-v4-flash` | thinking off | side study, subset only | | *S-high* | `deepseek-v4-flash` | thinking high | side study, subset only | **On the floor model.** It is `llama3.1:8b` rather than the `qwen2.5:7b-instruct` first stood up, because **Hermes refuses any model advertising under 64,000 tokens of context** and the local server reports the 7B at 32,768. A rung one harness cannot accept is not a rung. The refusal is a finding about the harness and is recorded as one. **On the fourth rung, added 21 August 2026 at the author's request.** The question it answers is whether a model measurably above `deepseek-v4-pro` changes the picture. `gpt-5.6-luna` is the vendor's cheap tier and it still leads pro on the composite intelligence index, 51.2 against 44.3, on SWE-bench Pro, 62.7 against 55.4, and on Terminal-Bench, 84.7 against 67.9, while costing 0.20 and 1.20 USD per million against pro's 0.66 and 1.98. Its middle tier, `gpt-5.6-terra`, scores a little higher again and costs ten times as much, which buys nothing this study needs. **On how the harnesses reach it.** Not the same way, and the row says which. That model refuses function tools on chat completions unless reasoning is explicitly off, so four harnesses run it with reasoning off, while two reach it through the Responses API and reason at their own default. The rung is one model; the wire is not. **On leaving deliberation at each harness's default.** It is not a missing control. It is the setting a person gets when they install the thing and type, it is recorded per cell as `effort_effective`, and the side study is what isolates it. **Equalisation rules, stated because each one changes a score.** - Harnesses that use two models internally (Axium primary plus continuation, Orange chat plus coder) get **both pinned to the same tier**. A harness is not allowed to smuggle a stronger model into a cheap role. - Local fastpaths that answer without a model call are **left switched on** and reported separately. They are a harness feature, they were paper 04's cleanest confirmed result, and disabling them would be measuring a harness nobody ships. - Every harness gets the same temperature and the same per turn output cap where it exposes them, and where it does not, that is recorded as an uncontrolled variable rather than quietly ignored. - Context window differs by model, not by harness, and is held constant within a tier. ### The suites **Suite V, behaviour.** The five `versus` scenarios from paper 04, unchanged, so this paper's numbers join that paper's record rather than replacing it: V1 repair, V2 restraint, V3 continuity across six turns, V4 blast radius, V5 economy. **Suite W, coding effect.** The six scenarios in `windows-agent/bench_coding.py`, ported into the `versus` scenario format so one runner drives both: W1 unstated scope with decoys, W2 repair then extend, W3 two interacting bugs with a byte identical golden output, W4 encoding hazard with Greek text, W5 an impossible sub goal that must be reported rather than faked, W6 tidy up without destroying user data. Two suites because a single suite measures one kind of intelligence. V asks whether an agent behaves; W asks whether it can actually write code that runs. The hypothesis worth stating in advance is that **model tier moves W much more than it moves V**, and if that holds, "how much does the model matter" has different answers for the two halves of the job. ### Scale | | | |---|---| | cells | 6 harnesses x 4 primary rungs, plus 4 harnesses x 2 side study rungs | | scenarios | 11, five in V and six in W | | repetitions | **2**, revised down from 3 on 21 August 2026. The grid grew a fourth rung, and two repetitions across four rungs buys more than three across three: a claim can still be shown as 2 of 2 or 0 of 2, and every cell publishes its per repetition scores so a reader can see the spread rather than a mean hiding it | | sessions | primary 6 x 4 x 11 x 2 = **528**, side study 4 x 2 x 11 x 2 = **176** | Nearly 800 multi turn sessions is roughly seventeen times paper 04's 45, and the wall clock, not the tokens, is the binding constraint. One turn at T0 measured 42 seconds through Axium and 16 through Orange on this machine, so the floor rung is affordable in time as well as in money. **The run is staged and each stage is gated**, so a design error costs one stage rather than the whole sweep. If a cell has to be dropped, it is dropped whole and named, never sampled thinner and reported as if it were complete. ### The negative control, before the first paid run The single most expensive lesson in this programme is that paper 05 published seven versions of a benchmark before anyone asked what an agent that does nothing would score. The answer was 22.8 per cent, and an agent that emitted plausible prose while touching nothing scored 36.8. **This benchmark ships its control or it does not ship.** Two of them: 1. **The null agent.** A no op adapter that accepts every turn and does nothing. Its score on both suites is the floor, published, and every table in the paper is read against it. 2. **The prose agent.** An adapter that answers plausibly and touches no file. This catches checks that pass by inaction, which is what 26 of paper 05's restraint checks did. `versus.runner --sanity` already refuses to start a paid run unless the acceptance suite is green on the untouched seed and every scenario grader fails before an agent touches anything. That gate stays, and the two controls run before every stage, not once at the beginning. --- ## What must be built | # | item | why | |---|---|---| | 1 | Re fetch the Hermes clone to `C:\agent-eval-thirdparty\hermes` | the paper 04 adapter points there and the directory no longer exists | | 2 | A T0 provider on localhost, OpenAI compatible | the floor rung. GPU present is an RTX 4060, so a 7B or 8B at 4 bit fits | | 3 | Adapter: dsh | headless mode exists, workspace pinning and state directory need proving | | 4 | Adapter: windows-agent | it is a subprocess CLI, so tool calls are read from its ledger rather than by wrapping a dispatcher, which is a different provenance and must be declared | | 5 | Adapter: OpenClaw | `openclaw agent --message` plus non interactive config | | 6 | Port suite W into the `versus` scenario format | one runner, two suites, one grading path | | 7 | Null and prose control adapters | see above | | 8 | `IGNORE_DIRS` entries for every new harness's own state | this is the bug that produced three false headline claims in paper 04 | | 9 | A tier sweep driver over `versus.runner` writing one JSONL per cell | resumable, because 792 sessions will be interrupted | | 10 | Cross contamination check across cells | paper 04's other recurring bug: a correct number on the wrong page | Item 8 deserves its own line in the paper. **The instrument in paper 04 was systematically biased against agents that keep state inside the working directory**, it failed reproducibly, and it looked right. Six harnesses means six state directories, and the same bug is available six times over. --- ## Threats to validity, and what is done about each | # | threat | handling | |---|---|---| | 1 | **Harness and model are not independent.** A harness tuned against one model family may be measured at its best on one rung only | Stated as the central limitation. Each harness's own default model is recorded, and cells running a harness at its home tier are flagged in every table | | 2 | **The tool protocol floor.** A weak model may fail at emitting a valid tool call, not at the task | Tool call validity is graded separately from task score. A T0 cell that fails is reported as protocol failure or task failure, never as one number | | 3 | **Third party version drift.** dsh is a developer preview whose APIs are stated to be changing | Exact version and commit pinned per harness, recorded in the ledger, and re run if a version moves mid sweep | | 4 | **Non determinism** | 3 repetitions per cell, per scenario variance published, and no claim made from a single session | | 5 | **Windows.** Every measurement is on one operating system, and one of paper 04's run killers was a file named `nul` | Declared. The `nul` guard from paper 04 is applied before the first run rather than after | | 6 | **Prompt surface differs by harness.** System prompts and tool counts are not equalised and cannot be | Recorded per harness: tool count, system prompt token count, whether it preloads a repository map. Paper 04 already found that preloading and planning passes cost 38 per cent more and bought nothing | | 7 | **Peak and off peak pricing.** DeepSeek doubles its rates in two daily windows | Every session records its start time in UTC and the rate table used. Cost comparisons are computed at one rate, stated on the exhibit | | 8 | **Author conflict.** Two of six harnesses are the author's own | Declared in the paper, on the cover, as in paper 04. The pre registered claims below are written before any result exists, and the third party harnesses are graded by exactly the same code | --- ## Pre registered claims Written now. Whatever the data says, these are the questions that get answered, and a claim that flips is published as flipped. | id | claim to test | direction predicted | |---|---|---| | P1 | Model tier explains more variance in suite W than harness identity does | model dominant on coding | | P2 | Harness identity explains more variance in suite V than model tier does | harness dominant on behaviour | | P3 | Asking before destruction is a harness property, invariant across tiers | invariant, extending paper 04's C3 | | P4 | The harness spread narrows as tier rises | insurance hypothesis | | P5 | T1 to T2, identical weights with deliberation switched on, moves score more than one harness change does | untested anywhere | | P6 | At T0 every harness fails, and the failure is protocol rather than reasoning | the floor | | P7 | Cost per completed task is not monotonic in tier: a cheaper model doing more turns can cost more | the result that would be most useful commercially | --- ## Phases and gates | phase | work | gate before the next phase | |---|---|---| | 0 | Re fetch Hermes, stand up T0 locally, pin versions | every harness answers one trivial prompt at every tier, by hand | | 1 | Adapters for dsh, windows-agent, OpenClaw, plus the two controls | `--sanity` green, every grader fails on the pristine seed, every harness's state directory ignored | | 2 | Port suite W | each W grader fails on its own pristine seed and passes on a hand fixed seed | | 3 | Pilot: 1 rep, all cells, suite V only | no cell errors out for a plumbing reason. Scores are not read as findings | | 4 | Full sweep, staged by tier | cross contamination check clean after every stage | | 5 | Analysis, exhibits, claim ledger | every headline figure recomputed from the JSONL by a second script | | 6 | Draft, audit, publish with data | the standing rules below | **Believability before reproducibility.** Every headline figure is checked for whether it is plausible before it is checked for whether it recomputes. Five errors in this programme were perfectly reproducible and wrong. --- ## Standing rules carried from papers 01 to 07 - Every number is `external-cited`, `own-data-derived` or `assumption`, in `claims/claim-ledger.csv`, and the reader can tell which. - Failed runs, dropped cells and abandoned harnesses are named in the paper, never omitted. - Data published: every session JSONL, the adapters, the graders, the sanity gate, the sweep driver and the analysis, under CC BY for the text and MIT for the code. - No emojis. No dashes as punctuation. - Store the evidence, never the verdict: every session keeps its raw turn text and tool arguments, so any regrade is free. - The instrument gets its own section, and its defects are findings rather than embarrassments.