# Can Axium be benchmarked? — CORRECTED, 6 August 2026

## Correction notice

An earlier version of this file concluded that *"Axium has no non-interactive entry point and cannot be driven by a benchmark adapter without a substantial build."*

**That conclusion was wrong.** It was reached from a partial read — `main.rs`'s mode dispatch and the HTTP route table — without reading the REPL loop itself or the README's own CLI section. The same version also raised doubt about whether Axium compiles, which had no basis at all: it is built and used daily.

The corrected position is below, with the line that settles it.

---

## Axium **is** drivable non-interactively

`src/channels/cli.rs:103-106`:

```rust
let line = match lines.next_line().await {
    Ok(Some(l)) => l,
    Ok(None) => break, // EOF
```

**The REPL exits cleanly on EOF.** So piping a brief in and closing stdin runs the task and terminates the process:

```
printf '%s\n' "<brief>" | axium --config <per-task-config.json>
```

The README documents the CLI as the *default* mode and states it carries the **full agent stack** — *"same classifier, compactor, memory, plugins, and all 31 tools as the browser UI"* — so this is not a degraded path. It is the primary one.

## Every obstacle I claimed has a supported answer

| Earlier claim | Reality | Source |
|---|---|---|
| No one-shot entry point | Piped stdin + EOF is one-shot | `cli.rs:105` |
| Working directory is a config file that must be rewritten | `--config <path>` is the **first** resolution rule, so a temp config per task works | `main.rs:236-243` |
| Sessions auto-resume and contaminate consecutive runs | `/new` clears the session, and it can be piped as the first line | `cli.rs:118-122` |
| Completion has no signal | **Process exit is the signal** | `cli.rs:105` |
| Would need a WebSocket client and protocol reverse-engineering | Not needed. That applies to `--server` mode, which is one of two modes, not the only one | `main.rs:139` |

The model is pinnable per run too: `config.json` carries separate slots for `primary`, `classifier`, `compactor`, `review` and `fallback`, each with its own provider (`config.example.json`).

## The adapter is small

Critically, **shopkit-bench grades artefacts, not transcripts.** Nothing needs to parse Axium's streamed output. The adapter only has to:

1. Write a temp `config.json` with `settings.working_directory` set to the build path and the model slots pinned.
2. `printf '/new\n<brief>\n' | axium --config <temp>`.
3. Wait for exit, then let the existing graders execute the build.

Cost and call counts are the one thing that needs more thought — Axium reports token usage as `AgentEvent::TokenUsage` to its UI, not to stdout in a documented format, so per-run cost may have to come from the SQLite session store or be recorded as unavailable. That is a detail, not a blocker.

## Revised decision

**Write `axium_adapter.py` and take the end-to-end measurement.** The earlier decision to ship without it was based on a false premise and is withdrawn.

Two things must be resolved first, and neither is assumed here:

- **Where the built binary lives on this machine.** This source checkout has no `target/` directory; the README describes installing via `install.sh` or as a systemd service, so the binary is elsewhere. It must be located, or built with the cargo 1.96.1 that is present.
- **Which API keys the run should use**, since Axium's config carries its own and the benchmark should pin a model comparable to the one Orange was measured with.

## What survives from the earlier version

One observation, and it is weaker than I first stated. Orange ships a purpose-built evaluation harness — objective graders, a negative control, cost tracking, ablation switches — and Axium does not. That remains true and is worth a paragraph. But it is a statement about **evaluation tooling**, not about whether the harness can be driven, and the earlier version conflated the two.

## Method note for the paper

This file is kept with its correction visible rather than silently rewritten. The error was: generalising from a partial read of two files to a categorical claim about a system, and then adding a gratuitous doubt about compilation on top of it. The fix was reading the twenty lines that actually governed the behaviour.

It belongs in the paper's limitations, because a paper that grades other people's evidence should show its own retractions.