# Labelling rule for the classifier benchmark

Written **before** any prompt was scored, 6 August 2026. The gold labels use Axium's own four classes, defined at `src/agent/classifier.rs:8-19` and constrained by the classifier's own system prompt at `:60`.

## The four classes, as Axium defines them

| Class | Axium's definition | Consequence if chosen |
|---|---|---|
| `trivial` | *"Trivial question — answer directly with the classifier model, skip primary."* Restricted by the system prompt to *"pure factual/math questions with no ambiguity (time, math, unit conversions, simple facts)"* | The cheap model answers. The primary model is never called |
| `simple` | *"Simple question — pass through to primary unchanged."* | Primary model, full pipeline, **reduced 18-tool set** |
| `medium` | *"Medium task — uses primary model but skips quality review and code review. Tests still run if detected. Covers clear code tasks that don't need enhancement."* | Primary model, review stages skipped |
| `complex` | *"Complex task — enhanced prompt replaces the original for the primary model."* | Classifier rewrites the prompt first |

## Labelling decisions taken, and why

1. **Capability, identity and memory questions are `simple`, never `trivial`** — even when they look like one-line questions. Axium's own classifier prompt forbids `TRIVIAL` for these, because only the primary model holds memory tools. A prompt like *"do you remember what I told you yesterday?"* is short and factual-looking and must not go to the cheap path.
2. **Greetings and acknowledgements are `simple`, not `trivial`.** Axium's fast path routes them to the primary model. Labelling them `trivial` would score the harness against a behaviour it deliberately does not want.
3. **`trivial` is reserved for questions answerable completely and unambiguously without tools, state or the user's project** — arithmetic, unit conversion, a stable general fact.
4. **A single, well-specified edit to a named file is `medium`.** It needs tools and the primary model, but the instruction is already clear, so enhancement would add nothing.
5. **`complex` requires at least one of**: more than one file or subsystem, a vague brief that must be interpreted before it can be executed, an explicit multi-step sequence, or a stated non-functional constraint (performance, security, migration safety).
6. **Destructive or irreversible requests are labelled by their task shape, not their risk.** Safety is a separate concern in Axium, handled by `ask_user` and the plan gate, not by the classifier. Labelling them `complex` because they are dangerous would test the wrong thing.
7. **Length alone does not make a prompt `complex`.** A long but single-purpose question is still `simple` or `medium`. This matters because the scorer has a hard override at 500 words and a length term at weight 0.08, so the set must contain long-but-simple prompts to expose that.

## Known limitation

**Single labeller, no second coder.** These labels are one considered view, produced by the same person analysing the results. This is the same weakness that paper 01 measured and reported in its motive column, where independent coding dropped agreement to κ 0.34.

The mitigation here is weaker than it should be, and the paper must say so: the labelling rule above is published so a reader can re-derive any row and dispute it, and the full prompt set with labels is published alongside. No inter-rater check has been run.

## What the benchmark measures

The deterministic path only — `quick_classify_trivial` then `score_prompt`. **No API calls.** It answers:

1. What share of prompts does the local path resolve without an LLM? (Axium's comment at `:918` claims 60–70%.)
2. When it resolves locally, is it right?
3. When it is wrong, does the error land on the safe side? Misrouting `complex` down to `simple` degrades quality; misrouting `simple` up to `complex` only wastes money.

The local path can emit only `simple` and `medium`. **`trivial` and `complex` are structurally unreachable without the LLM** — `quick_classify_trivial` returns `Simple` in every branch, and scores at or above 0.25 deliberately fall through so the LLM can enhance. Any `trivial` or `complex` prompt therefore *must* cost an LLM call by design, and that is a property of the architecture rather than a failure of the scorer.