# Axium's classifier, measured

Phase 3a, 6 August 2026. 102 labelled prompts, deterministic path only, **no API calls, zero cost**. This is the first measurement of any kind ever taken on Axium.

Artefacts: `bench/classifier/{scorer.py, prompts.csv, run.py, results.csv, LABELLING-RULE.md, parity_notes.md}`.

---

## The claim, and the result

`classifier.rs:918` states: *"The scorer handles 60-70% of requests locally; the LLM handles ambiguous cases."*

**Measured: 36.3% (37 of 102).** The claim is not supported on this prompt set.

But the interesting part is where the 36.3% comes from.

| Stage | What it is | Prompts resolved | Share |
|---|---|---|---|
| 1 | Deterministic keyword patterns (`quick_classify_trivial`) | 35 | **34.3%** |
| 2 | The ten-dimension weighted scorer | **2** | **2.0%** |
| 3 | Deferred to the LLM | 65 | 63.7% |

**The most elaborate component in the classifier resolves two prompts in a hundred.** A hand-written list of greetings, identity phrases and memory verbs does seventeen times more work than the weighted scorer it exists to feed.

---

## Why the scorer almost never fires

Not a tuning problem. A structural one.

Confidence is a sigmoid on distance from the **nearest** tier boundary:

```
confidence = 1 / (1 + exp(-10 · min_distance))
```

with boundaries at **−0.02** and **0.25**. Clearing the 0.75 gate therefore requires being at least **0.11 away from both**. The Medium band between the boundaries is only **0.27 wide**, so a score landing anywhere inside it can be at most 0.135 from a boundary — a maximum achievable confidence of about 0.794, and only at the exact midpoint.

The observed distribution makes this fatal. Of the 67 prompts that reach the scorer:

| statistic | value |
|---|---|
| median confidence | **0.5498** |
| 90th percentile confidence | 0.7311 |
| maximum confidence | 0.9000 |
| clearing the 0.75 gate | **4 of 67** |
| median score | **−0.016** |
| inside the Medium band | 35 of 67 = **52%** |

The median score is −0.016, which sits **0.004 from the −0.02 boundary**. The typical prompt lands almost exactly on a decision boundary, where the scorer is by construction least able to be confident. Half of all prompts fall in the narrow band where high confidence is arithmetically unreachable.

---

## When it does fire, it is right

**Accuracy on locally-resolved prompts: 36 of 37 = 97.3%.**

The confusion matrix is nearly diagonal: 35 `simple` correct, 1 `medium` correct, one error.

That single error is the one that matters, and it is worth quoting in full:

> `first audit the api for injection risks, then fix them, then add regression tests`
> gold `complex` → classified `medium`, score 0.1372, confidence 0.7555

A three-stage security task cleared the gate by 0.0055 and was routed to the pipeline that **skips quality review and code review**. This is the failure mode with real cost: not wasted money, but a security refactor losing its review stages. One case in 102, and it is the case the design should most want to avoid.

---

## The gate is well chosen; the docstring is not

Sweeping the confidence gate over the same set:

| gate | local resolution | accuracy | complex tasks captured by the cheap path |
|---|---|---|---|
| 0.60 | 53.9% | 72.7% | **6** |
| 0.65 | 49.0% | 76.0% | 5 |
| 0.70 | 42.2% | 86.0% | 1 |
| 0.73 | 42.2% | 86.0% | 1 |
| **0.75 (shipped)** | **36.3%** | **97.3%** | **1** |
| 0.80 | 34.3% | 100.0% | 0 |
| 0.90 | 34.3% | 100.0% | 0 |

Three things follow.

1. **The shipped 0.75 is close to the right answer.** It sits at the knee: accuracy collapses from 97.3% to 86.0% for six extra points of local resolution.
2. **Reaching the claimed 60–70% would require a gate near 0.60**, costing 6 complex tasks captured by the cheap path and 24 points of accuracy. **The claim is not achievable at acceptable quality on this prompt mix.**
3. **Above 0.80 the scorer contributes nothing at all** — resolution falls to exactly the 34.3% the deterministic patterns already provide. There is a real threshold above which the entire weighted-scoring apparatus is dead weight, and it is 0.80.

---

## Where the local path works and where it does not

| group | prompts | resolved locally |
|---|---|---|
| greeting | 20 | **100%** |
| identity | 10 | **100%** |
| memory | 5 | **100%** |
| structured | 4 | 25% |
| multi-step | 12 | 8% |
| single-edit | 10 | **0%** |
| specified-task | 10 | **0%** |
| factual | 10 | **0%** |
| state-query, ambiguous, creative, destructive, reasoning, long-simple | 18 | **0%** |

The pattern is clean: **the local path handles conversational furniture and nothing else.** Every category involving actual work — a single well-specified file edit, a specified coding task, a project-state question — resolves at 0%.

That is precisely inverted from the design intent. The `Medium` class exists to let clear, well-specified code tasks skip the review stages. Those are exactly the `single-edit` and `specified-task` groups, and the scorer fires on none of the twenty.

---

## Two structural consequences

**1. `Trivial` is unreachable without paying for the LLM.** Despite its name, `quick_classify_trivial` returns `PromptClass::Simple` in every branch. All 10 arithmetic and general-fact prompts deferred to the LLM classifier — 100%. So a "what is 2+2" pays for a classifier call *in order to be told* the cheap model can answer it. The cost saving the `Trivial` class exists to capture is partly consumed by the call that identifies it.

Five of those ten sit at confidence **0.7311**, 0.019 below the gate.

**2. The dead keyword.** `classifier.rs:848` lists `"O("` in the CONSTRAINTS keyword set; `:851` matches that set against the lowercased string. A capital `O` cannot appear in a lowercased haystack, so **the keyword has never matched in production**. Verified against three realistic complexity-notation prompts. Weight 0.04 with the count capped at 1, so the practical effect is small, but the signal the dimension was written to capture does not exist. Found by transcription, not by reading.

---

## What this does not establish

- **One labeller, no second coder.** The gold labels are a single considered view by the person analysing the results. Paper 01 measured what that is worth: independent coding dropped its motive column to κ 0.34. The labelling rule is published so any row can be disputed, but no inter-rater check has been run, and the accuracy figures inherit that weakness.
- **One prompt set, chosen by the author.** 102 prompts weighted towards conversational traffic — 35 of 102 are greetings, identity or memory. A set drawn from real logs would give a different resolution rate, almost certainly lower, since real traffic to a coding assistant contains fewer greetings. **The 36.3% is probably an overestimate of the local-resolution rate on production traffic.**
- **The port is not differentially tested against a compiled Rust build.** Axium was not compiled. Parity was established by transcription and reasoning, and the deviations are recorded.
- **Nothing about accuracy above the local path.** The LLM classifier's own accuracy is untested — that requires API calls and is a separate job.

## Fair statement of the finding

The claim in the code comment is wrong on this set, by a wide margin, and the reason is a specific and fixable interaction between the boundary spacing and the confidence function. The shipped confidence gate, by contrast, is well chosen — measurably close to the best available trade. And the deterministic keyword path, which is the least sophisticated thing in the file, is carrying the feature.