← Alle Beiträge

Jev vs LLMs: what the first evaluations show on accuracy, cost and speed

Jev, the decision model TypeSafe AI released on 15 September 2026, is about as accurate as mid-priced large language models on simple business decisions, a few points behind the best ones, and far cheaper and faster per decision. After the first ten days of evaluations the honest summary is this: it wins on cost and speed, loses on hard, numeric and fine-grained questions, and works best as the first step of a cascade that sends its unsure answers to a larger model. The headline speed and price multipliers describe one step, not a whole system.

What is Jev, in one paragraph?

Jev is a decision model, not a chatbot. You send it a piece of data and a set of typed questions, each with options you define, and it returns one option per question with a probability for every option. It writes no text, so it cannot answer outside your options. It costs 0.042 US dollars per million input tokens, output is free, and TypeSafe quotes about a tenth of a second of server time per call (TypeSafe, launch post). How typed decisions work, and how to design them, is covered in Jev and structured outputs.

How accurate is Jev compared with large language models?

On TypeSafe's own workflow evaluation, Jev scored 67.8%, level with Claude Sonnet 5 at 67.8% and a mid-tier OpenAI model at 67.9%, and behind the top of the table, Claude Opus 5 at 73.1% and OpenAI's strongest entry at 74.1% (TypeSafe, workflow evaluation). Read the reference labels before the ranking: they are the average answer of two frontier models at high reasoning, so the test measures agreement with those models, not human truth. The per-workflow numbers say more than the average:

  • Customer service: 76.0% against 78.3% for the best model. Nearly level.
  • Security incidents: 61.7% against 66.2%. Close.
  • Invoice processing: 61.8% against 79.1%. Far behind, on the most numeric workflow, even though the sums and dates were computed in code rather than asked.

The first independent paper points the same way. Across 18 social science labelling tasks and 19 language models, Jev trailed the best model on 14 of 15 evaluation tasks, by a median of 11.6 macro-F1 points, at a median cost 44 times lower (Ibrahim et al., arXiv 2609.24574).

How much cheaper and faster is it?

Per decision, by a wide margin. In the vendor's evaluation Jev cost 0.0004 US dollars and 0.4 seconds per case, against 0.1174 dollars and 78.1 seconds for Claude Sonnet 5 at the same accuracy. An independent phishing benchmark measured 0.038 dollars per thousand emails for Jev against 0.462 for Claude Haiku 4.5, and a median latency of 239 milliseconds from France, of which 163 were network, because the service answers from the United States (jev-phishing-bench).

At small volume the difference hardly matters. By my arithmetic, a Greek business sorting 10,000 support emails a month at about 600 tokens each would pay roughly 25 US cents a month on Jev and about 6.50 dollars on Claude Haiku 4.5. Both are pocket change, so for that business the choice turns on accuracy, language and fit, not price. At millions of decisions a month, the same ratio is the budget. The same arithmetic for language models is in AI text classification.

Do the headline speed and price gains survive a real system?

Only for the step you replace. The "193.6 times faster, 444.6 times cheaper" figures on TypeSafe's homepage roughly match the most favourable pairs in its own test, 78.1 against 0.4 seconds and 0.1761 against 0.0004 dollars, and TypeSafe itself calls them the higher end of real-world gains. A real pipeline also reads data, calls tools, waits on people and writes results, and none of that gets faster. Before you count on a saving, time the whole task end to end with the decision step swapped, not the decision step alone.

Speed has one more alternative worth knowing. When the same small set of questions is asked thousands of times a day, a small model trained on your own examples can be faster still: the 5-million-parameter router of BSLM, which I trained from scratch to recognise 35 request types in Greek and English, answers in a median 3.77 milliseconds on an ordinary processor, with no network in the way. The price is building and maintaining it, as covered in small language models.

When should you use Jev, a large model, or both?

Use both, in a cascade, whenever the decision matters. Jev answers first; confident answers are accepted, and the rest go to a large model or a person. The two papers that tested this found it works: sending low-confidence items to a language model matched or beat the language model alone at a quarter to half of its cost (arXiv 2609.24574), and a judge that accepted Jev's confident verdicts and escalated the rest kept 99% of the strong judge's accuracy, with Jev alone within 3 points of it at 0.36% of its fee (Li et al., arXiv 2609.26550).

Two details decide whether a cascade works for you. Set the threshold per question on a sample you labelled yourself, because Jev is overconfident on some question types and underconfident on others, and the author of the first calibration study advises against thresholding on its separate confidence field at all (jev-ood-calibration). And keep the large model for what Jev cannot do: writing, multi-step reasoning, numbers, dates and questions whose answer is not in the text. The general pattern of sending easy requests to a cheap model is covered in LLM routing, and grading with a model in LLM as a judge.

Does Jev work in Greek?

Untested in public so far. TypeSafe states that English is its primary and most accurate language and that other languages work with lower accuracy, and I have not found a published Greek benchmark as of September 2026. Treat Greek as unproven: label 200 real Greek examples, including messages typed without accents and in Greeklish, and compare Jev with the model you use today before it decides anything. Why Greek behaves differently in AI models is explained in Greek in AI models.

Frequently asked questions

Is Jev better than ChatGPT?

It does a different job. ChatGPT and the models behind it write, reason and hold a conversation. Jev only picks among options you define, which makes it cheaper, faster and impossible to answer outside your list, and less accurate than the best language models on hard decisions.

Can Jev replace the language model in my AI system?

Only for the steps that are pure decisions: routing, tagging, checking, filtering. Anything that writes a reply, a summary or a document still needs a language model, and so do decisions that need numbers or several steps of reasoning.

Can a business in Greece use Jev today?

Yes, with care. TypeSafe paused new signups on 22 September 2026 because of demand, and Jev is also offered through OpenRouter in beta (OpenRouter, Jev guide). TypeSafe says it does not train on customer requests. Before sending personal data, read its data terms as you would for any provider, as described in GDPR and AI.

Choosing the right model for each step is part of how I build AI agents for business. If your team makes the same decision hundreds of times a day, send me twenty examples and the options, and I will tell you whether a decision model, a large model or a cascade fits.