Most AI pilots stall because they were built to impress in a demo, not to run on real data against a measured target. The projects that reach production share a pattern: one narrow, frequent task, a success number fixed before anything is built, a test set of real cases including the awkward ones, a person approving the output until the record justifies otherwise, and a running cost and an owner planned from the first week. The failure rate is high enough to plan for: Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027. The ones that survive look deliberately boring: one workflow, one number, one owner.
Why do most AI pilots never reach production?
Most AI pilots never reach production because nobody defined what production would require. Gartner attributes its forecast of cancelled agentic AI projects to escalating costs, unclear business value and inadequate risk controls, and adds that many projects are early experiments driven by hype (Gartner, 25 June 2025). An MIT report from 2025, based on interviews, a survey and 300 public deployments, found that only about 5% of AI pilots achieved rapid revenue gains while the rest stalled with little measurable impact, and that tools bought from specialised vendors or built with partners succeeded about 67% of the time, against about a third for internal builds (Fortune, 18 August 2025). Its definition of success is narrow, but the causes it lists match what I see: demo data instead of real data, no baseline, integration left for later, and no one who owns the result. Greek businesses start from further behind, as the figures in AI adoption in Greece show, which makes a first project that works more valuable, not less.
How do you choose the right first AI project?
You choose the first project by where the gain is measurable and the risk is small. Six tests:
- the task happens often, daily or hourly, so a small gain adds up;
- it is narrow, with inputs and outputs you can list;
- the data it needs exists and you can reach it;
- a mistake can be caught and reversed before it costs money;
- today's cost, time or error rate is known, so improvement can be shown;
- one person owns the workflow and wants it fixed.
The research adds one more filter. Across the studies I reviewed, gains concentrate where the task or the worker is least established: support agents with a generative assistant resolved 14% more issues per hour on average and 34% more among the least experienced, while experienced developers in mature code bases were measured 19% slower (The adoption gap). Start where the work is repetitive and the knowledge is scattered, not where your best people already excel. Typical candidates are listed in AI use cases for small businesses. First projects covered in this series include customer support for an e-shop, a knowledge base on your own documents, checking the books against myDATA and preparing a shop for AI agents. Most of them need one agent, not several, as explained in multi-agent systems, and a narrow, high-volume task may not need a large model at all, as covered in small language models.
What should you measure before you build?
Before you build, measure the current process and fix the target. Record how long the task takes today, what it costs and how often it goes wrong. Then build the test set: 30 to 100 real cases from the workflow, graded the same way every time, by code where possible and by a tested model judge for the rest, as described in LLM evals and LLM as a judge. Run a do-nothing control through it, so you know what score inaction earns, and write down the number that means success and the number that means stop. Measure every change against it, because changes do not always help. In my own cost study, two of six version changes made a production agent worse (What an agent actually costs).
How do you measure the cost of an AI pilot honestly?
You measure cost per completed unit of work, never per run or per month alone. In the same study, the best-scoring version of the agent appeared to cost 3.3 times the one before it, which looked like a clear case of diminishing returns. It was not: that version ran every task three times, and per execution it cost 1.10 times as much and ran faster. Counting runs as tasks would have killed a good change. Compare versions on cost per completed task, including failed attempts and retries, and read the provider's price list for the tokens you actually send, as explained in what an AI agent costs to run.
What does it take to run an AI system in production?
Running an AI system in production takes the same discipline as any other software, plus a few controls specific to models:
- Integration in stages: read-only access first, write access later, and only through actions that can be logged and reversed.
- Human approval for anything irreversible, until the measured record justifies removing it, as described in human in the loop.
- Monitoring: every request, answer, tool call, cost and hand-off logged, with a sample reviewed each week.
- Re-testing on model changes: providers update and retire models, so the test set runs again before any switch, as covered in how to choose an AI model.
- Security and data rules: untrusted input handled as data, as covered in prompt injection, personal data minimised under GDPR, and the AI Act duties met, including staff training under Article 4, as covered in AI literacy under Article 4.
- A fallback: when the model or the provider fails, the work goes to a person, not into a void.
How long should an AI pilot take?
An AI pilot should take weeks, not quarters, and end with a decision. A structure that works: two weeks to collect real data, measure the baseline and build the test set; two weeks to build the first version, running in draft mode; two to four weeks inside the real workflow, with people approving every output while the numbers are collected; then a decision to scale, fix or stop. A pilot without an end date is how projects drift for a year without ever being judged.
When should you stop an AI pilot?
You should stop an AI pilot when it misses the stop number you wrote down at the start. That usually means it does not beat the baseline on the test set after a fair number of iterations, the cost per completed task exceeds the value of the task, or it makes a type of error the business cannot accept. Stopping on evidence is a result, not a failure: the business learns what the task needs, often better data or a narrower scope, for a fraction of the cost of a project that runs for a year on hope.
Frequently asked questions
How much does an AI pilot cost?
Mostly people's time: data preparation, integration, testing and supervision. The model usage is usually the smaller part. Fix a budget and a stop number before starting, so the pilot cannot quietly grow.
Do we need a data scientist for an AI pilot?
For most business pilots, no. You need someone who owns the workflow and knows what a good result looks like, and an engineer who can connect the systems and build the tests.
What if the pilot works but people do not use it?
Then it has not reached production. Involve the people who do the work from the first week, train them on what the system does and does not do, and measure usage alongside accuracy.
If you have an AI pilot that stalled, or one you are about to start, send me the task, the data it needs and the number that would mean success. If the number is missing, that is the first thing to fix. How I take systems from pilot to production is on the agentic AI page.