The demo always works
Forty vendors will show you an agent booking a meeting. The demo takes ninety seconds, the API answers instantly, the data is clean, and everyone in the room nods. I have built these demos. They are easy.
The question that decides whether you have a system or a toy is different: what does the agent do at 3am, when the API times out, the response comes back malformed, and nobody is watching?
The four failures that always come
Run any agent long enough in production and you meet the same four events. Not might meet - will meet.
- The tool call fails. Every external API fails eventually. The agent's answer to that - retry, switch approach, escalate - has to be designed before launch, not discovered after.
- The data shifts. A supplier renames a column. A form starts sending empty strings. A script ships wrong answers quietly; an agent has to notice that what came back makes no sense.
- The model changes. A model update can silently change behaviour. Without evals - automated checks that the system still does what it did last week - your customers become your test suite.
- The loop runs away. An agent in a loop is a credit card in a loop. Cost ceilings live in code, not in hope.
What the 3am checklist looks like
When I take an agent to production, the launch list is mostly about failure: what gets retried and how many times, what gets escalated to a human and through which channel, what the daily token budget is and what happens at 80% of it, and which eval has to go red before a deploy is blocked.
None of that appears in a demo, because none of it is visible when everything works. It is the part you are actually paying for.
How to use this
Next time someone shows you an agent, let the demo finish and ask one question: "show me what happened the last time it failed." A real production system has an answer - logs, a retry trail, an escalation. A demo has a pause.
I run agents in production and that question has answers. If you have a process that should run itself, bring me the process and the number it burns.