How to tell whether an AI agent is actually working
Almost every AI agent demos well. That is what demos are for, and it is why they are nearly worthless as evidence. The question that decides whether an agent belongs in your business is not whether it handled the example in the meeting; it is how often it is wrong on your real work, in which direction, and whether you would find out.
Most businesses buying an agent have no way to answer that. This is how to get one, whether we build it or somebody else does.
An agent is a system that is right most of the time. That is not a flaw to be engineered away; it is the nature of the tool. Knowing your actual number, and designing around it, is the whole exercise. Reaching 100% is not available.
The artefact that separates a system from a demo is an evaluation set: 100 to 200 real cases from your own history, deliberately sampled, labeled by someone who does the job, with 20 to 30% held back. Measure accuracy and deferral rate together or the figure is meaningless, pick the confidence threshold with the business rather than for it, and re-run the held-back set on a schedule, because the agent will get worse and you need to notice.
Start with an evaluation set
An evaluation set is a body of real cases from your own history, each with a known correct answer, that you measure the agent against. It is the single artefact that separates a system from a demo, and it is the thing most often missing.
Building one is less work than it sounds.
- Take 100 to 200 real cases. Past support emails, past inquiries, past documents. Whatever the agent will actually see. Fewer than about 80 and your accuracy figure has error bars too wide to act on.
- Sample deliberately, not randomly. A random sample from a real inbox is mostly easy cases, and an agent scoring 97% on easy cases tells you nothing. Include the awkward ones on purpose.
- Have a person label the correct answer. Someone who does the job today. This is the actual cost of the exercise, usually a few hours.
- Hold some back. Keep 20–30% aside and never tune against it. Otherwise you end up optimizing for the test rather than the task, and your measured accuracy quietly stops meaning anything.
Do this before the build, not after. It also has a useful side effect: labeling 150 real cases is the fastest way to discover that your process has four rules nobody wrote down.
Measure the right thing
"Accuracy" is usually the wrong headline number, because it hides the distinction that matters: which kind of wrong.
| Measure | What it tells you | When it is the one to watch |
|---|---|---|
| Accuracy | How often it got the answer right overall | Balanced tasks where both error types cost about the same |
| Precision | When it acts, how often it should have | When acting wrongly is expensive: refunds, escalations, anything sent to a customer |
| Recall | Of the cases it should have caught, how many it did | When missing one is expensive: a complaint, a compliance flag, an urgent ticket |
| Deferral rate | How often it declined and handed to a person | Always. This is your actual automation rate. |
| Cost per case | Model spend per item handled | Before volume scales, not after |
Precision and recall trade against each other, and the right balance is a business decision rather than a technical one. A triage agent that routes urgent tickets should be tuned for recall: better to over-flag than to miss the one that mattered. An agent that issues refunds should be tuned hard for precision and defer anything it is not sure about.
The number nobody asks for
Deferral rate. If an agent is 95% accurate but defers 40% of cases to a person, your real automation rate is 60%, and the business case has to be built on 60%. Vendors quote accuracy on the cases the agent chose to answer. Ask for both, together, or the figure is meaningless.
Confidence thresholds are the actual product decision
Every serious agent can decline. It produces a confidence signal alongside its answer, and you choose the level below which it stops and asks a person instead.
That threshold is a dial between two costs. Set it high and the agent is nearly always right but hands over a lot, so you save less time. Set it low and it handles more but is wrong more often. There is no correct setting in the abstract. It depends entirely on what a mistake costs you versus what a handover costs you.
The practical method: run the evaluation set at several thresholds, tabulate deferral rate against error rate at each, and pick with the business rather than for it. It is a ten-minute conversation once the numbers exist, and an unanswerable argument without them.
Can it say "I don't know"? An agent with no deferral path does not become more accurate. It invents an answer instead, confidently, in the cases where it should have stopped. Those are exactly the cases you most need it to stop on.
It will get worse, and you need to notice
An agent that is right 94% of the time in March is not automatically right 94% of the time in September. Three things drift underneath it.
- Your business changes. New products, new pricing, new policies. The agent is answering from material that is quietly out of date, and it will answer confidently from the old version.
- Your input changes. A new marketing campaign brings a different kind of inquiry. A new supplier sends a document in a format the extraction has not seen.
- The model changes. Providers update and deprecate models. Behavior shifts, usually slightly, occasionally not.
The defense is unglamorous: re-run the held-back evaluation set on a schedule, and again after any change to the prompt, the model or the source material. If accuracy has moved, you find out from a number rather than from a customer. Anything that alters behavior and is not re-measured is a guess.
Log everything, and read some of it
Every exchange stored: input, output, confidence, whether it deferred, what the person did afterwards. Three reasons, and the third is the one people miss.
- Disputes. When a customer says the agent told them something, you can see whether it did.
- Retraining. Real failures are the best possible material for improving it.
- Free labels. Every case a person corrects after a deferral is a labeled example, produced as a by-product of normal work. After a few months this is a better evaluation set than the one you started with.
Then actually read a sample, twenty conversations a month. Aggregate metrics tell you the rate; reading tells you the shape, and the shape is where you find the failure nobody has coded a check for.
What to ask a vendor
Five questions. The answers are short if they have done the work and evasive if they have not.
- "What is the accuracy on our data, and what is the deferral rate at that accuracy?" Both numbers or neither.
- "How large is the evaluation set and who labeled it?" It should be your cases, labeled by someone who knows the job.
- "Show me a case it got wrong." A vendor who cannot produce one has not looked. Everything is wrong sometimes.
- "What happens below the confidence threshold?" There must be a handover path with the conversation attached, not a generic apology.
- "How will we know if it degrades?" A schedule and a held-back set, not "let us know if you notice anything".
What this costs
Building an evaluation set takes a few hours of somebody's time, and it is included in the setup ($3,500 for a chat agent, $5,000 for voice) rather than being an extra. Re-running it is minutes. Ongoing retraining as your material changes is what a Growth or Scale plan is actually for on an agent, more so than on a workflow.
A quote materially below that range is usually missing this work. It is also the work that decides whether the thing still functions in month three, which makes it a strange place to economize.
Already running an agent and unsure how good it is?
Send us a sample of real cases. We will build an evaluation set, measure it properly and tell you the number, including when the answer is that it is fine.
You will talk to Fayez or Kareem. Nobody else.