The exception rate decides whether operational AI pays

There is a number that decides the economics of an operational system, and it is usually not the one in the business case. It is not accuracy, it is not time saved per task, and it is certainly not the volume of alerts generated.

It is the proportion of cases the system handles without a person needing to intervene. Everything else follows from it.

Why accuracy misleads

A system that is 90 per cent accurate sounds good. In an operational setting it can be worse than useless, and the reason is simple: the user does not know which 90 per cent.

If one flag in ten is wrong and there is no way to tell which, every flag has to be checked. The system has not removed the work. It has added a screen to look at before doing the work that was already there.

This is why evidence matters so much more than scoring. A system at 85 per cent accuracy where the reasoning is visible can be more valuable than one at 95 per cent where it is not, because visible reasoning lets a person dismiss a false positive in seconds instead of minutes.

The arithmetic

Take a planner who spends roughly two hours a day working out which orders are in trouble and what to do about them. Call it ten hours a week of genuinely skilled work.

A system that handles 90 per cent of that assessment reliably leaves one hour a week of real exceptions. That is a transformation of the role.

A system that handles 60 per cent does not leave four hours. It leaves four hours of exceptions plus the time spent determining which cases were exceptions, which is most of the original assessment work. The saving is far smaller than the percentage implies, and that is before the cost of building and running it.

Between roughly 50 and 85 per cent, an operational system often costs more than it saves. The curve is not linear and the middle is a bad place to stop.

Why the middle is so easy to land in

The first 60 per cent of any operational problem is the clean, well-structured part: the purchase order is late in the system, the field is populated, the logic is unambiguous. It is fast to build and it demos well.

The remaining 40 per cent is where the real operation lives. The supplier confirmed by email rather than in the system. The part has two codes after a migration. The customer has a verbal agreement about split shipments that is written nowhere. A quantity was fixed on the floor and nobody updated the record.

That work is unglamorous and it is the entire difference between a pilot and a system. Projects that stop when the demo looks good stop precisely at the point where the economics have not yet turned.

Narrow the scope rather than accept the middle

If a high completion rate is not achievable across the whole problem, the right response is usually to make the problem smaller, not to accept a mediocre rate across all of it.

One product family, one site, one customer segment, handled at 90 per cent, is worth more than everything handled at 60. It also gives you a clean read on whether the approach works, and a credible basis for extending it.

What to measure from the first week

• The proportion of cases that complete with no human involvement at all.
• For the remainder, why a human was needed: missing data, ambiguous rule, genuine judgement call, or system error.
• How long a person spends on an exception, which tells you whether the evidence is doing its job.
• The false positive rate specifically, because that is what destroys trust fastest.

The second of those is the most useful and the most often skipped. Exceptions caused by missing data are a data problem with a known fix. Exceptions caused by genuine judgement are the ones you want to keep with people permanently. Grouping them together as the failure rate hides the distinction that tells you what to do next.

The uncomfortable version

If you cannot see a credible path to a high completion rate on a scoped problem, the honest conclusion is that the problem is not ready, or the scope is wrong, or the data underneath needs work first.

That is a better answer to reach during discovery than eighteen months into a deployment that technically works and has not changed anybody's week.