Building trust in operational AI

Operational AI fails adoption more often than it fails technically. The model works, the integration works, and six weeks after go-live the planners are back in the spreadsheet they built themselves.

That outcome is predictable, and it is usually set in motion by design decisions made long before deployment.

Trust is earned at a specific moment

There is a moment, early in every deployment, when the system tells an experienced person something they do not believe. That moment decides everything that follows.

If they can check the claim in under a minute, and the system turns out to be right, it gains standing. If they can check it and the system is wrong, but they can see exactly why it was wrong, it survives: the logic was visible and the input was bad. If they cannot check it at all, it does not matter whether it was right. They will assume it was wrong, and they will be unwilling to act on it next time.

An operational system is not judged on its accuracy. It is judged on whether its reasoning can be inspected by the person being asked to act.

Start with rules you can explain

There is a strong pull toward sophisticated modelling early, because it demonstrates capability. We think that is the wrong order for operational work.

Explainable rules have a property that matters more than accuracy at the start: when they are wrong, a domain expert can say precisely why, and the rule improves. A learned model that is wrong produces a conversation about retraining that the operations team cannot participate in.

There is a second benefit. Writing the rules forces the business to state how the process actually works, which is frequently the first time anyone has. A surprising amount of value in these projects comes from that exercise alone, before any software ships.

What generative AI should and should not do here

Language models are genuinely useful in operational intelligence, and the useful jobs are narrower than the marketing suggests.

• Summarising evidence that has already been gathered by deterministic means, so a person reads a paragraph instead of six records.
• Explaining a chain of dependencies in plain language for someone who did not build the system.
• Helping a user interrogate a situation conversationally rather than through filters.
• Drafting the communication that follows a decision, for a person to review and send.

What it should not do is originate operational facts. If a model states that a purchase order is late, that has to come from a record, not from inference. The distinction between retrieving a fact and generating a plausible one is the whole ballgame in an operational context, and it is invisible to the reader unless the system is built to make it visible.

Read-only is a feature, not a limitation

A system with read-only access to source systems can do almost all of this. It can detect, explain, prioritise, propose and record. What it cannot do is change anything on its own, and at the start that is exactly right.

It removes an entire category of risk from the conversation with IT, it shortens the security review considerably, and it means the worst case of a bug is a wrong recommendation rather than a wrong action already taken in the ERP.

Write access can come later, narrowly, for specific actions, once the system has a track record. Starting there, on the argument that it is more efficient, trades a large amount of trust for a small amount of convenience.

Permissions are an operational question, not an IT one

Who can see which orders, who can approve an expedite, who can commit to a new date with a customer: these are organisational facts that exist already, usually informally.

Systems that ignore them cause trouble quickly. A planner who sees commercial margin data they were never meant to see is a problem. A system that lets anyone approve a recovery action that costs money is a bigger one. These decisions belong to the operations leadership, not to whoever configures roles at the end of the build.

Record the decision, not just the alert

Most operational systems log what they detected. Far fewer log what a human then decided and what happened as a result.

That record is where the long-term value accumulates. It answers whether the warnings were useful, whether the interventions worked, and whether the system is earning its place. It also means that when something goes badly wrong, there is a clear account of what was known, when, and who decided what, which matters more than any dashboard the day it is needed.

What we are committing to

For our own pilots the starting position is read-only access to approved source systems, evidence attached to every risk with its source record, permissions that mirror how the business is actually run, human approval for anything consequential, and a recorded outcome against every decision.

None of that is novel. It is simply the set of constraints that make an operational system worth trusting, and they are much easier to build in at the start than to retrofit after the first time somebody stops believing it.