Five questions between the business case and the build.

The problem

Enterprise AI programmes rarely start without a business case. There is an objective, a sponsor, an expected return, and a proposal that survives procurement and a steering committee.

Then the work moves closer to the technology, and something gets lost.

“Reduce the time lawyers spend searching previous matters” becomes “build a knowledge assistant.” “Improve complaint handling” becomes “summarise complaints with an LLM.” “Reduce manual review” becomes “introduce an AI triage model.”

The technology may be sensible. The problem is that a business objective has been replaced by a capability description. The objective has not disappeared, but it is no longer precise enough to determine what should be built.

That gap can be written down. It is five questions long.

The five questions

Question 1 and the eventual value check are the same measurement taken twice. Capture the baseline before the build, or you will only ever be able to measure activity inside the system: more users, more prompts, more documents summarised.

Questions 3 and 4 determine whether the proposed system has something real to work with. They are also the two most often left implicit.

Question 2: a decision, not a direction

An outcome describes where you want to go. A decision can be inspected.

Take a professional services team that wants to reduce time spent on knowledge retrieval. That could mean at least four different systems:

• find the correct previous document
• identify comparable matters
• answer a question by synthesising evidence across matters
• decide whether the archive contains enough evidence to support a proposition at all

They share an objective. They have different data requirements, different evaluation methods, different risk profiles and different economics. They sound like variations on one AI product. They are not.

So the useful starting question is not what the model can do. It is which specific piece of work is changing.

Question 4: good enough needs a comparator

Accuracy, precision, hallucination rate and benchmark scores only become business decisions when something sits on the other side of the comparison.

The Epic Sepsis Model is the clearest illustration. External validation at Michigan Medicine found clinicians would have needed to evaluate 109 patients to identify one additional case of sepsis that their existing process missed.

The interesting question stopped being whether the model could predict sepsis. It became whether that prediction improved the workflow enough to justify the extra review. Those are very different tests, and only the second one is a business decision.

This is why acceptance criteria have to be set while failure is still cheap. Evaluation designed after a system exists has a structural temptation: to find the metric on which that system looks acceptable.

A worked example, and it did not go the way I wanted.

Building a legal retrieval system, I designed a control that would use retrieval dispersion to decide when evidence was too thin to answer on.

Across 16 calibration queries in four intent categories, two dispersion measures showed near monotonic separation between weak, ordinary, strong and deliberately sparse evidence.

The signal looked encouraging. But there was no defensible cut point. Any threshold I could justify for one intent category was wrong for another, and picking one anyway would have converted a promising pattern into a certainty the data did not support.

So the gate was removed. The requirement did not disappear, it moved: the interface now exposes coverage information directly, and deterministic checks handle the cases where a genuine gap can be identified.

An evaluation framework should be capable of killing the thing being evaluated. Otherwise it is mostly a way of certifying decisions already made.

The chain is the deliverable

None of these questions is novel on its own. The difficulty is that organisations distribute ownership of them.

Strategy owns the objective. Product owns requirements. Data teams discover the evidence. Engineering builds. Risk asks about thresholds. Operations inherits the review step. Finance asks about value after deployment.

Every group can do its part competently while the chain between them stays implicit.

The test worth applying is not whether an organisation has an AI strategy, a pilot in production, or a growing number of users. It is whether it can trace the system back to the decision it was meant to improve, say what standard that system has to meet, and name the evidence that would make it stop, redesign or scale.

If that chain is missing, the problem started before the model was built.