How to tell if an AI pilot worked: 6 metrics to set before you start

Six months after a company rolls out AI, someone in leadership asks a simple question: "What has improved?"

Too often, nobody can answer. Not because nothing improved, but because nobody recorded where things stood before, or agreed what improvement would look like. The pilot ends with opinions, a few enthusiastic users and a slide full of projections.

A projection is not a result. If you want a pilot to end with a decision instead of a debate, the work starts before the AI is switched on. Here are six metrics to agree on, in writing, before day one.

1. Time per case

Pick one process and define its unit of work: a document reviewed, a ticket resolved, a contract checked, an email triaged. Then measure how long one case takes today, end to end, including waiting time and rework.

This is your baseline. Without it, any later claim of "it's faster now" is a feeling. Measure it over enough real cases to be representative, and record how you measured it, so you can measure the same way after the pilot.

2. Quality of the output

Speed is worthless if the output gets worse. Define what a good result looks like for this process: error rate, completeness, compliance with internal rules, or the share of cases that need to be redone.

Make the people who do the work today part of this definition. They know where errors hide. During the pilot, the same reviewers should judge both the old output and the AI-assisted output, using the same criteria.

3. Human review effort

AI should propose; a responsible person should validate. That review takes time, and it belongs in the numbers.

Track how long reviewers spend checking and correcting AI output, and how often they reject it. A use case where the AI drafts in seconds but reviewers spend longer fixing it than doing the work themselves has not improved anything. This metric also tells you whether trust is growing: if review effort drops over the pilot while quality holds, the process is maturing.

4. Full cost per month

Licences are only part of the picture. Add usage costs for models and tools, integration and maintenance effort, and the time of the people involved. Express it per month and, where possible, per case.

This lets you compare honestly: the cost of the current process against the cost of the new one, and one model or tool against another. Models and tools change every quarter, so a clear cost per case also makes it easier to switch when something better or cheaper arrives.

5. Adoption by the team

A pilot that works only when the project team runs it is not ready to scale. Measure whether the people who own the process actually use the new way of working, in their normal day, without being pushed.

Useful signals: the share of eligible cases that go through the new process, whether people return to the old way, and what they say when asked what is getting in the way. Low adoption is not a failure in itself; it is information about training, roles or the design of the solution.

6. The decision criterion

This is the metric most pilots skip, and the one that matters most. Before starting, agree and sign a success criterion: what result, on which of the metrics above, would justify scaling, adjusting or stopping.

For example: "If time per case drops and quality stays at or above baseline, with review effort falling over the last month, we scale to the next team. If quality drops, we stop." Writing down what would make you stop is what makes the pilot credible. Pilots with exit criteria are the ones that earn permission to grow.

Keep four kinds of evidence apart

When the pilot ends, resist the temptation to merge everything into one impressive number. Keep four things in separate columns:

• Projection: what could happen if the volume and usage assumptions hold.
• Operational result: what the recorded volume, quality and times actually show.
• Released capacity: time freed for other work, with an explicit valuation.
• Financial saving: an amount supported by sources and confirmed by the business.

Released time does not become a financial saving until it is evidenced. Each level needs its own proof, and mixing them is how pilots lose credibility with finance and leadership.

A simple structure for the pilot

A focused pilot on one real use case typically runs for about 90 days. That is an indicative timeline, not a promise: it depends on scope, data, permissions and the team's availability.

1. Choose the case. A process that matters, an owner who can make decisions, and agreed success criteria.
2. Measure the starting point. Time, quality and cost of the current process.
3. Improve it with AI. Test models and tools on the real case and compare results.
4. Validate with the team. People review and approve; measure before and after.
5. Decide with data. Scale, adjust or stop.

Why this matters beyond one pilot

Measurement is not a reporting exercise at the end. It is one of the foundations of an enterprise AI capability, alongside AI engineering, models and tools, governance and security, and people and skills. When every use case is measured the same way, the second and third pilots get easier: you can compare them, see where AI pays off, and stop what doesn't, without starting from scratch each time a new model appears.

The question "What has improved?" will come. The companies that set these six metrics before they start are the ones that will be able to answer it.