The AI pilot acceptance checklist: owner, baseline and the decision to go to production
A demo shows a model can answer a question. A pilot has to show whether a workflow is better with it. This is the checklist we fill in before a pilot starts, whether or not you run it with us.
Why a demo is not a pilot
A demo shows that a model can produce a plausible answer from a plausible question. That is worth almost nothing on its own. A pilot has to show whether one real workflow, done by real people against real data, gets better with the system in it, and whether that improvement is worth the cost of running it.
The difference between the two is not more impressive output. It is six things written down before anyone starts building: an owner, a baseline, acceptance criteria, a rollback plan, a support plan and a named point where someone decides whether the pilot goes to production. Skip any one of them and the pilot will still produce a demo. It just will not produce a decision.
This is the checklist we use before we start a pilot. It is useful whether you run it with us or with anyone else.
The six things every pilot needs
1. Owner
Name one business-unit sponsor and one process owner. The sponsor holds the budget and makes the final call. The process owner knows the workflow today and can say, in plain language, what a correct result looks like.
Write both names down before anything else on this list. A pilot with no named owner drifts until whoever asked for it changes jobs.
2. Baseline
Measure the workflow as it runs today, before any system touches it. How long does one case take. How often does it go wrong, and in what way. How many cases move through it in a normal week. Who does the work now, and what else are they not doing while they do it.
This has to happen before the pilot starts, not after. A baseline measured retroactively is a guess dressed up as a number, and it will not survive a real argument about whether the pilot worked.
3. Acceptance criteria
Write the pass condition down, in specific terms, before the pilot begins. Not "the team likes it" but a stated comparison against the baseline: what result, over what volume of real cases, reviewed by whom, counts as a pass.
Acceptance criteria written after the pilot has already produced results are not acceptance criteria. They are a description of whatever happened, chosen to justify a decision someone already made. Agree them first, in writing, with the sponsor and the process owner both signing off.
4. Rollback
Decide, before the pilot starts, how you return to the current process if it needs to stop. Which system is the record of truth while both run. What happens to work that is in progress. How long it takes to hand the workflow back to the people who ran it before.
A pilot without a tested rollback is a one-way door, and the people using it will treat it that way, whether or not anyone says so out loud.
5. Support
Agree who maintains the pilot while it runs and, separately, who would maintain it if it goes to production. These are not the same question. A pilot can survive on attention from the people who built it. A production system needs an answer in writing: who upgrades it, who reviews what it does, who is on call when it breaks, and what that costs every month.
Settle this before the pilot starts, not at the moment it passes. A team that only asks "who maintains this" after a pilot succeeds is choosing between rushing an answer and letting a working pilot stall for lack of one.
6. Production decision
Set the date, the decision-maker and the three possible outcomes before the pilot begins: stop, extend with a new baseline, or move toward production. Write down who has the authority to make that call, and confirm it is one person or one small group, not a consensus that never quite forms.
A pilot with no scheduled decision point does not end. It just becomes something people stop mentioning in meetings.
What the acceptance test actually checks
Run these checks before you call the pilot finished, whatever the result.
| Check | What it proves |
|---|---|
| Result compared against the written baseline, not against general impressions | The pilot changed the number that mattered, not just the mood in the room |
| A permission test: the system tried to read or act outside its scope and was blocked | Access is actually limited to what was agreed, not just documented that way |
| Every logged exception reviewed by the named reviewer | The cases the system got wrong were seen by a person, not silently dropped |
| Rollback executed once, before go-live, not only described | The way back works when someone actually needs it |
| Operating cost measured against the baseline, not estimated after the fact | The pilot's real cost, including review time, is known before anyone scales it |
| Named sign-off recorded from the sponsor and the process owner | Someone with authority actually accepted the result |
Where the decision goes
| Decision | What happens next |
|---|---|
| Stop | The workflow returns to its baseline process. Document what was learned; it is not wasted if it changes the next attempt. |
| Extend | The pilot continues against a revised baseline and a new date, with the same named owner and sign-off requirement. |
| Move to production | Support terms are finalized in writing, the rollout is staged to more users, and the audit trail and evaluation harness carry forward unchanged. |
Every one of these is a legitimate outcome. A pilot that stops after an honest test has still done its job.
Common mistakes
- Treating the demo as the pilot. A demo run by the people who built it, on data they chose, proves the system can work under ideal conditions. It does not measure the workflow.
- No named owner. Without one person accountable for the decision, the pilot survives on enthusiasm, which runs out before the evidence does.
- Acceptance criteria written after the results come in. This is not measurement. It is a story about what already happened.
- No tested rollback. If going back has never actually been tried, it is not a plan, it is a hope.
- Setting a production date before the pilot has a baseline. The baseline sets the date, not the other way around.
- No named sign-off. A pilot nobody formally accepted is a pilot everybody can later dispute.
Using this with us, or without us
This checklist is useful whatever you decide, including staying exactly where you are. If you want help running a pilot against it, that is the second step of our process, after a discovery call, and you keep everything the pilot produces: the baseline, the logs, the evaluation results and the decision itself.