Scope a first AI agent pilot. Twelve decisions, in order.
A first AI agent pilot works when it is small enough to test and real enough to matter: one repeated task, one owner, agreed inputs and permissions, written stop rules, and a test set you can rerun. This checklist walks through the twelve decisions in the order we make them with clients, and ends with the decision that matters most: whether to expand.
Why most pilots stall
Pilots rarely fail because the model was weak. They stall because nobody agreed what “done” meant, the sources were not ready, or the person who had to approve the result was never in the room. MIT NANDA’s 2025 State of AI in Business report found that 95% of the organizations it studied had seen no measurable profit-and-loss impact from their generative AI pilots. Gartner predicts that more than 40% of agentic AI projects will be canceled by the end of 2027. The fix is not a bigger pilot. It is a smaller one with sharper edges.
The twelve decisions
Pick one task
Choose a repeated job with accessible examples and a clear output. Reporting, intake, follow-up, document checks, and ticket triage are common first tasks. Avoid “clean everything” and “an assistant for the whole company.”
Name the owner and the reviewer
One person owns the workflow and can say whether a result is acceptable. One person (often the same) reviews outputs during the pilot. If nobody can spare the time, the pilot is not ready.
Write the trigger and the finish
What starts the task (an email arrives, a form is submitted, the month ends) and what a finished result looks like (a record, a draft, a checklist with sources). One sentence each.
List the inputs and sources, with versions
Which systems, folders, and documents the agent may read. Who keeps them current. How a removed or restricted document is handled. Redacted or synthetic examples are enough at this stage.
Decide what it may read, propose, and do
A permission matrix per system: read, propose, write. Writes that matter (send, file, pay, delete, change customer data) wait for a person. Least privilege for every connection.
Write the stop rules
Missing data, conflicting records, restricted content, an amount above a threshold, anything irreversible. A stop rule says what the agent does instead: mark what is missing and route to whom.
Collect twenty to thirty test cases
Ordinary cases and the awkward ones: a duplicate, a stale policy, an unreadable page, a request outside scope. Each with the expected result. This set is rerun after every change.
Agree the acceptance criteria
What must be correct, what may be uncertain, and what must stop the workflow. Written with the owner before the build. The acceptance method and worksheet show the format.
Estimate cost per accepted task
Model calls, tools, infrastructure, retries, and review time, divided by accepted results. Set a ceiling now so the pilot can fail on cost as well as on quality. The calculator does the arithmetic.
Decide the review routine
Who looks at what, how often, and what the review screen shows: the source, the proposed output, the reason, one-click decisions. The human-in-the-loop guide gives five patterns.
Plan the handoff
Who runs it after the pilot: credentials, monitoring, alerts, the operating record, and the route for changes. A pilot without a handoff plan becomes a permanent side project.
Define the expansion decision
Before the pilot starts, write what evidence would justify expanding it, revising it, or stopping. Numbers from the test set and the cost line, not impressions from a demo.
Put it on one page
The twelve decisions fit on one page. Our project brief template asks the same questions in the same order; the acceptance worksheet holds the test cases and the decision. Both are plain Markdown you can paste into your own documents.
What we leave out on purpose
A pilot is not the place for a custom model, a company-wide rollout, a new data platform, or an agent that acts without review. Each of those may come later, with evidence. Leaving them out is what makes the pilot cheap enough to run and honest enough to judge.
Sources
External references used in this guide. Figures are quoted with their original baseline; none is a measurement of our own work.