Human in the loop. Where people stay, and why.
Human-in-the-loop means a person checks or approves specific steps of an AI workflow before they take effect. The right amount of review depends on how reversible the action is, how costly a mistake would be, and how honestly the system can say it is unsure. This guide gives five review patterns and a simple table for choosing between them.
What “in the loop” actually means
Three positions are worth telling apart. In the loop: a person approves each result before it takes effect. On the loop: results take effect, and a person watches samples and exceptions with the power to stop the workflow. Out of the loop: the workflow runs and nobody looks until something breaks. Most business workflows should start in the loop, earn their way to on the loop with evidence, and never be out of the loop where money, records, or customers are affected.
NIST’s AI Risk Management Framework frames this as governance: decide who is accountable, map where the risks are, measure them, and manage them with controls that fit. Review points are the most practical control there is.
Five review patterns
- 1. Propose and approve
- The workflow prepares a draft, a record, or a recommendation. A named person accepts, edits, or rejects it. Nothing happens without the click. Right for sending, filing, paying, and anything customer-facing.
- 2. Exception routing
- Clear cases pass by rule; anything uncertain, conflicting, or outside scope goes to a queue with the source and the reason attached. Right for high-volume checks where most items are routine.
- 3. Sample review
- Results take effect, and a person reviews a random or risk-weighted sample every day or week. Right for reversible, low-cost actions once the test set and a period of full review have passed.
- 4. Confidence gates
- The workflow reports how sure it is, based on evidence rather than a feeling: sources found, fields matched, rules satisfied. Below a threshold, the item goes to review. Thresholds are tuned on the test set, not guessed.
- 5. Audit-only
- Everything is logged with sources and decisions, and a person can investigate any item later. Right only for actions that are cheap to undo and carry no external effect. Never the only control for money or customer data.
Choose by risk and reversibility
Two questions settle most cases: can the action be undone cheaply, and how much does a mistake cost before anyone notices?
| Action | Low cost of a mistake | High cost of a mistake |
|---|---|---|
| Reversible | Audit-only, moving to sample review if volume grows | Sample review with exception routing |
| Irreversible | Propose and approve, one reviewer | Propose and approve, with a second person for amounts or changes above a threshold |
“Irreversible” is broader than it sounds. An email cannot be unsent. A payment can be clawed back, but not cheaply. A merged customer record loses history. When in doubt, treat the action as irreversible until the owner says otherwise in writing.
Three examples
- Invoices. The workflow extracts fields, matches the purchase order, and flags likely duplicates. Finance approves posting; payment authorization stays with the existing controls. See invoice processing.
- Support tickets. The workflow classifies, gathers the account and policy, and drafts a reply with the source. The agent sends. Safety, legal, and account-security issues route to a specialist without a draft. See ticket triage.
- Owner updates. The workflow drafts a property owner update from work orders and marks anything unconfirmed, such as a completed order that a later note contradicts. The property manager confirms before it goes out. See owner update drafts.
Make review cheap, or it will not happen
Review that takes longer than doing the task by hand gets skipped. Design the review screen for a decision in seconds:
- Show the source next to the proposed output, not in another tab.
- Show what changed and why, as a diff or a short reason, not a wall of text.
- Offer one-click decisions: accept, edit, reject, escalate.
- Batch similar items so a reviewer can clear a queue in one sitting.
- Record every decision so the test set grows from real corrections.
Measure the review burden
Three numbers tell you whether the loop is healthy. Review minutes per accepted task shows whether the workflow saves time or moves it. Override rate (how often the reviewer changes the output) shows whether the workflow is ready for lighter review. Escalation rate shows how often the stop rules fire; a rate near zero usually means the rules are too loose, not that the world is tidy. Track all three in the same record as cost per accepted task.
When review becomes theater
If a reviewer approves two hundred items in ten minutes, nobody is reviewing. The fix is not to scold the reviewer. It is to move routine items to sample review, route the uncertain ones with evidence attached, and make the screen show the one thing that needs a human judgment. Review should feel like deciding, not like clicking.
Sources
External references used in this guide. Figures are quoted with their original baseline; none is a measurement of our own work.