What the exception queue must answer
An exception is not merely a low score. It may be a missing owner, possible duplicate, unclear service area, do-not-contact signal, conflicting source event, sensitive content, or a request that needs a human decision. A single label such as "manual review" hides the operational reason and gives the next system too little context.
A reviewer opening a held item needs to know why it stopped and who can resolve it. Keep the dated source event beside the hold reason, with the open question and the next allowed action. A confidence score may help prioritize a check, but it cannot establish an event or supply missing permission.
On a narrow screen, scroll the table horizontally.
| Required field | What it proves | Unsafe substitute |
|---|---|---|
| Hold reason | The exact condition that paused automation | Generic "needs review" label |
| Last verified event | The most recent source-backed state | A model's summary of likely activity |
| Named owner | Who can decide, clarify, suppress, or escalate | A team inbox with no assignment rule |
| Open question | What remains unknown | A fabricated completion or inferred intent |
| Next safe action | The bounded step allowed from this state | An unrestricted "continue" command |
Start by grouping the held records by reason and owner, then check their age and missing fields. Counts show where to look; they do not prove that another agent would improve business results. A large "manual review" bucket may be one problem or six unrelated problems hiding under the same label. Split it until each category points to a different decision or repair.
Then read a few source records from every major category. Check whether another reviewer can trace the label to a dated source event and identify the next allowed step. If the reviewer has to invent missing context, the handoff needs work.
Separate uncertainty from failure
Some exceptions are expected. A new service area may need a human fit check. A duplicate can be reconciled after comparing source events. A missing timestamp may require a system owner to decide whether the field is trustworthy. Do not turn every hold into a system error or every unresolved row into a bad lead.
Keep a distinct Unknown state. Use it when the available record cannot support a decision. Show the last verified event and the next source check. Keep the row on hold instead of translating Unknown into permission to proceed.
Inspect the handoff, not just each agent
An agent can produce a plausible classification while leaving the next agent with an unusable record. Agent one may classify a record, but agent two cannot safely draft or route it if the handoff omits the original source, consent or stop signal, event time, ownership, or unresolved question.
Test the event vocabulary as well. A draft is not a sent message. A sent message is not proof of delivery. Delivery is not a customer reply, and a reply is not a booking or closed job. Check the event type in the receiving system, not just the wording of the generated summary.
A minimum handoff contract
- Stable record ID and source system
- Current owner and owner-assigned time
- Last verified event and timestamp
- Specific hold or release reason
- Permission and stop-signal status
- Open question and allowed next action
- Draft, send, delivery, and reply as separate events
- Reviewer decision and rollback reference
Turn vague holds into controlled reason codes
A reason code should describe a condition that an operator can verify. "Low confidence" is rarely enough because it does not say which evidence is missing. "Owner not assigned," "possible duplicate across two source events," "contact permission unknown," and "service area requires review" each lead to a different safe action.
Keep the list small enough to use consistently, but do not force unrelated cases into one bucket. Define who may apply each code, which fields are required, which actions are prohibited while it is active, and what evidence clears it. Retain a free-text note for context, but do not make free text the only operational control.
Review the oldest and fastest-growing reason codes separately. An old item may expose missing ownership; a rapidly growing category may expose an upstream data change. Neither automatically supports adding another agent. Write down whether the delay comes from too much eligible work, a broken integration or an unresolved decision rule. Only the first of those is a capacity problem.
Use a small, redacted mixed sample
Do not begin with a full customer export. Select a small sample with different reasons and remove private details. Include at least one duplicate candidate, one missing-context item, one stop-signal item, and one item that appears ready. The purpose is to see whether the definitions help a human decide, not to estimate response rates or business performance.
For each row, compare the displayed state with the underlying source event. Then ask a reviewer to choose one outcome: release, clarify, suppress, merge, escalate, or remain unknown. Record the reason. When reviewers disagree, keep their reasons. Resolve the missing fact or unclear rule before asking an agent to apply it.
A worked example: five fictional records
The records below are invented to show how the review works. They are not customer data, test results or a claim about how often these problems occur. In this example, the proposed second agent may draft an internal handoff note. It cannot send a message, merge records or change the source system.
On a narrow screen, scroll the example table horizontally.
| Record and evidence | Hold reason | Owner and next action |
|---|---|---|
| Example A: two form events refer to one request, but the IDs differ | Possible duplicate | Operations reviewer compares both source events. Keep both originals; the drafting agent must not merge them. |
| Example B: a request exists, but the service location is blank | Missing context | Service owner checks the permitted source for location. No service-fit conclusion or promise until that fact is known. |
| Example C: a stop request arrived after a draft was prepared | Do not contact | Contact owner confirms the stop in the authoritative system. The earlier draft cannot authorize a send. |
| Example D: a queue label says sent, but the only source event is draft-created | Conflicting status | System owner reconciles the event log. Keep send status unknown; do not retry a potentially completed action. |
| Example E: source, owner and required context are present; no unresolved hold is recorded | Eligible for the limited test | Named reviewer permits one internal draft. Preserve its source reference and mark it draft, never customer contacted. |
Example E is eligible only for the action described in the test. The absence of a stop flag alone does not establish contact permission. If another system later supplies a conflicting event, withdraw that eligibility and return the record to its owner.
Notice how little extra generation helps Examples A through D. Their next steps are source checks or owner decisions. A useful second agent might prepare the internal note for Example E after those controls exist. It cannot create evidence that the source systems never recorded.
Put the human decision at the consequential boundary
Human review belongs where a decision could affect contact permission, privacy, safety, service fit, sensitive content, or a customer-facing promise. The reviewer should see the evidence and record the decision, not merely approve or reject a black-box score.
Make the outcome explicit: approved draft, hold for more context, do not contact, duplicate merged, or stop because the scope is unclear. The reason must remain visible to the next operator. For reply workflows, the separate human hold queue guide shows how pause reasons and reviewer actions can remain inspectable.
Fix repeated exception causes before adding capacity
If the same exception appears repeatedly, look upstream. A missing owner may call for an assignment rule. Duplicate records may need source reconciliation. Repeated service-area uncertainty may need a controlled reference list. Unsafe customer-facing drafts may need a stronger approval boundary. A routing problem is not automatically an AI problem, and generating more text does not fill a missing field.
For example, an owner-assignment rule may clear a recurring routing hold without another model call. After repairing it, review fresh exceptions. If the same gap persists, investigate the source rather than treating a lower queue count as proof of a fix.
Build the decision packet before the architecture diagram
Write a short proposal that the operator responsible for the queue can check. State the repeated task in plain language, the records allowed to enter, every required field, the outputs the agent may create, the events it may never claim, the human escalation path, the logging method, the rollback owner, and the conditions that end the test. Link each statement to an observed queue condition rather than to a hoped-for efficiency gain.
Include a redacted example of a ready record and an ambiguous one. Walk both through the proposed handoff. If the ambiguous record can cross the boundary because a field is blank, a label is inferred, or a reviewer is optional, the design is not ready. If the ready record cannot produce an inspectable event trail, the design is also not ready.
Finally, name the source of truth for each state. A CRM field, messaging provider event, form submission, reviewer decision, and agent output may come from different systems. The packet should say which source wins when they disagree and who investigates the conflict. Do not resolve a conflict by choosing whichever source makes the queue look complete.
Apply a go/no-go test
On a narrow screen, scroll the table horizontally.
| Decision | Use when | Required evidence |
|---|---|---|
| Go to a bounded pilot | The task repeats, inputs are complete, and uncertainty routes to a named human | Handoff contract, stop conditions, audit log, reviewer, rollback |
| Repair first | The queue is dominated by missing ownership, conflicting events, duplicates, or vague states | Corrected source, state, ownership, or permission rule |
| No-go | The task requires unsupported inference, hides consequential decisions, or lacks a safe stop path | Documented reason and retained human process |
A "go" is permission for one reversible test, not a general release. Define which records may enter, which fields the agent may read, what output it may create, which events it may never claim, and how the operator disables it. Review the exceptions created by the pilot before widening the scope.
Stop the pilot if the handoff loses source evidence, a stop signal is ignored, a draft is recorded as sent, ownership disappears, or the rollback cannot be verified.
Test the stop path before giving the agent any write permission. Start with a copy or read-only mode where possible. Ask the operator to disable the pilot and show which records remain pending, who now owns them and where the source history is kept. A successful stop must leave the work understandable to the next person.
Rollback has limits. Turning off an agent does not unsend an email, undo a disclosure or retract a promise a customer already received. For the fictional internal-draft pilot above, returning to manual preparation is a testable recovery path. A proposed pilot that can contact customers or change records needs separate controls and authorization for those effects.
Define pilot evidence before defining pilot success
Success criteria should be observable process facts: eligible records entered, prohibited records stayed out, required fields arrived intact, every output kept its source reference, human escalations reached the named owner, stop signals worked, and rollback restored the prior path. Time saved or volume processed can be secondary observations, but they do not excuse a broken control.
Review every exception created by the pilot, not just the records that completed. Look for new ambiguity introduced by the second handoff: overwritten timestamps, lost source links, duplicated actions, ownership changes, or outputs recorded as real-world events. Keep the pilot closed if the new handoff makes those exceptions harder to investigate, even when routine records move faster.
Sources and limits
The NIST AI RMF Core describes documented scope, human oversight, testing, monitoring, and risk controls across an AI system's lifecycle. The NIST AI RMF Playbook measure guidance discusses documenting test sets, metrics and measurement methods. This article adapts those voluntary resources into a queue-review method; the tables and test choices here are our proposed working format, not a NIST checklist or certification. Neither source establishes that a particular workflow is safe or compliant.
For sample handling, the FTC guide to protecting personal information recommends understanding how personal information flows through a business, keeping only what is needed, and limiting access. Applicable privacy, consent, retention, and industry rules depend on the business and jurisdiction; this article does not provide legal advice.
Frequently asked questions
What should be in an AI exception queue?
Each held item should show a specific reason, current owner, last verified event, evidence timestamp, permission or safety boundary, open question, and next safe action. Draft, sent, delivered, replied, and closed must remain separate states.
When is a second AI agent justified?
A second agent is justified only when it removes a repeated, well-defined task, receives enough evidence to act within a documented boundary, routes uncertainty to a named human, and can be disabled or rolled back without losing the audit trail.
How large should the first exception-queue sample be?
Use a small, redacted, mixed sample rather than a full customer export. Include different hold reasons such as a duplicate candidate, missing context, a stop signal, and one apparently ready item. The sample is for finding control gaps, not estimating business performance.
Does an exception mean the lead or AI system failed?
No. An exception can be an expected pause for service-area review, duplicate reconciliation, missing ownership, sensitive content, or unclear permission. Keep an explicit Unknown state instead of turning uncertainty into a guessed customer intent or a generic error.