Set one bounded task, approved samples, a manual baseline, and retest rules before comparing total time and quality. Expand only when predefined conditions are met.
Limit a pilot to one business task with an owner, defined inputs, and checkable outputs. Record manual time and errors on the same samples, then measure Agent run, review, and rework time. Test typical, missing, duplicate, and conflicting cases. Expand only when quality meets the agreed bar, total cost is acceptable, and recovery ownership is clear.
| Work item | Manual baseline (minutes) | Agent option (minutes/cost) |
|---|---|---|
| Prepare/organize inputs | _____ staff minutes | _____ staff minutes; _____ elapsed run minutes |
| Review output | _____ staff minutes | _____ staff minutes |
| Rework/correction | _____ staff minutes | _____ staff minutes |
| Tool/model fees | Not applicable or current amount | _____ amount and currency |
| Case | Expected behavior | Evidence of passing |
|---|---|---|
| Typical record | Draft from sources; figures traceable | Fields, row references, and figures match |
| Missing field/denominator | Flag missing data; pause affected conclusion | Missing item listed; no guessing or comparison |
| Duplicate record | Flag for confirmation; do not count twice | Duplicate ID and pending status traceable |
| Conflicting sources | Show both sources and hand off for review | Sources, timestamps, and reviewer recorded |
| Execution failure | Stop, retain record, and hand off | Failure state, recovery steps, and owner recorded |
Write a pilot card covering the task and exclusions, input owner, reviewer, recipient, approved data, human takeover, and conditions that trigger a pause. For a first trial, keep the Agent to drafts; do not automate refunds, inventory changes, customer messages, or publication. Before a consultation, prepare: task owner; authorized data samples; scope and exclusions; acceptance samples and pass conditions; tool permissions and human approval; expected deliverable files; maintenance, fees, and exit terms.
Record the manual baseline: time to do the task, check the result, and rework it, plus errors or omissions. For the Agent, record run time, human review time, rework time, and any tool or model fees separately. Compare the same samples; do not treat one successful run as general savings or omit review cost.
Agree on pass conditions before the pilot, such as correct critical fields, flagged exceptions, and no unauthorized actions. The business owner sets thresholds for the task’s risk; example thresholds are not industry standards.
Hand off to a person and keep a record when inputs change, sources conflict, permissions are unclear, a run repeats, or the result cannot be checked. Assign who fixes data, restarts work, and prevents duplicate writes.
Retest typical inputs, missing fields, duplicates, conflicting values, and one recoverable failure. Pause and narrow scope if quality misses the bar or rework removes the benefit. If it passes, add one input type or a small batch, retest, then discuss any new actions such as write-back or sending.
Compare the example input with the result and acceptance checks.
Fictional planning brief: a team wants to evaluate a weekly campaign-brief draft. Sample count, owner, manual baseline, acceptance thresholds, approved data, and tool permissions have not yet been set. This example fills in a plan; it does not run an Agent.
From this incomplete brief, return a fillable pilot-plan JSON with task and exclusions, owner, approved samples, pass conditions, manual baseline (preparation/processing/review/rework minutes), Agent elapsed run time, human review/rework minutes, fees, permissions and approval, deliverables, maintenance owner, and pause/exit conditions. Write “to be filled” for unknowns. Do not claim the pilot ran, results were measured, or a particular exception occurred.
{
"task_and_exclusions": "to be filled",
"owner_and_reviewer": "to be filled",
"approved_samples_and_pass_conditions": "to be filled",
"manual_baseline_minutes": "preparation / processing / review / rework: to be filled",
"agent_elapsed_run_time_and_fees": "to be filled",
"human_review_and_rework_minutes": "to be filled",
"permissions_and_approval": "draft only until approved",
"deliverables_and_maintenance_owner": "to be filled",
"pause_and_exit_conditions": "to be filled",
"pilot_status": "plan only; not executed"
}Why it is included: Use it for code or system security reviews, with references matched to the project language and framework.
When it does not fit: It is not a runtime guard that blocks a particular malicious document instruction.
Why it is included: Map entry points, identities, assets, and attacker capabilities to prioritized threats.
When it does not fit: A low-risk offline summary does not need a full production-system threat model.
Why it is included: Add testable rejection conditions for user input and final answers, distinguishing per-tool checks.
When it does not fit: A prompt/output check cannot replace tool permission boundaries.
Why it is included: Pause an agent run before an external change so a person can inspect and approve the exact action.
When it does not fit: A local draft with no external send/publish action needs no approval integration.
Why it is included: Expose a narrowly scoped application function as a tool with an explicit parameter contract.
When it does not fit: If the task only reads a Skill or static resource, no tool schema is needed.
Why it is included: Verify application-owned tool dispatch, handoffs, and output handling without calling a live model service.
When it does not fit: Mock-model tests do not measure the quality of a real model response.
Why it is included: Correlate model calls, tool execution, guardrails, and handoffs in one workflow timeline for diagnosis.
When it does not fit: A single offline decision can use a simple test; distributed spans are unnecessary.