Back to catalog / Agent workflows
Canary-test and roll back a prompt version
Compare prompt versions on labeled examples and pin production back to the old version after regression.
Goal
Goal: Compare prompt versions on labeled examples and pin production back to the old version after regression.
Steps
- Keep a fixed set of labeled support tickets and define accuracy and critical-misclassification thresholds. [prompt]
- Create an immutable version with
agents.update(..., version=known_version, system=...); pin test sessions usingagent={"type":"agent","id":...,"version":N}. [prompt] - Score v1 and v2 on the same labels; promote v2 only when it meets thresholds and roll back by restoring v1 in config. [prompt]
How to verify
Record the dataset, version, per-class scores, and promotion/rollback decision for repeatable comparison.
Troubleshooting
Do not use a bare latest agent ID in production; older versions remain available for rollback but require controlled config promotion.
Permissions
Prerequisites
- Python 3.11+, Anthropic API key, Managed Agents beta SDK, and a de-identified labeled regression set.
Permissions
- Validate on test data only; canary gradually and separate release authority from evaluation-data access.
Limitations
Measure prompt changes against scores and coverage, then review production monitoring and critical cases with a person.
Pinned upstream source
- Repository
- anthropics/anthropic-cookbook
- File path
- managed_agents/CMA_prompt_versioning_and_rollback.ipynb
- Pinned commit
9c1e493ddd2b82df399d2a61b9c806c539e18be6- Source checked
- 2026-10-14 · Reviewed against the pinned upstream source
Repository metrics
These values describe the upstream repository, not popularity or likes for an individual skill or playbook.
- Stars
- Not collected
- Forks
- Not collected
- Observed at
- Not collected
No metrics snapshot is available yet.
License: MIT
Main topic
Turn ecommerce campaign materials into a reviewable weekly brief