Back to catalog / Agent evaluation
Build agent regression evaluation from recorded conversations and tool traces
Use representative, redacted conversations to define expected answers and tool trajectories and detect regressions in pytest.
Goal
Turn agent behavior into repeatable quality checks covering answers and tool choices.
Steps
- Select common, boundary, and recovery cases and remove private fields. [eval]
- Save redacted conversations as an
*.test.jsonEvalSet with user turns, expected answers, and expected tool trajectories. [eval] - Run
AgentEvaluator.evaluate(agent_module="my_agents.agent", eval_dataset_file_path_or_dir="tests/eval/")from pytest. [eval] - Review trajectory differences for failures and distinguish wording changes from wrong tools or true regressions. [eval]
How to verify
CI reports answer and tool scores per case and identifies the conversation and deviation on failure.
Troubleshooting
If scores fluctuate, stabilize cases and evaluator settings; do not loosen thresholds to hide tool-trajectory errors.
Permissions
Prerequisites
- Python 3.11+, google-adk compatible with the pinned revision, and an isolated test environment
- Configured model-provider credentials for live inference; never place them in code or logs
- Evaluation runs the agent for real and requires model credentials and inference budget even with deterministic scorers
Permissions
- Use authorized test data only; isolate sessions and credentials per user; obtain human authorization before external side effects
Limitations
This procedure follows Google ADK at the pinned revision; other versions, models, or remote services may differ in APIs and event semantics. Recheck permissions, retention, and recovery before production.
Pinned upstream source
- Repository
- google/adk-python
- File path
- docs/guides/evaluation/agent_evaluator/index.md
- Pinned commit
e9b74d36140f2594b24c481fb7a3825b2cb01e96- Source checked
- 2026-10-14 · Reviewed against the pinned upstream source
Repository metrics
These values describe the upstream repository, not popularity or likes for an individual skill or playbook.
- Stars
- Not collected
- Forks
- Not collected
- Observed at
- Not collected
No metrics snapshot is available yet.
License: Apache-2.0
Main topic
Take a coding Agent change from context to verification