Back to catalog / Agent workflows

Compare tool designs against a task set

Run agent tool calls on a fixed task set and measure success, latency, and tool feedback.

Goal

Goal: Run agent tool calls on a fixed task set and measure success, latency, and tool feedback.

Prepare answer-checkable task fixtures and candidate tool schemas; pin model, prompt, and run limits. [tool-eval]
Run each task through agent_loop(prompt, tools=...); record tool sequence, inputs, latency, and final answer. [tool-eval]
Check answers against expected outputs and summarize actionable feedback on names/parameters/descriptions; repeat cases to estimate variance. [tool-eval]

Report per-case success/failure, tool use, and latency; compare schema revisions on the same baseline.

Count unparsable output as a failed check, refine the contract, and assess tool quality from validated results and execution evidence.

Use mock or read-only tools; do not connect production write endpoints to the benchmark.

Model feedback is diagnostic, not ground truth; cost grows with task and tool count.

These values describe the upstream repository, not popularity or likes for an individual skill or playbook.

No metrics snapshot is available yet.

License: MIT