Back to catalog / Agent workflows

Reproduce a search-agent benchmark and score accuracy

Run a search agent on a fixed question set and quantify results with reference answers and a defined grader.

Goal

Goal: Run a search agent on a fixed question set and quantify results with reference answers and a defined grader.

Pin sample questions, target model, tool versions, effort, and token task budget; record each configuration. [search-eval]
Generate answers with sample(question, max_turns=100) and save search traces, terminal state, and raw output. [search-eval]
Use grade_response for precision/recall/F1; use the specified A/B/C grader for BrowseComp-style cases and report abstentions separately. [search-eval]

The pinned configuration reruns consistently; report sample count, F1/accuracy, abstentions, budget, and latency.

If scores vary, inspect normalization and question mix; label small-sample results and avoid cross-model comparisons with different effort.

Python 3.11+, Anthropic API key, benchmark dependencies, and an authorized question set.

Cap token budget/cost; obtain rights for restricted questions and retain only necessary benchmark data.

Results depend on model version, tools, web availability, and randomness; benchmark scores are not production guarantees.

These values describe the upstream repository, not popularity or likes for an individual skill or playbook.

No metrics snapshot is available yet.

License: MIT