Back to catalog / Agent evaluation

Measure RAG retrieval and final-answer quality separately

Use questions with gold source labels to compute retrieval Precision/Recall/MRR separately from end-to-end accuracy.

Goal

Identify whether a regression or improvement comes from retrieval or answer synthesis.

Prepare question, gold relevant chunk links, and reference answer per item, including multi-document synthesis cases. [rag-eval]
Use the notebook’s evaluate_retrieval(...) for precision, recall, F1, and MRR; run evaluate_end_to_end(...) separately. [rag-eval]
Write method results to separate CSV/JSON with fixed data/model versions and compare both retrieval and answer metrics. [rag-eval]

Baseline and improved scores/per-item details are reproducible; retrieval and answer failures can be distinguished.

If answers are correct but MRR is low, inspect labels/equivalent sources; if recall is high but accuracy low, inspect synthesis and abstention.

Python, Anthropic/Voyage API credentials, a labeled document-chunk set, and a local evaluation environment.

Use a redacted evaluation set without unapproved customer questions or documents.

Synthetic questions and model scoring can be biased; pair scores with human spot checks and avoid optimizing a single retrieval metric.

These values describe the upstream repository, not popularity or likes for an individual skill or playbook.

No metrics snapshot is available yet.

License: MIT