Back to catalog / Agent evaluation

Generate privacy-safe synthetic test cases for a prompt template

Generate diverse inputs within template-variable constraints, then compare prompt versions on them.

Goal

Build a reproducible prompt test set when real user examples are unavailable or prohibited.

List each template variable, type, valid range, and exclusion rule. [synthetic]
Use the notebook’s generator prompt to create common, boundary, and combined JSON cases; validate fields before saving. [synthetic]
Run the same set against prompt versions with fixed model/settings and spot-check variable validity. [synthetic]

The dataset covers intended categories without real personal data, and prompt comparisons are reproducible.

Discard cases that violate constraints instead of editing them into passing examples; add missing boundary rules and regenerate.

Python, Anthropic SDK, an API key, and the template definition; no real user examples are needed.

Generated inputs must contain no real personal information; store outputs in an approved isolated test directory.

Synthetic distributions may not represent real users and cannot alone establish release readiness or fairness.

These values describe the upstream repository, not popularity or likes for an individual skill or playbook.

No metrics snapshot is available yet.

License: MIT