The full answer
A prompt is effective when it meets measurable success criteria on a representative test set, consistently, at acceptable cost and speed. I evaluate it with automated checks where possible, a model grader with a rubric for open-ended qualities, and human review to calibrate both.
1. Start from criteria Before writing the prompt, agree what good means for this use case, for example:
- task accuracy: the right category, the correct extracted fields;
- faithfulness: every claim supported by the supplied context;
- completeness: all required points covered;
- format: valid JSON or the required structure;
- tone and length;
- safety: harmful requests refused, normal ones not.
2. Assemble an evaluation set Representative real inputs, edge cases, and adversarial inputs, each with an expected answer or a list of properties the answer must have. Grow it whenever production shows a new kind of failure.
3. Metrics and methods
| What to measure | Method |
|---|---|
| Exact answers (labels, numbers, extracted fields) | Exact match, accuracy, precision and recall, F1 |
| Format | Parse the output; validate against a JSON schema |
| Generated code or SQL | Run it against tests or a sample database |
| Faithfulness and relevance | Model grader with a rubric, given the source text |
| Similarity to a reference | Embedding similarity; older overlap metrics such as BLEU and ROUGE correlate poorly with quality for open-ended text |
| Preference between versions | Side-by-side comparisons by people or a grader |
| Consistency | Run each case several times and measure variation |
| Cost and latency | Tokens per request, response time |
4. Making model graders trustworthy
- Give the grader a specific rubric and ask for a reason before the score.
- Prefer simple scales (pass or fail, or 1 to 5 with each level described).
- Compare its scores with human labels on a sample, and watch for biases such as preferring longer answers or the first option shown.
5. Comparing versions Change one thing at a time and run the full set; look at which cases changed, not just the average. Keep a record of each prompt version and its scores, like code under version control.
6. In production Track user signals (ratings, edits, retries, escalations to a human), sample conversations for review, and re-run the evaluation when the model or the data changes.
Last updated 16 October 2026
Sources