How can you evaluate the quality of a prompt?

Judge a prompt by its results on a fixed set of test inputs, scored against criteria you defined before writing it, not by how well it reads or how one example turned out.

1. Define success first Write specific, measurable criteria: "classifies support tickets into the right one of 8 categories at least 95% of the time", "summaries under 120 words that mention every action item", "never gives medical dosage advice". Vague goals such as "good answers" can't be tested.

2. Build a test set

• Real inputs from the use case, including edge cases, ambiguous inputs and adversarial ones (attempts to make it ignore instructions).
• Expected outputs or properties for each, where possible.
• Enough cases for the differences to be meaningful: tens to start, hundreds for important prompts.
• Keep it separate from the examples inside the prompt, or you are testing memory.

3. Choose how to grade

• Code-graded: exact match, a regular expression, valid JSON against a schema, or a test that runs generated code. Fast, cheap and objective; use it wherever you can.
• Model-graded: another model scores each answer against a clear rubric (faithfulness to a source, tone, completeness). Scales to open-ended outputs, but check its scores against human judgement on a sample, and give it the reference answer when there is one.
• Human review: the most reliable for nuance, the slowest and most expensive. Use it to set the standard and audit the other graders.

4. Measure what matters for the task

• Accuracy or pass rate against the criteria.
• Faithfulness: does the answer stay with the supplied context, without invented facts?
• Format compliance: does it parse?
• Consistency: run each case several times, since outputs vary.
• Safety: refusal of what should be refused, no over-refusal of normal requests.
• Cost and latency: tokens in and out, and time to answer.

5. Compare and iterate Change one thing at a time, re-run the whole set, and keep the version that scores better without breaking other cases. A change that fixes one example often breaks another, and only the full set shows it.

6. Keep evaluating in production Log real inputs and outputs (with privacy in mind), collect user feedback such as thumbs up or down and edits, add failures to the test set, and re-run the evaluation when the model version changes. Pinning a specific model snapshot keeps behaviour stable between deliberate upgrades.