LLM Evaluation and Safety

Not this time. Start with specific, measurable success criteria, then collect realistic inputs with edge cases. Without criteria, there is nothing to grade against.

Not this time. They documented position bias (favouring an answer because of where it appears), verbosity bias (favouring longer answers) and self-enhancement bias (favouring their own outputs). Judging in both orders corrects position bias.

Not this time. Faithfulness is the share of claims in the answer that the retrieved context supports, from 0 to 1. A low score means the model added things the sources do not say.

Not this time. LLM01:2025 is prompt injection: input, typed directly or hidden in content the model reads, that changes its behaviour.

Not this time. Passing model output to a browser, database or shell without validation and encoding is LLM05:2025, improper output handling. A fooled model could inject a script.

Not this time. Assume the system prompt can leak. Keep secrets out of prompts entirely: the tool code holds the credential and the model only asks for the action.

Not this time. Prompts can be overridden and classifiers can be fooled. A permission check in code, limits on amounts and a person's approval enforce the rule whatever the model outputs.

Not this time. Real inputs have the messiness, length and range that hand-written examples lack. Evaluations should mirror the real distribution, including edge cases, or they overstate quality.

Not this time. Anthropic gives at least 60 days' notice before retiring a publicly released model and advises testing with newer models well before the retirement date.

Not this time. A judge is a measuring instrument. Checking how often it agrees with people on a labelled sample shows whether its scores can be trusted, and should be repeated when the judge model or prompt changes.

Not this time. Look at the per-question differences (a paired comparison) and their confidence interval. Pairing removes the variation from question difficulty, so the interval is much tighter than comparing two separate scores.

Not this time. Assume it can leak. Keep credentials and permission rules out of it and enforce controls in code, so a leak exposes nothing harmful.

Not this time. A jailbreak tries to make the model break its own safety rules. Prompt injection, often hidden in content the model reads, hijacks what the application does with its tools and data. OWASP treats jailbreaking as a form of injection.

Not this time. Poisoned training or fine-tuning data can plant behaviour that stays hidden until a specific trigger appears, which makes it hard to find in ordinary testing.

Not this time. Denial of wallet, part of OWASP's unbounded consumption risk. Rate limits, quotas, input and output caps and a budget with alerts defend against it.