How to choose an LLM eval framework you can trust
Ragas, DeepEval, promptfoo and Inspect all run LLM-as-judge checks and fit in CI; pick the one that matches how your team already tests. What decides whether the scores mean anything is the question set you own, built from real user requests and failures, plus error bars on every score and a check that your offline score moves with what users do.
Tecyfy · Published · 8 min read
Ragas, DeepEval, promptfoo and Inspect all score answers with an LLM judge and run in CI, so pick the one that fits how your team already tests. The framework is not what makes evals trustworthy. 3 things do: a question set you own, built from real user requests and failures; an error bar on every score, so noise is not read as progress; and a regular check that your offline score moves with what users actually do.
The 4 open-source frameworks, side by side
| Ragas | DeepEval | promptfoo | Inspect | |
|---|---|---|---|---|
| Best fit | RAG metrics in Python | Teams that test with pytest | Comparing prompts and models from YAML | Agent and tool-use evals |
| How you write tests | Python | Python, run with deepeval test run | A YAML config, run from the command line | Python: dataset, solver, scorer |
| LLM as judge | Its LLM-based metrics | G-Eval with your own criteria | llm-rubric assertion | model_graded_qa() |
| RAG metrics | Faithfulness, context precision and recall, response relevancy and more | Faithfulness, contextual precision and recall, answer relevancy | Evaluates RAG pipelines | Write your own scorer |
| Agents | Tool call accuracy, agent goal accuracy | Task completion over a whole run | Not its focus | Agents, MCP tools, sandboxes in Docker or Kubernetes |
| Repeated runs | --epochs runs each sample several times | |||
| Licence | Apache-2.0 | Apache-2.0 | MIT | MIT |
All 4 run on your own machine. DeepEval adds an optional hosted platform (Confident AI). promptfoo also does red-teaming. Inspect comes from the UK AI Security Institute and is built for long agent tasks. Ragas has moved to a new GitHub home (vibrantlabsai/ragas) and was last updated in February 2026, so check its activity before you build on it.
How to choose in a minute:
- Your tests are pytest: DeepEval.
- You compare prompts or models and want a config file, not code: promptfoo.
- Your app is an agent that uses tools over many steps: Inspect.
- You need RAG metric names your team already knows: Ragas, or the RAG metrics in DeepEval.
Why the scores move and users don't
A common story: a team swaps in a better embedding model, context recall on the test set goes up, and nothing changes for users. Each metric was right about what it measures. It just measures something next to what users care about.
- The questions are not the users' questions. A test set written by the team, or generated by a model from the documents, asks what the documents answer. Users ask what they need, in their own words, often about what the documents do not cover.
- The metric is one step of the chain. Better retrieval does not help if the answer is too long, hedged, or wrong about the one detail the user asked about.
- Users fail in ways no metric checks. A slow answer, a refusal, a link that 404s, an answer to the wrong question.
So the first job is not choosing metrics. It is choosing questions.
Step 1: own your eval set
Build your eval setEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped. from real use:
- Sample real requests. Take a few hundred from logs (with personal data removed) and keep the ones that represent common jobs.
- Add every failure. Thumbs down, complaints, escalations to a person, conversations where the user asked the same thing twice. These are the questions that matter.
- Write what a good answer must contain, not a perfect answer: "names the refund window; links the form; does not promise a date".
- Keep it versioned next to the code, and add each new failure as a question. The set grows with the product.
50 questions you trust are worth more than 1,000 generated ones. The framework you picked runs them; it cannot write them for you.
Step 2: check the judge before you trust it
Every framework above uses an LLM judgeLLM as a JudgeUsing a language model to grade another model's output against a rubric, for qualities code cannot check. It scales to thousands of answers, but must be checked against a person's verdicts before it is trusted. for open answers. A judge is a model with a rubric, and it can be wrong in steady ways. Have people grade 50 to 100 answers, compare them with the judge's verdicts, and fix the rubric where they disagree. Re-check when you change the judge model. How to write a rubric you can trust covers this step by step.
For agents, score the whole run as well as single answers: did it finish the task, with which tools, in how many steps. How to evaluate an AI agent covers what to check at each step.
Step 3: put error bars on every score
An eval score is an estimate from a sample of questions, so it has a margin of error. Report a standard error with every score. The size surprises most teams:
| Questions | Score | 95% margin, either way |
|---|---|---|
| 50 | 80% | about 11 points |
| 200 | 80% | about 5.5 points |
| 1,000 | 80% | about 2.5 points |
On 200 questions, a change from 80 to 83 percent can be chance. 3 habits keep CI honest:
- Compare the same questions, paired. Run the old and new versions on the same set and look at the questions that changed. Paired comparison removes the noise of question difficulty, the largest source of it.
- Run each question more than once when the model samples (temperature above 0, or an agent). Average per question. Inspect's
--epochsdoes this. - Fail the build on a drop larger than the margin, not on any drop. Report the rest as a trend.
The eval sample size calculator says whether a difference is real and how many questions you need to see the change you care about.
Could be chance
A change of 3 points on 200 questions sits inside its margin of 7.6: the interval crosses zero, so it could be chance. Comparing on the same questions narrows it.
95% intervals, from the standard error of a pass rate. On the same questions, the number is how alike the 2 versions score each question (the correlation); 0.3 to 0.7 is what was found between frontier models on popular evals. The sums are those of the eval sample size calculator.
Step 4: tie the offline score to real use
Pick one signal from real use that says the product worked: a task completed, no thumbs down, no follow-up asking the same thing, no hand-off to a person. Each time you ship a change, write down the offline score change and, a week later, the change in that signal. After a few releases you know whether your eval predicts what users feel. If the two never move together, change the questions, not the framework.
A starter loop
- Pick the framework that matches how you already test.
- Write 50 questions from real requests and failures, with what a good answer must contain.
- Check the judge against people on a sample.
- Run old and new versions on the same questions, with error bars.
- Gate CI on drops larger than the margin.
- Add each new production failure to the set.
- Every few releases, check the offline score against the real-use signal.
For retrieval systems in particular, retrieval evaluationRetrieval EvaluationTesting the search step of a RAG system on its own, with questions whose answering passages you know, and scoring it with recall at k, the share of those passages found in the top k results. explains what the recall and precision metrics measure, and where they stop.
Questions
- What is the best LLM evaluation framework?
- There is no single best one. Ragas suits RAG metrics in Python, DeepEval suits teams that test with pytest, promptfoo suits YAML configs and prompt or model comparisons from the command line, and Inspect suits agent and tool-use evaluations with sandboxes. All 4 are open source and support LLM-as-judge scoring. The question set you evaluate on matters more than the framework.
- Why did my RAG metrics improve but users are not happier?
- Because the metrics measure something next to what users care about. A new embedding model can raise context recall on your test questions while users still ask different questions, want shorter answers, or give up for reasons no metric checks. Build the question set from real requests and check that your offline score moves with a signal from real use.
- How noisy are LLM evals in CI?
- Noisier than most teams assume. On 200 questions with a score of 80 percent, the 95 percent margin is about 5.5 points either way, so a change from 80 to 83 can be chance. Report a standard error with each score, compare the same questions run by both versions, and repeat runs when the model samples.
- How do I build my own eval dataset?
- Start from real traffic: take a sample of user requests, add every request that went wrong (complaints, thumbs down, escalations), and write what a good answer must contain for each. 50 well-chosen questions you trust beat 1,000 generated ones. Add each new failure as a question, so the set grows with the product.
- Can I trust LLM-as-judge scores?
- Only after you check them. Have people grade a sample of 50 to 100 answers, compare their verdicts with the judge's, and fix the rubric where they disagree. Re-check whenever you change the judge model or the rubric.