The customer has no labelled data. How do you build an evaluation set with them?

Build it from their own historical work and with their experts. Pull a representative sample of past cases, use what already happened (the reply that was sent, the field that was entered, the decision that was made) as the first draft of the right answer, have their experts correct and agree on a small set, and turn their judgement into a written rubric. Start with 50 to 100 cases in the first week, and grow the set from every failure in the pilot.

1. Mine the history

Most businesses already have answers, just not labelled as such:

• support: the agent's final reply and the resolution code,
• documents: the values someone typed into the system from that document,
• triage: the team the case was finally assigned to,
• approvals: the decision and its stated reason.

These are imperfect (people make mistakes too), but they are a strong starting point.

2. Sample properly

Take cases across categories, difficulty, channels and time, in proportion to real volume, plus deliberate edge cases (unusual formats, angry customers, ambiguous requests). Remove or mask personal data before it leaves their environment.

3. Have experts label and agree

• Run a short labelling session with 2 or 3 of their best people.
• Each labels the same 20 cases independently. Where they disagree, discuss and write down the rule. That conversation reveals the real policy.
• Measure how often experts agree. If they agree only 80% of the time, do not expect the model to reach 99% on that task.

4. Write the rubric

Turn their judgement into criteria that a person or an LLM judge can apply: "Mentions the correct return window", "Doesn't promise a refund before inspection", "Uses the customer's name". Each criterion scored pass or fail.

5. Choose graders

• Exact checks in code for structured fields and classifications.
• An LLM judge with the rubric for free text, checked against the experts' labels before trusting it.
• Expert review of a sample every week.

6. Agree the bar before seeing results

"At least 90% of replies pass all policy criteria, and none contain a wrong order status." Set this with the sponsor up front, so the pilot's outcome is not argued afterwards.

7. Keep it alive

Every failure found in the pilot, and every case where users corrected the AI, becomes a new test. Keep a held-out part that is never used for prompt tuning.

What this shows an interviewer

That you can create measurement where none existed, use the customer's expertise rather than guessing their standards, and protect their data while doing it.