The full answer
Treat it as a statistics problem first and a code problem second. Run the same task many times to measure the failure rate, collect full traces of passing and failing runs, find the step where they diverge, classify the cause (an ambiguous instruction, a confusing tool, a flaky tool, a context problem), fix that one cause, and re-measure on the same set.
1. Measure the rate
Run the task 20 to 50 times. "It sometimes fails" might mean 5% or 50%, and the fix you need is different. Keep these runs as a fixed set so you can compare before and after. Because scores vary between runs, a change from 70% to 76% on a handful of runs may be noise; Anthropic's work on statistical evaluation recommends reporting standard errors so you can tell.
2. Get the traces
For each run: the prompt and tools as sent, every model turn, every tool call with arguments and results, token counts, and the final output. Without full traces you are guessing.
3. Find the fork
Put a passing and a failing trace side by side and find the first step where they differ. Common forks:
| What you see at the fork | Likely cause | Fix |
|---|---|---|
| The agent picks a different tool | Two tools overlap, or descriptions are vague | Merge or rename tools; say when to use each |
| Same tool, wrong arguments | Unclear parameter names or formats | Clearer schema, enums, examples in the description |
| A tool returned an error or empty result | Flaky tool or timeout | Retries in code; clearer error messages |
| A huge tool result, then confusion | Context overload | Paginate, filter, clear old results |
| The agent stops early and claims success | No check of the end state | Add a verification step or tool; tighten the stop condition |
| The instruction is read two ways | Ambiguous task wording | Rewrite the prompt with the missing detail |
4. Fix one thing at a time
Change one cause, rerun the set, and compare the failure rate. Several changes at once make it impossible to know what helped.
5. Reduce randomness where it does not help
Make tools deterministic where you can (sorted results, stable ids). Pin the model version. Give the agent explicit stop criteria and a final check of the result.
6. Keep the cases
Every failing trace you diagnose becomes a regression test. Anthropic's tool-writing guide describes reading agent transcripts in exactly this way to improve tool names, descriptions and responses.
What interviewers look for: repetition and traces before opinions, and fixes in tools and structure rather than ever-longer prompts.
Last updated 16 October 2026
Sources