An API that one of your agent's tools calls changed its response format, and the agent started giving wrong answers without any errors. How do you catch this kind of failure?

Treat every tool as a contract and check it in code at the boundary, before the model sees anything. The model is very good at making sense of whatever it receives, which is exactly the problem: a renamed field or an empty list becomes a confident wrong answer instead of an error. Validate each tool response against a schema, turn violations into explicit errors, and monitor what tools return, not only whether they succeeded.

What went wrong

Say the inventory API renamed quantity_available to available_qty. The tool passed the raw JSON to the model. The model found no quantity, assumed none were in stock, and told customers items were sold out. Every call returned HTTP 200, every run finished, no alert fired.

1. Validate at the tool boundary

Parse the response into a typed model and fail loudly when it does not fit:

class StockLevel(BaseModel):
    sku: str
    available: int = Field(ge=0)

def check_stock(sku: str) -> dict:
    raw = inventory_api.get(sku)
    try:
        stock = StockLevel(sku=raw["sku"], available=raw["quantity_available"])
    except (KeyError, ValidationError):
        log_contract_violation("inventory", raw)
        return {"error": "Stock level is unavailable right now. Do not guess; tell the customer you will check."}
    return {"sku": stock.sku, "available": stock.available}

Two things changed: the code, not the model, decides whether the data is usable, and the agent gets a clear, actionable error instead of a puzzle. Returning only the fields the agent needs also keeps it from reading meaning into fields it should ignore.

2. Test the contract before production

• Contract tests in your test suite call the real API (or its published schema) and check the fields and types your tool depends on.
• Pin API versions where the provider supports it, and subscribe to its change notices.
• Recorded responses in the agent's evaluation set, including an empty list, a missing field and an error body, so you know how the agent behaves when a tool returns bad data.

3. Monitor what tools return

Success rates miss this failure. Track, per tool:

• Contract violations and error responses.
• Empty or null results, as a share of calls.
• The distribution of key values. "Out of stock" jumping from 3% to 60% of answers overnight is a signal even with no errors.

And at the end of the chain: user feedback, escalations, and a sample of transcripts read each week.

4. When it happens anyway

• Find the first bad run from the traces and the change date of the API.
• Fix the mapping, and replay affected conversations to see who got wrong answers.
• Add the case to the evaluation set.

What interviewers want: the insight that agents turn data errors into fluent wrong answers instead of exceptions, schema validation in the tool rather than trust in the model, actionable error messages, contract tests, and monitoring on the content of tool results.