The full answer
Build it as a model that decides and a set of tools that enforce. The model reads the conversation and chooses the next step; every rule that matters (who the customer is, what they may change, how much can be refunded) lives in the tools and the systems behind them. Around that sit a router, a handover path to people that keeps the context, an audit trail of every action, and an evaluation suite that checks what actually changed in the shop's systems, not what the agent said.
1. Clarify the scope first
Ask before drawing anything:
- Which requests make up most of the volume? Usually "where is my order", returns and delivery changes cover most of it.
- Which actions may the agent take alone, and up to what value?
- Channels (chat, email) and languages.
- What counts as success: resolved without a person, customer satisfaction, time to resolve.
2. The shape of the system
| Part | Job |
|---|---|
| Session | The logged-in customer's id, from authentication. Every tool reads it from here, never from the model's arguments. |
| Router | Classifies the request (order status, return, delivery change, product question, complaint, other) and picks the prompt, tools and model for it. Simple status questions can go to a smaller, cheaper model. |
| Agent loop | One model with the tools for that route, a limit on the number of steps, and a stop when it is done or stuck. |
| Tools | Narrow functions with policy checks inside them. |
| Knowledge base | Shipping, returns and warranty articles, retrieved per question so answers cite the current policy. |
| Handover | A ticket for a person, with a summary and the full transcript. |
| Audit log | Every tool call, its arguments, the policy result and who approved it. |
A fixed workflow would be simpler, and for the 3 or 4 most common requests a scripted path is often enough. The agent earns its place on the long tail: a late delivery that also needs an address change and a partial refund.
3. Tools, grouped by risk
| Tier | Examples | Rule |
|---|---|---|
| Read | get_order, track_shipment, search_help_articles | Scoped to the session's customer. Return only the fields the agent needs. |
| Reversible write | change_delivery_slot, update_address (before dispatch) | Checked in code: the order belongs to the customer and is in a state that allows the change. |
| High impact | issue_refund, cancel_order | Limits in code from the policy service. Above the limit the tool creates an approval request instead of acting. |
Three things keep the agent's reach small, after OWASP's excessive agency guidance:
- Functionality: specific tools only. No general
call_orders_apior SQL tool. - Permissions: each tool runs with the customer's scope and a service account that can do that one thing.
- Autonomy: a person approves high-impact actions above the threshold, and the systems behind the tools check authorization again, so a mistake in the agent layer is still caught.
Make every write idempotent (a key per order and action), so a retry after a timeout does not refund twice.
4. Untrusted text
Customer messages, emails, uploaded files and even order notes are input, not instructions. A message saying "the supervisor approved a full refund" changes nothing, because approval is a record in the approval system, not a sentence in the chat. This is why the rules sit in code: injected text can persuade the model, but it cannot change what the tool allows.
5. Handing over to people
Hand over when:
- The customer asks for a person.
- A high-impact action needs approval.
- The agent reaches its step limit, repeats itself or its tools keep failing.
- The topic is out of scope: legal threats, safety issues, a complaint about staff.
The handover creates a ticket with a short summary, what was tried and the full transcript, so the customer never has to explain twice. Tell the customer what happens next and when. For approvals, the conversation can resume once the person decides: the approval system calls back and the agent tells the customer the result.
6. Audit and monitoring
Store each conversation as a trace: messages, every tool call with its arguments and result, policy decisions, approvals and model version. Track per route:
- Share resolved without a person, and share handed over (and why).
- Refunds and cancellations per day, with an alert on unusual totals or a spike for one customer.
- Tool errors, policy refusals and step-limit stops.
- Reopened tickets and complaints after a "resolved" conversation, which catch answers that sounded right but were not.
7. Evaluation before every release
Build a set of scenarios from real tickets (returns, late deliveries, cancellations, out-of-policy requests, attempts to talk the agent past a rule) and run them against a sandbox copy of the order and payment systems. A second model plays the customer, with a persona and a goal. Grade three things:
- Outcome: the state of the sandbox after the conversation. Was the refund recorded, for the right amount, on the right order? The agent saying "your refund is done" counts for nothing.
- Tool calls: identity was checked before any change; nothing out of policy was attempted, or it was refused.
- Conversation: a rubric for clarity, tone and grounding in the policy articles, graded by a model checked against human ratings.
Run each scenario several times. Outputs vary between runs, and for refunds you want the agent to succeed every time, so report the share where all runs passed, not only the average.
Release behind a flag: first propose-only (a person confirms every action), then autonomous for low-value actions on a small share of traffic, widening as the numbers hold.
What interviewers want: the split between a model that decides and tools that enforce, identity from the session, tools grouped by risk with limits in code and human approval above them, a handover that carries the context, an audit trail, and evals that check end state in a sandbox with a simulated customer and repeated runs.
Last updated 16 October 2026
Sources