Walk through how you take an AI system from a successful pilot to full production.

Scale in stages, each with an exit check: harden what the pilot cut corners on (security, reliability, monitoring, evaluation), expand to more users or more categories gradually with shadow runs and comparisons, put operations and ownership in place (who watches it, who fixes it, how changes ship), and train users. Production is not "the pilot, bigger". It is the pilot plus everything that keeps it working when you are not watching.

1. Confirm the pilot result

Results against the agreed baseline, per category, with costs and incidents. Decide with the sponsor what scales first (more users, more ticket types, more languages, more automation) and what does not yet.

2. Harden

• Security: final security review, production credentials with least privilege, secrets management, data retention settings.
• Reliability: retries and timeouts, fallbacks when the model or a tool is down (route to people, a simpler model, or a queue), rate limits, and limits on cost per request and per day.
• Evaluation: the test set rebuilt from pilot traffic, run automatically on every change of prompt, model, retrieval or code.
• Monitoring: traces, quality sampling, cost and latency dashboards, alerts.
• Versioning: prompts, models and configuration under version control, with pinned model versions.

3. Expand gradually

• Shadow mode for new categories: the AI produces answers that are compared but not used.
• Then a share of traffic behind a flag (10%, 25%, 50%, 100%), comparing metrics with the control group.
• Increase automation one level at a time: drafts, then auto-answers with hand-off, then actions with limits.

4. Operations and ownership

• A named owner at the customer, and a support agreement with you.
• A runbook: what to do when quality drops, costs spike or a provider has an outage.
• A change process: who may edit prompts or tools, and how changes are tested.
• A plan for model updates. Providers retire models on a schedule (Anthropic gives at least 60 days' notice), so testing replacements is routine work, not an emergency.

5. People

• Train users on what the system does well, where it fails, and how to correct it.
• Give them an easy feedback button, and show them that feedback leads to fixes.
• Update job processes: who reviews what, and how escalations work.

6. Measure value continuously

Report the business metric monthly against the baseline, alongside cost. This is what renews and expands the contract.

7. Hand over

When you step back, the customer's team can run, measure and change it without you: documentation, access, training, and a review rhythm.

What interviewers listen for: staged rollout with measurement, failure handling, and ownership after the FDE leaves.