AI deployment interview questions for Forward Deployed Engineer roles

A forward deployed engineer (FDE) works inside a customer's world to make the product solve that customer's real problem, end to end: understanding the workflow, connecting the product to their data and systems, building what is missing, and staying until it delivers results in production. At AI companies the product is usually a model or an agent platform, so the FDE turns "we have access to a powerful model" into "our claims team closes cases 30% faster", and feeds what they learn back to the product team.

Where the role comes from

Palantir made it famous. Its blog describes two big engineering roles: software engineers ("Devs") who build one capability used by many customers, and forward deployed software engineers ("Deltas") who deliver many capabilities for one customer, measured by the impact on that customer's goals. AI labs and AI startups adopted the model because their products only create value once fitted to each customer's data, workflows and risks.

What the work looks like

• Discovery: sit with the people doing the work, find the costly, repetitive or slow steps, and pick a problem worth solving.
• Scoping: define a pilot with a measurable target and an evaluation set.
• Building: prompts, retrieval, tools and integrations; often a working prototype within days.
• Deploying: security reviews, access to data, connecting to real systems, monitoring.
• Driving adoption: training users, fixing what gets in their way, showing results to sponsors.
• Feeding back: turning repeated custom work into product features, and reporting gaps and bugs.

How it differs

An FDE writes production-quality code like a software engineer, talks to executives and end users like a solutions engineer, and owns outcomes. Unlike a consultant, they represent a product and push the lessons back into it.

What interviews test

• Technical depth: LLM apps, RAG, agents, integration, data, debugging.
• Customer skills: discovery, scoping, managing expectations, explaining trade-offs to non-engineers.
• Judgement: when to build something custom, when to say no, when to escalate to the product team.

A good one-liner: "I own the outcome at the customer, not just the code, and I make the product better with what I learn there."

Turn the wish into one narrow, measurable workflow. Find out what support work costs them most, pick a slice where AI can help with low risk, agree a success metric with a baseline and a test set from their own tickets, define what the AI may and may not do, and time-box the pilot (often 4 to 8 weeks) with a clear decision at the end: expand, change course or stop.

1. Discover before proposing

• Who are the users: customers, support agents, or both?
• Volumes, ticket categories, current handle time, resolution rate, cost per ticket, satisfaction scores.
• Which systems hold the answers: help centre, CRM, order system, policies?
• What goes wrong today, and what does leadership actually want: lower cost, faster replies, 24-hour cover, better quality?
• Sit with agents for a few hours. The real bottlenecks are often not the ones in the request.

2. Pick the slice

Choose where volume is high, answers exist in documents or systems, and mistakes are recoverable. For example: "order status and return-policy questions in English, which are 35% of tickets." Avoid the hardest or riskiest categories (billing disputes, legal complaints) in the first pilot.

Also choose the mode, starting with the least risky that still pays:

1. Draft replies for agents to edit (a person sends).
2. Answer automatically, with hand-off when unsure.
3. Take actions (refunds, changes) with limits and approvals.

3. Agree success criteria up front

• The primary metric with today's baseline: deflection rate, handle time, first-response time.
• Quality bar: accuracy on a test set of, say, 200 past tickets with known good answers; no policy violations.
• Guardrails: escalation rate, satisfaction must not drop.
• Written down and signed off by the sponsor, so the end-of-pilot decision is not a matter of opinion.

4. Define boundaries

Data the system may access, actions it may take, languages, hours, the hand-off rule, and who reviews failures. Bring their security and legal teams in early. They are the most common cause of delay.

5. Plan the pilot

• Week 1 to 2: data access, test set, first prototype measured on the test set.
• Week 3 to 4: draft mode with a few agents; daily review of failures.
• Week 5 onwards: wider use or limited auto-answers; weekly results against the baseline.
• Final review: results, costs, risks and the plan to scale.

Why not "an agent"?

Often a well-grounded answer bot plus a few lookups beats an autonomous agent: simpler, cheaper, easier to trust. Start simple, and add agency where the workflow needs it.

What interviewers listen for: you narrowed the scope, insisted on a baseline, and planned for the decision before writing code.

A use case fits an LLM when the work involves language or unstructured information, a good-enough answer is valuable even if not perfect, mistakes can be caught or are cheap, and there is enough volume to repay the effort. It is a poor fit when a rule, a query or ordinary software can do the job exactly, when every answer must be 100% right with no review, or when the data the model would need does not exist or cannot be shared.

Questions I ask

1. What is the input and output? Emails, documents, conversations, code, free text: good fit. Numbers in a table that need an exact calculation: use code (or let the model call code).
2. Is there a right answer, and can we check it? If correct output can be verified (by code, a person, or a downstream system), we can measure and improve.
3. What does an error cost? A wrong draft that an agent edits costs seconds. A wrong medication dose or a wrong payment is serious, so that needs a person in the loop or a different solution.
4. How is it done today, and how much of it? Volume times time per item times cost per hour gives the size of the prize. Low-volume work rarely repays a careful build.
5. Is the needed knowledge available? In documents, systems or people's heads? If only in heads, the first project is capturing it.
6. Can the data be used? Privacy, contracts, regulation, data residency.
7. Would a simpler tool do? A search box, a form, a rule engine or a report may solve it more reliably.

Good fits (typical)

• Drafting replies, summaries and reports for a person to review.
• Extracting fields from messy documents into a system.
• Classifying and routing requests.
• Search and question answering over internal documents.
• Coding help for internal teams.

Poor fits (typical)

• Exact arithmetic or accounting without tools.
• Decisions with legal effect on people without human review.
• Tasks needing knowledge that exists nowhere in writing.
• Very low volume or one-off work.

How to say no well

Do not just refuse. Offer the better option ("this is a rules problem; a validation step in your existing system will be exact and cheaper"), or a smaller AI piece that does fit ("the model can draft the explanation; the calculation stays in your system").

Check quickly

A one-day prototype on 30 real examples often settles the question faster than a week of debate. If it does well on easy cases and fails on the important ones, that is your answer.

What interviewers want: judgement. Telling a customer where AI will not help builds more trust than promising it will help everywhere.

Answer with a data-flow diagram and the provider's written terms, not reassurance. Show every place data travels and rests (the application, the model provider or cloud platform, logs, caches, vector stores, evaluation sets), and for each one state: who can access it, where it is located, how long it is kept, and whether it is used for training. Then offer the options that close any gaps: a zero-retention arrangement, a cloud deployment in their own account and region, masking, and shorter log retention.

1. Draw the flow

User → the customer's application → (masking) → model endpoint → response → application → logs and traces. Add retrieval stores, caches, file uploads and any third-party tools. Security teams trust a diagram they can question.

2. The model provider's terms, from the documents

Using Anthropic as an example, its documentation says:

• Training: inputs and outputs from commercial products, the API included, are not used to train models by default.
• Retention: the API does not keep prompts and responses by default, except for certain models that require 30-day retention. Zero data retention can be arranged per organisation. Some features that store data by nature (files, batches, code execution containers) have their own retention and may not be covered.
• Cloud platforms: on Amazon Bedrock or Google Cloud, the cloud provider is the data processor. AWS says Bedrock's model providers have no access to customer prompts and completions, and offers encryption and private network connections (VPC, PrivateLink).

Always link the current documents and contract terms. Never quote them from memory: they change.

3. Your application's part

Often the bigger exposure is your own system: prompts in logs, traces in an observability tool, documents in a vector store. State the location, access controls, encryption and retention of each, and how deletion requests are handled.

4. Offer controls

• Masking of personal data before the call.
• Region and residency choices; deployment in the customer's own cloud account.
• Zero-retention arrangements where needed.
• Short log retention, restricted access, and no production data in evaluation sets without anonymisation.
• Single sign-on and role-based access for the application itself.

5. Paperwork

Data processing agreement, sub-processor list, security certifications (for example SOC 2 or ISO 27001 reports from each vendor), and a completed security questionnaire. Have a standard pack ready. It shortens every deal.

Tone

Be precise and honest about what you do not know ("I'll confirm the retention for batch jobs in writing"). A security team's trust is lost by one over-confident answer.

Build the estimate from measurements, not guesses: run the prototype on a representative sample of their real inputs, record tokens, model calls and latency per task, then multiply by their volumes with current prices. Present a range (typical and heavy cases), show the levers that move it (caching, smaller models, batching), and compare the total with the value of the work it replaces.

1. Measure per task

On 50 to 100 real examples, record for each task:

• number of model calls (an agent may make 5 to 30),
• input, output and cached tokens per call,
• other costs: embeddings, reranking, search, tools,
• time to first token, total time, and the slowest 5%.

2. Multiply out

Monthly cost = tasks per month × average cost per task, plus fixed costs (vector database, hosting, monitoring). Do it for a typical month and a peak month. For example: 40,000 tickets a month, 3 model calls each, about 9,000 input and 500 output tokens per ticket, at the chosen model's current prices, plus retrieval costs. Use the provider's current price list; Tecyfy's cost calculator at /prices/cost-calculator does this sum with current prices.

3. Show the levers

4. Latency

Report time to first token (what users feel, with streaming) and total time, as typical and slowest 5%. Name what dominates: retrieval, tool calls or generation. If a target matters ("reply in under 3 seconds"), design for it now, because it may decide the model choice.

5. Compare with value

Cost per task against the cost of the current process: an agent's time per ticket × hourly cost, error costs, delays. A solution at 4 cents per ticket that saves 3 minutes of agent time is an easy decision; at 2 euros per ticket it may not be.

6. State the assumptions

Volumes, average input size, share of complex cases, prices on the date of the estimate. Prices change often, mostly downwards, so promise to re-check before contract renewal.

What interviewers look for: an estimate the customer can check and re-run themselves, and an honest range rather than one optimistic number.

Be honest early: no model, and no person, is right 100% of the time. Reframe the goal from "never wrong" to "better than today, with errors caught before they cause harm". Measure how accurate the current process is, set a target per type of task, and design the system so the cases that must be right are verified by code or reviewed by a person.

1. Say it plainly, early

Anthropic's own guidance on hallucinations says the techniques reduce errors but do not eliminate them, and that critical information must be validated. Setting this expectation in the first meeting avoids a failed project later. Promising perfection is the fastest way to lose a customer's trust.

2. Measure the current baseline

Ask: how accurate is the process today? Sample past work and check it. People make mistakes too, often more than expected: copying errors, missed policy details, inconsistent decisions between staff. Measured human agreement on the task is a realistic ceiling.

3. Separate by risk

4. Design for catching errors

• Verify in code what can be verified: amounts against the system of record, ids that must exist, dates in order.
• Show sources with every answer so users can check quickly.
• Let the system say "I'm not sure" and hand off; measure and tune that rate.
• Make corrections easy, and feed them back into the evaluation set.

5. Agree measurable targets

"Order-status answers: 98% correct on the test set, zero wrong statuses for high-value orders (checked against the order system). Policy answers: 95% pass the rubric, all refunds over 100 euros reviewed." Targets per category, agreed with the sponsor, beat one overall number.

6. Show the trade-off in their terms

"At 95% automatic accuracy with review of the rest, your team handles 60% fewer tickets and quality goes up, because reviewers focus on hard cases." Value usually comes from speed and coverage with safeguards, not from perfection.

What interviewers want

Honesty without killing the deal, a measured baseline, and a system design that makes errors visible and cheap. "Accurate enough, with a safety net" is a plan; "100%" is a promise you cannot keep.

Scale in stages, each with an exit check: harden what the pilot cut corners on (security, reliability, monitoring, evaluation), expand to more users or more categories gradually with shadow runs and comparisons, put operations and ownership in place (who watches it, who fixes it, how changes ship), and train users. Production is not "the pilot, bigger". It is the pilot plus everything that keeps it working when you are not watching.

1. Confirm the pilot result

Results against the agreed baseline, per category, with costs and incidents. Decide with the sponsor what scales first (more users, more ticket types, more languages, more automation) and what does not yet.

2. Harden

• Security: final security review, production credentials with least privilege, secrets management, data retention settings.
• Reliability: retries and timeouts, fallbacks when the model or a tool is down (route to people, a simpler model, or a queue), rate limits, and limits on cost per request and per day.
• Evaluation: the test set rebuilt from pilot traffic, run automatically on every change of prompt, model, retrieval or code.
• Monitoring: traces, quality sampling, cost and latency dashboards, alerts.
• Versioning: prompts, models and configuration under version control, with pinned model versions.

3. Expand gradually

• Shadow mode for new categories: the AI produces answers that are compared but not used.
• Then a share of traffic behind a flag (10%, 25%, 50%, 100%), comparing metrics with the control group.
• Increase automation one level at a time: drafts, then auto-answers with hand-off, then actions with limits.

4. Operations and ownership

• A named owner at the customer, and a support agreement with you.
• A runbook: what to do when quality drops, costs spike or a provider has an outage.
• A change process: who may edit prompts or tools, and how changes are tested.
• A plan for model updates. Providers retire models on a schedule (Anthropic gives at least 60 days' notice), so testing replacements is routine work, not an emergency.

5. People

• Train users on what the system does well, where it fails, and how to correct it.
• Give them an easy feedback button, and show them that feedback leads to fixes.
• Update job processes: who reviews what, and how escalations work.

6. Measure value continuously

Report the business metric monthly against the baseline, alongside cost. This is what renews and expands the contract.

7. Hand over

When you step back, the customer's team can run, measure and change it without you: documentation, access, training, and a review rhythm.

What interviewers listen for: staged rollout with measurement, failure handling, and ownership after the FDE leaves.

Make yourself unnecessary on purpose. Leave the customer's team with an owner, the means to measure (the evaluation set and dashboards), the means to change it safely (versioned prompts and configuration, a tested release process), written knowledge (architecture, decisions, runbooks), and practice: they should have handled at least one change and one incident with you watching before you go.

1. A named owner and team

Someone at the customer is accountable for the system: its results, its costs and its changes. Without an owner, AI systems quietly decay as documents, products and models change around them.

2. Measurement they can run

• The evaluation set, with instructions to run it and a clear pass bar.
• Dashboards for quality samples, cost, latency, errors and hand-off rates, with alerts going to their team, not to you.
• A monthly report on the business metric against the baseline.

3. Safe change

• Prompts, tool definitions, retrieval settings and model versions in version control.
• A release checklist: run the evaluation set, compare with the current version, roll out to a share of traffic, watch the metrics.
• A way to roll back in minutes.

4. Documentation that answers real questions

• Architecture and data flow: what connects to what, with which credentials.
• Decisions and why they were made ("we chose hybrid search because product codes failed with vectors alone").
• Known limitations and out-of-scope requests.
• Runbooks: quality drops, cost spikes, provider outage, a document source changes format, a model is announced for retirement.

5. The changes that will come

• Content: new products and policies, so how documents get into the index and who checks them.
• Models: providers deprecate and retire models on a schedule, so a model upgrade is a routine task with a tested procedure.
• Scope: new requests from users, so how they are evaluated before being added.

6. Practice before leaving

Let their team make a real change (a prompt improvement, a new document source) and handle a simulated incident while you watch and help. Fix the documentation where they got stuck.

7. Stay reachable for a while

A planned check-in after 30 and 90 days catches drift early and often leads to the next project.

What this shows an interviewer

That you measure success by the system still working and improving a year later, not by the launch date. That is what turns a pilot into a lasting customer.

Start from the business goal and the work as it is really done, not from the technology. Interview the sponsor about outcomes, then sit with the people doing the work, map the workflow step by step, measure where time, cost and errors pile up, and score candidate problems by value, feasibility, data readiness and risk. Leave with one or two well-defined use cases, each with a baseline and an owner.

1. With the sponsor (an hour)

• What business result matters this year: cost, speed, revenue, quality, compliance?
• How is it measured today, and what would "success" look like in numbers?
• Constraints: budget, timeline, regulation, data rules, systems that cannot change.
• Who owns the processes, and who will use the result?

2. With the people doing the work (half a day or more)

• Watch real work, not a slide about it. Ask them to share their screen while they handle real cases.
• Map each step: inputs, systems, decisions, hand-offs, waiting time.
• Ask: what is boring, what is slow, what goes wrong, what do you look up repeatedly?
• Collect real examples (anonymised): emails, documents, tickets, forms.

3. Size the opportunities

For each candidate step: volume × time per item × cost per hour, plus the cost of errors and delays. Often a dull step done 10,000 times a month beats an exciting step done 20 times.

4. Score and choose

Pick high value, high feasibility, data available, recoverable errors and a clear owner.

5. Play it back

Write a short summary: the workflow map, the opportunities with numbers, the recommended first use case, the baseline, what data and access are needed, and the risks. Agree it with the sponsor before building.

Pitfalls

• Taking the first request ("we want a chatbot") as the problem.
• Talking only to managers; the workers know where the time really goes.
• Ignoring data access until week three.
• Choosing the most impressive use case instead of the most valuable and achievable one.

What interviewers look for: curiosity about the work itself, numbers to size value, and the discipline to narrow down to one problem worth solving first.

Find out why before pushing harder. Look at the usage data, then talk to the people not using it and watch them work. The usual reasons are that it doesn't fit their workflow, they don't trust its answers, it doesn't save them time, or they fear what it means for their jobs. Fix the specific reason, make the tool part of the normal process, enlist respected users as champions, and show results in their terms.

1. Look at the data

• Who uses it, how often, for what, and where they drop off.
• How often outputs are edited heavily or thrown away.
• Differences between teams: one team using it well is a clue to what works.

2. Talk to non-users

Ask open questions and watch real work. Typical findings:

3. Make it part of the process

• Build it into the existing system and steps, not a separate place.
• Update procedures with managers: for example, "draft with the assistant, then review".
• Make feedback one click, and visibly fix what users report.

4. Champions

Find early adopters respected by their peers. Let them shape improvements and share how they use it. Peer examples persuade better than vendor demos.

5. Show results

Report in users' terms: minutes saved per case, fewer after-hours backlogs, less copying between systems. Report to leadership in business terms against the baseline.

6. Measure again

Set an adoption target with the sponsor (for example, used on 70% of eligible cases), track weekly, and keep iterating.

What not to do

• Mandate usage without fixing the reasons; people find ways around it.
• Blame users. Low adoption is product feedback.

What interviewers want: investigating before acting, empathy for users, and treating adoption as part of the engineering job, not someone else's problem.

It depends mostly on the customer's cloud, procurement, security and data-residency rules, and on which features they need. A cloud platform keeps data, billing, identity and networking inside an account the customer already governs, which often makes security approval and purchasing much faster. The provider's own API usually gets new models and features first. Compare the two on the customer's actual requirements, and keep the code able to switch.

Why a cloud platform often wins in enterprises

• Procurement: spending goes through an existing cloud contract and commitments, no new vendor to approve.
• Data and security: the cloud provider is the data processor under terms the customer has already reviewed. AWS says Bedrock's model providers have no access to customer prompts and completions, and offers encryption, private networking (VPC, PrivateLink) and activity logging in CloudTrail.
• Identity and access: the customer's existing IAM roles, policies and audit.
• Region choice: models served from regions the customer already uses for data residency.

Why the provider's API can win

• Newest models and features often arrive there first; some features (particular tools, betas) may be missing or later on cloud platforms. Check the specific features you need on each platform.
• One integration across clouds for a multi-cloud customer.
• Simpler setup for smaller teams without a cloud governance process.

What to compare, concretely

1. The models and versions available on each platform, in the customer's regions.
2. The features the solution needs: tool use, structured outputs, caching, batch, citations, file handling.
3. Data terms: retention, training use, zero-retention options, who processes the data. Anthropic's data retention documentation, for example, notes that on Bedrock and Google Cloud the cloud provider is the data processor.
4. Quotas and rate limits, and how fast they can be raised.
5. Price, including discounts from existing commitments.
6. Support and incident processes.

Keep it portable

Put model calls behind one small interface, keep prompts and evaluations independent of the platform, and test on both if a switch is likely. Platform choices change with contracts.

A sample recommendation

"You're on AWS with a strict data-residency requirement and an existing commitment, so Bedrock in eu-central-1. We'll confirm the features we need are available there, and design so we can use the direct API for development or switch if a feature we need lags."

What interviewers look for: knowing that the decision is driven by the customer's governance and feature needs, and checking specifics rather than assuming.

Agree the business metric and its baseline before launch, measure it the same way afterwards, ideally against a comparison group, and translate the change into money, time or risk, with costs included and assumptions stated. Report briefly and regularly: the headline number, how it was measured, what it cost, and what comes next.

1. Before launch: baseline and metric

• One primary metric tied to the business goal: handle time per ticket, cases closed per person per day, time from request to answer, error rate, revenue per rep.
• Its current value, measured over enough time to see normal variation (weeks, not days).
• Secondary and guardrail metrics: customer satisfaction, escalations, quality audits, cost of the AI.

2. Measure the effect fairly

• Comparison group: some teams or a share of cases without the tool, at the same time. This separates the tool's effect from seasonal swings or other changes.
• If that is impossible, compare before and after, and note other things that changed.
• Measure from system data, not surveys alone. Surveys are useful for satisfaction, not for time saved.
• Keep the same definitions throughout.

3. Translate into value

Example: handle time fell from 9.0 to 6.5 minutes on 30,000 tickets a month. That is 75,000 minutes, or 1,250 hours a month. At a loaded cost of 40 euros an hour, about 50,000 euros a month of capacity. Subtract the AI's running costs (model usage, infrastructure, support) for the net value. Say whether the capacity is being saved (fewer contractors), redeployed (backlog cleared, faster service) or not yet used.

4. Report it well

• One page: the headline result, the method, the cost, the net value, the risks, and the next step.
• Ranges, not false precision: "between 2.1 and 2.9 minutes saved per ticket".
• Quality alongside speed: faster but worse is not a win.
• A user quote or example to make it concrete.

5. Keep reporting

Monthly or quarterly, the same format. Value that is measured and shown keeps budgets and leads to expansion; value assumed tends to be questioned at renewal.

Pitfalls

• Counting time saved that nobody uses.
• Cherry-picking the best team or the best week.
• Ignoring the cost of review and rework.
• Changing the metric after launch.

What interviewers look for: baselines agreed up front, a fair comparison, honest translation into money, and costs on the same page as benefits.

Go back to the business goal the sponsor agreed, put every request on one list, and score each by value against that goal, effort, risk and readiness (data, access, an owner). Agree the order with the sponsor in the open, so the decision is theirs and visible, say clearly what is not being done now and why, and keep a short path for urgent issues. Never quietly try to do everything.

Why it happens

An AI project attracts requests: operations wants automation, compliance wants more controls, sales wants a demo for a client, IT wants it on their platform. Each is reasonable. Trying to satisfy all of them at once is how pilots stall.

1. Collect requests in one place

One visible backlog, with who asked, what problem it solves, and how they would measure success. Requests made in corridors get written down too.

2. Score them

3. Separate kinds of request

• Must-haves for go-live, such as a compliance control, are not optional and go first.
• Scope growth competes on value.
• Platform changes are planned as their own piece of work.
• Nice-to-haves wait, with a date to revisit.

4. Decide with the sponsor

Bring the scored list to the sponsor and the requesting teams together. The sponsor owns the trade-off; you own the honest estimates. Record the decision and share it, so nobody hears "no" second-hand.

5. Say no well

"Not now, because the go-live depends on X; it's next on the list for the following month" keeps the relationship. Offer something small where you can: a workaround, a read-only view, a clear date.

6. Revisit regularly

Every two weeks or each milestone, re-score with what you learned. Priorities change once the system is live and real usage data exists.

Signs it's going wrong

• The team is doing many things and finishing none.
• Requests go straight to engineers, skipping the list.
• The success metric agreed at the start is no longer mentioned.

What interviewers look for: anchoring on the agreed outcome, transparent prioritisation owned by the customer's leadership, and the confidence to say "not yet" while keeping people on side.

Build it from their own historical work and with their experts. Pull a representative sample of past cases, use what already happened (the reply that was sent, the field that was entered, the decision that was made) as the first draft of the right answer, have their experts correct and agree on a small set, and turn their judgement into a written rubric. Start with 50 to 100 cases in the first week, and grow the set from every failure in the pilot.

1. Mine the history

Most businesses already have answers, just not labelled as such:

• support: the agent's final reply and the resolution code,
• documents: the values someone typed into the system from that document,
• triage: the team the case was finally assigned to,
• approvals: the decision and its stated reason.

These are imperfect (people make mistakes too), but they are a strong starting point.

2. Sample properly

Take cases across categories, difficulty, channels and time, in proportion to real volume, plus deliberate edge cases (unusual formats, angry customers, ambiguous requests). Remove or mask personal data before it leaves their environment.

3. Have experts label and agree

• Run a short labelling session with 2 or 3 of their best people.
• Each labels the same 20 cases independently. Where they disagree, discuss and write down the rule. That conversation reveals the real policy.
• Measure how often experts agree. If they agree only 80% of the time, do not expect the model to reach 99% on that task.

4. Write the rubric

Turn their judgement into criteria that a person or an LLM judge can apply: "Mentions the correct return window", "Doesn't promise a refund before inspection", "Uses the customer's name". Each criterion scored pass or fail.

5. Choose graders

• Exact checks in code for structured fields and classifications.
• An LLM judge with the rubric for free text, checked against the experts' labels before trusting it.
• Expert review of a sample every week.

6. Agree the bar before seeing results

"At least 90% of replies pass all policy criteria, and none contain a wrong order status." Set this with the sponsor up front, so the pilot's outcome is not argued afterwards.

7. Keep it alive

Every failure found in the pilot, and every case where users corrected the AI, becomes a new test. Keep a held-out part that is never used for prompt tuning.

What this shows an interviewer

That you can create measurement where none existed, use the customer's expertise rather than guessing their standards, and protect their data while doing it.

The demo used clean, hand-picked inputs, and production gets the real distribution. So first collect and categorise the failures from production traces, then find which step fails for each category (input handling, retrieval, prompt, model, integration, or a mismatch with what users actually want), fix the biggest category first, and add those cases to the evaluation set so the gap between demo and reality closes for good.

1. Get real failures, not impressions

"Accuracy is poor" is a feeling. Ask for examples, pull traces of 50 to 100 recent requests, and label each: correct, wrong, partly right, refused, too slow. You now have an accuracy number and a list of failure examples.

2. Categorise

Common patterns behind demo-to-production drops:

• Inputs differ: scanned PDFs instead of clean text, other languages, very long emails with long reply chains, abbreviations and internal jargon.
• Content gaps: the answers are not in the documents the system can see, or the documents are out of date.
• Retrieval misses: the right document exists but is not found (chunking, no keyword search, filters, user vocabulary).
• Generation errors: retrieved correctly but the answer ignores or misreads it.
• Integration bugs: wrong customer id passed, a timeout returning an empty tool result, truncated context.
• Expectation mismatch: users ask questions the system was never scoped for, or "correct" means something different to them.

3. Trace each category to a step

For each failure, look at the input, what was retrieved, what the tools returned and the final prompt. Retrieval logs settle most arguments quickly: if the right passage was not in the prompt, no prompt change will fix it.

4. Fix in order of impact

• Parsing and data problems first. They affect everything downstream.
• Then retrieval (hybrid search, better chunking, query rewriting).
• Then prompts and examples for the cases that remain.
• Out-of-scope requests: detect them and hand off politely, and talk with the sponsor about scope.

5. Fix the measurement

The demo set did not represent production. Rebuild the evaluation set from real traffic in proportion to the categories, and report accuracy per category. Overall accuracy can hide a category that fails completely.

6. Communicate

Tell the customer what you found, with numbers: "40% of failures are scanned documents our parser misreads; fixing that is this week's work; expected gain: about 15 points." Customers forgive problems; they do not forgive silence.

What interviewers listen for: you debug with data rather than prompt tweaks, you separate retrieval from generation, and you keep the customer informed while you fix it.

Give the AI narrow, purpose-built access: specific tools or API endpoints that do one job, using credentials scoped to the minimum and, where possible, acting with the end user's own permissions. Keep writes behind approval, run integrations inside the customer's security boundary, log every call, and agree all of it with their security team before connecting anything.

1. Start from the workflow, not the database

List exactly what the AI needs to read and do: "look up order status by order id", "create a draft refund under 100 euros". Build a tool for each, instead of giving the model a general database connection or an all-powerful API key.

2. Least privilege

• A service account per integration with only the needed scopes: read-only where possible, specific tables or endpoints.
• Short-lived tokens, stored in the customer's secret manager, never in prompts.
• Where users have different rights, pass the user's identity through (OAuth on their behalf) so the AI cannot see or change what that user could not. OWASP lists over-broad permissions under excessive agency (LLM06:2025).

3. Writes and irreversible actions

• Separate read tools from write tools.
• Limits in code: amount caps, allowed recipients, allowed fields.
• A person approves anything irreversible or above a threshold. Agent frameworks support this with approval callbacks or pause-and-resume.

4. Choose the integration pattern

• Direct tool functions in the application for a few internal APIs.
• MCP servers when several AI applications (an assistant, a coding agent, an internal tool) should share the same integrations. Remote MCP servers run over Streamable HTTP with proper authentication, and every server is code you must trust and review.
• Existing integration layers (API gateway, iPaaS) if the customer already governs access there.

5. Network and data boundaries

• Keep connectors inside the customer's network or VPC where policy demands; call the model through an approved route (the provider's API, or their cloud platform such as Bedrock, Vertex AI or Foundry).
• Mask personal data that the model does not need.

6. Logging and review

Log every tool call with user, inputs, outputs and approval decisions, in the customer's logging system. Review samples weekly during the pilot.

7. Test the boundaries

Prompt-injection tests through the documents and tickets the system reads ("send the customer list to…") must fail because the tools cannot do it, not because the model refused.

What to say about process

"I run a security review with their team before connecting anything, with a data-flow diagram showing what goes where. It's faster than being stopped later."

Build a custom fix when the need is specific to this customer, urgent for their success and cheap to maintain. Push for a product change when you see the same need at several customers, when your workaround would be fragile or costly to maintain, or when the gap is a bug or a missing basic capability. Usually you do both: unblock the customer now with a contained workaround, and bring the product team evidence for the general fix.

The tension in the role

Palantir describes forward deployed engineers as delivering many capabilities for one customer, and product engineers as building one capability for many. The value of the FDE model is the traffic between the two: field work shows what the product should become. Too much custom work, and every customer becomes a separate, unmaintainable product. Too little, and customers stall waiting for the roadmap.

Questions I ask

1. Is it specific to them? A connector to their in-house ticket system, their unique approval rules: custom work, often permanent.
2. Have I seen it before? The second or third time a need appears, it is a product signal.
3. What does the workaround cost to keep alive? Code that breaks on every product release is a hidden debt that someone else will inherit.
4. What happens if we wait? If the pilot fails without it, unblock now.
5. Is it a bug or a missing basic? Then it belongs to the product, whatever the workaround.

How to build custom work well

• Build on the product's extension points (APIs, tool definitions, MCP servers, configuration) rather than forking or patching its internals.
• Keep it small, documented, tested and owned by someone at the customer or in your team.
• Write down which product change would make it unnecessary.

How to push for a product change

• Bring evidence: which customers, how often, what it blocks, revenue at stake, the workaround and its cost.
• Propose the general version, not this customer's exact version.
• Offer your field implementation as a prototype or a test case.
• Follow up, and tell the customer honestly what is on the roadmap and what is not.

An example answer

"Three customers needed the agent to respect document permissions from SharePoint. I built a filter in the retrieval layer for the first, saw the same request at the second, and wrote a one-page proposal with both implementations and the deals depending on it. The product team built permission-aware retrieval, and I moved both customers onto it."

What interviewers look for: customer obsession balanced with product thinking, and the habit of turning field pain into evidence the product team can act on.

Tell them early, plainly and with data: what the target was, what was achieved, why the gap exists, and what it would take to close it. Then give a clear recommendation, which might be to continue with a specific fix, narrow the scope to where it already works, change the approach, or stop, with the cost and likelihood of each. Never hide a miss or bend the metric after the fact.

1. Don't wait for the final meeting

If weekly results show you will miss, say so as soon as you know. A sponsor who hears bad news early can adjust plans; one surprised at the end loses trust in you.

2. Explain the result honestly

• The agreed target and the measured result, on the agreed evaluation set and in production.
• Results by category: often the system meets the target on most cases and fails badly on one.
• The reasons, from your failure analysis: data quality, missing information, an integration gap, a harder task than expected, user adoption.

3. Separate what was learned

Even a missed pilot answers questions: which parts work, which do not, and why. Present this as evidence, not excuse.

4. Give options with a recommendation

Include the cost, time and confidence of each, and your recommendation with reasons.

5. Protect the relationship

• Acknowledge your part of the miss, if any, without excessive apology.
• Keep the metric as agreed. Changing it to declare success destroys trust.
• Offer something concrete: a smaller go-live, a fix plan, or a better-chosen next use case.

Example opening

"We aimed for 90% correct replies and reached 81%. On order-status and returns we're at 95%; on billing questions we're at 52%, because billing answers depend on data our integration can't yet see. I recommend going live on the first two categories next month, and fixing the billing integration as phase two, which should take about four weeks."

What interviewers look for: honesty, analysis before opinion, options with trade-offs, and a recommendation, not just a report.

Handle it as an incident: contain it first (switch off or restrict the affected path), assess the impact (who saw what, what harm, what obligations), inform the customer quickly and honestly, find the root cause from traces, fix and add tests so it cannot recur, and review what controls failed. Speed and honesty matter more than a perfect first explanation.

1. Contain (minutes to hours)

• Use the kill switch or feature flag: turn off automatic sending, fall back to human review, or block the affected category.
• Stop further damage before investigating in depth.
• Preserve the evidence: traces, prompts, retrieved documents, model and prompt versions.

2. Assess impact

• How many users received similar outputs, and when? Search logs for the pattern.
• What kind of harm: wrong information, offensive content, personal data exposed, an action taken?
• Are there legal or contractual duties: data breach notification, regulator reporting, contract terms on incidents? Involve the customer's legal and security teams early.

3. Communicate

• Tell the customer's sponsor and incident contact quickly: what happened, what you have done, what you are still finding out, when the next update comes.
• Help them prepare messages to affected users, and corrections if wrong information was given.
• Never minimise or guess. "We don't know yet; next update at 15:00" is fine.

4. Root cause

From the traces, find the chain: what input, what retrieval, what prompt, what output, and why no control caught it. Typical causes:

• a new kind of input the system was never tested on,
• a document with wrong or injected content,
• a model or prompt change that skipped the evaluation gate,
• an output check that was missing or too loose.

5. Fix and prevent

• Fix the immediate cause.
• Add the case, and variations of it, to the evaluation and red-team sets.
• Strengthen the control that should have caught it: an output check, human review for that category, a permission.
• Re-enable gradually, with closer monitoring.

6. Review

A blameless written review with the customer: timeline, impact, causes, fixes, and follow-up actions with owners and dates.

What makes this easier

Systems built with kill switches, full traces, versioned prompts and models, and an incident plan agreed before launch. Building those in is part of the deployment, not an extra.

What interviewers look for: containment first, honest communication, data-driven root cause, and fixes that change the system, not only the prompt.

Build an agent-assist tool, not an autonomous bot: it reads the conversation and the customer's records (with the agent's own permissions), retrieves the relevant policies and procedures, and drafts an answer with citations that the human agent checks and sends. Actions such as refunds or card blocks stay in the bank's existing systems, behind their normal controls. Wrap it in strong data protection, audit logging, evaluation and monitoring.

1. Clarify requirements (say these out loud)

• Users: internal service agents, on chat and phone.
• Goals: faster, more consistent answers; shorter training for new agents.
• Constraints: strict data protection and regulation, audit trails, data residency, no automated decisions affecting customers without human review.
• Scale: for example 2,000 agents, 50,000 conversations a day.

2. Architecture

1. Inputs: the live chat or call transcript (speech-to-text for calls), and the customer id.
2. Customer context: read-only tools calling the bank's APIs (accounts, recent transactions, open cases) with the agent's identity, returning only the fields needed.
3. Knowledge retrieval: policies, product terms and procedures in a search index (hybrid search and reranking), with versions and effective dates, filtered by the agent's entitlements.
4. Generation: a prompt with the policy passages, customer context and conversation; output as a draft reply plus citations to the policy passages, and suggested next steps (for example "open a dispute in system X").
5. Agent interface: inside the existing service desktop; the agent edits and sends; one-click feedback.
6. Data protection: masking of card numbers and other sensitive fields before the model; a model hosted in the bank's approved cloud and region, with retention terms agreed (zero retention where required).
7. Logging and audit: every draft, its sources, the agent's edits and what was sent.

3. Safety and controls

• No write tools for the model. Actions happen in core systems with existing controls.
• Customer messages are untrusted input (possible prompt injection); kept clearly separated as data.
• Output checks: no account numbers or other customers' data in drafts; tone and compliance phrases.
• High-risk topics (complaints, vulnerable customers, fraud) flagged for special handling.

4. Evaluation and rollout

• Evaluation set from past conversations with approved answers, graded for policy correctness, completeness and tone, with compliance review of samples.
• Shadow mode, then a pilot team, then gradual rollout, measuring handle time, first-contact resolution, quality audits and agent satisfaction.

5. Operations

• Policy updates flow into the index the same day; superseded versions removed.
• Monitoring of latency (drafts within a few seconds), cost, citation accuracy and edit rates.
• Model upgrades through the evaluation gate.

Trade-offs to mention

Agent-assist is slower to show savings than full automation, but much safer in a regulated setting, and it builds the evaluation data needed to automate simple cases later.

What interviewers look for: clarifying requirements, a human in the loop, permission-aware data access, citations, strong data protection, and a measured rollout.