LLM apps and RAG interview questions for AI Engineer roles

Retrieval-augmented generation (RAG) looks up the passages relevant to a question in your own documents and puts them in the prompt, so the model answers from them. Use it when the answer depends on facts the model was not trained on, or that change: your policies, products, tickets, contracts. Fine-tuning changes how a model behaves (format, tone, a narrow task), not what it knows. And if all your documents fit comfortably in the prompt, you may need no retrieval at all.

How RAG works

1. Index (ahead of time): split documents into chunks, turn each into an embedding, and store them in a search index, often a vector database, usually with a keyword index alongside.
2. Retrieve (per question): embed the question, find the most similar chunks, and often rerank them.
3. Generate: put the best chunks in the prompt with an instruction to answer only from them and cite them.

The name comes from Lewis et al. (2020), who paired a retriever with a generator so answers could draw on knowledge outside the model's weights.

Choosing between the three

The "just put it in the prompt" option

Anthropic's contextual retrieval write-up says a knowledge base under about 200,000 tokens can simply go in the prompt, and prompt caching makes repeating it cheap. That avoids every retrieval failure. It stops working when the documents outgrow the context window, when cost per question matters, or when different users may only see different documents.

Why not fine-tune facts in?

Facts trained into weights cannot be cited, are hard to update (every change means retraining), and can still be misremembered. RAG answers can quote the exact passage and change the moment a document changes. Fine-tuning is also less available than it was. OpenAI's model optimisation guide says it is winding down its fine-tuning platform for new users, so check what your provider offers before planning on it.

What an interviewer wants to hear

That you pick the simplest thing that meets the need, and that you know RAG has its own failure points: bad chunking, missed retrieval, and the model ignoring the context. Those are measured separately, which is the subject of the follow-up questions.

Write it like a brief for a capable new colleague: say who the output is for and why, give clear, specific instructions, separate instructions from data with clear structure (for example XML tags), show 3 to 5 varied examples of good output, and state the format you need. Then treat the prompt as code: version it, test it against an evaluation set, and change it only when the numbers say so.

The parts of a good prompt

1. Role and context. What the product is, who the user is, what the output will be used for. Anthropic's prompting guide notes that explaining why an instruction matters helps the model apply it sensibly to cases you did not foresee.
2. The task, clearly. Specific and explicit. If you want thoroughness, say so; current models follow instructions closely rather than guessing at hidden intentions.
3. Rules and boundaries. What to do when information is missing ("say you don't know and offer a hand-off"), what is out of scope, the tone.
4. Examples. Few-shot examples are the most reliable way to show format and style. Make them varied, so the model does not copy one example's details, and mark them clearly as examples.
5. Input data, clearly separated. Wrap documents, user messages and variables in tags such as <context>, <document> and <input>, so the model never confuses data with instructions.
6. Output format. A schema (better: structured outputs), a template, or a length limit.

A skeleton

You draft replies for the support team at an online shop. Agents edit and send them.
<instructions>
Answer only from the policy and order data below. If they don't cover the question,
say so and suggest a hand-off. Keep replies under 120 words, friendly and plain.
</instructions>
<examples> ... 3 varied examples ... </examples>
<policy> {policy} </policy>
<order> {order_json} </order>
<customer_message> {message} </customer_message>

Production habits

• Keep the stable parts at the start and the variable parts at the end, which also lets prompt caching work.
• Store prompts as versioned templates, not strings scattered through code.
• Run the evaluation set before every change, and log which prompt version produced each output.
• Remove instructions that no longer help. Prompts grow by patching; prune them.

What not to rely on

The prompt is not a security boundary. Rules that must never break (permissions, limits) belong in code.

A strong closing line: "I write the prompt for clarity, then let the evaluation set tell me whether each change helped."

Split documents along their own structure (sections, headings, paragraphs) into pieces that each make sense alone, typically a few hundred tokens, with a little overlap. Keep the document's title and section with every chunk. Then judge the chunking by retrieval results on real questions, not by eye: the right chunk size is whichever finds the needed passage most often.

Why chunking matters

Retrieval returns chunks, not documents. A chunk that is too small loses its meaning ("It must be renewed every 12 months" — what must?). A chunk that is too large mixes several topics, so its embedding matches none of them well, and it crowds the prompt. Most RAG systems that "can't find" an answer are failing here.

A sound default (chunking)

1. Parse properly first. Extract text from PDFs and HTML with headings, lists and tables intact. Bad extraction makes every later step worse.
2. Split on structure. Sections, then paragraphs, then sentences. Never cut a table row or a code block in half.
3. Size and overlap. Start at roughly 300 to 800 tokens with 10 to 20% overlap, then tune on your questions.
4. Carry context. Store the document title, section heading, date and source with each chunk. Prefix the heading to the text you embed.

Contextual chunks

Anthropic's contextual retrieval goes one step further. A model writes a short explanation (50 to 100 tokens) of where each chunk sits in its document ("This chunk is from ACME's Q2 2025 report and covers revenue growth…"), and that text is prepended before embedding and keyword indexing. Combined with keyword search, it cut failed retrievals by 49% in their tests, and by 67% with reranking added.

Special content

• Tables: keep a table whole, or one row per chunk with its column headers repeated.
• Code: split by function or class.
• FAQs, tickets: one question and its answer per chunk.
• Very short documents: do not split them at all.

How to know it is good

• Build 50 to 100 real questions, each with the passage that answers it.
• Measure recall@k: how often the right passage is in the top k results. Compare chunk sizes and strategies on that number.
• Ragas' context recall metric measures the same thing against reference answers.
• Read failures. If the right passage exists but is split across 2 chunks, adjust the splitter. If it is found but ranked 15th, add reranking.

Common mistakes: one fixed size for every document type, losing headings, and tuning chunk size by looking at outputs rather than measuring retrieval.

An embedding model turns a piece of text into a list of numbers (a vector) so that texts with similar meaning land close together. Vector search embeds the question the same way and finds the stored vectors nearest to it, usually by cosine similarity. For more than a few hundred thousand vectors you use an approximate index. HNSW is the usual choice for its speed and recall. IVF builds faster and uses less memory but needs tuning and data to train on.

Embeddings in practice

• Use the same model for documents and questions. Vectors from different models cannot be compared.
• The vector length (for example 1,024 numbers) fixes storage size. Some models allow shorter vectors at a small cost in quality.
• Changing the model means re-embedding everything, so record which model made each vector.
• Some models expect different input types for queries and documents. Follow the model's guidance.

Exact vs approximate search

Exact search compares the question with every vector. It has perfect recall and is fine up to perhaps a few hundred thousand vectors. pgvector does this by default. Beyond that, approximate nearest neighbor search trades a little recall for a lot of speed.

HNSW vs IVF (as pgvector describes them)

Distance

pgvector's operators are <-> (Euclidean), <=> (cosine distance) and <#> (negative inner product). Use the metric the embedding model was trained for. For normalised vectors, cosine and inner product rank the same way.

Choosing a store

If your data already lives in Postgres, pgvector keeps vectors next to the rows, with transactions, joins and permission filters in one query. A dedicated vector database earns its place at very large scale, or when you need built-in hybrid search and multi-tenancy.

What to measure

Recall@k against exact search on a sample of queries, and query latency at your real data size. Then tune ef_search or probes until recall is good enough at an acceptable latency. Remember that vector search finds similar text, not correct text, which is why hybrid search and reranking come next.

Vector search finds passages with similar meaning but can miss exact terms such as an error code, a product id or a name. Keyword search (BM25) finds exact terms but misses paraphrases. Hybrid search runs both and merges the results, so each covers the other's blind spot. A reranker then reads the question with each candidate and reorders them by true relevance, so the best few go in the prompt. Together they are the cheapest large gain in most RAG systems.

Why vectors alone fall short

Ask "What does error TS-999 mean?" An embedding captures "an error code in a technical context", and nearby vectors may be about other error codes. BM25 matches TS-999 exactly. Anthropic's contextual retrieval write-up uses this very example.

Merging the two lists

The scores are on different scales, so simple addition does not work. Reciprocal rank fusion (RRF) uses only rank positions: each document scores the sum over lists of 1 / (k + rank), with k = 60 by default in Elasticsearch. A document near the top of both lists wins, with no tuning needed. Weighted score blending also works, but needs calibrating.

Reranking

• Retrieve broadly, for example the top 50 to 150 from hybrid search.
• A reranker (a cross-encoder model, or a hosted rerank API) scores each pair of question and passage together. That is much more accurate than comparing two separate vectors, but too slow to run over the whole index.
• Keep the top 10 to 20 for the prompt.

What it is worth (Anthropic's measurements)

Starting from a top-20 retrieval failure rate of 5.7%:

• contextual embeddings: 3.7% (35% fewer failures)
• plus contextual BM25: 2.9% (49% fewer)
• plus reranking: 1.9% (67% fewer)

They also found that passing the top 20 chunks did better than the top 10 or 5.

Costs to mention

• Reranking adds latency (a model call per query) and cost. Measure whether the recall gain is worth it.
• Two indexes to keep in sync with the documents.
• Filters (permissions, dates, tenants) must apply to both searches before merging, or one path leaks documents the user may not see.

How to show you would build it

"I'd measure recall@20 on a labelled question set for vector only, BM25 only, hybrid with RRF, and hybrid plus rerank, then keep the cheapest setup that meets the target."

You cannot remove hallucinations, but you can make them rare and catchable. Ground the model in your own sources, allow it to say "I don't know", make it quote and cite what each claim rests on, check the claims in code or with a second pass, and keep a person in the loop where a wrong answer is costly.

1. Ground it

Give the model the facts it needs (retrieval, a database lookup, a tool call) instead of relying on what it remembers. Instruct it to use only the provided documents, not general knowledge. Anthropic's guide lists this as external knowledge restriction.

2. Allow uncertainty

Explicitly tell the model it may answer "I don't have enough information to answer that". Anthropic's guide calls this simple permission one of the most effective fixes. Then make sure your product handles that answer well, for example by offering a search or a human hand-off.

3. Quotes first, then the answer

For long documents (the guide says over about 20,000 tokens), ask the model to extract the exact quotes that bear on the question first, then answer from those quotes only. This keeps it tied to the text.

4. Cite, and retract what cannot be cited

Ask for a citation for every claim, then a check: for each claim, find a supporting quote; if none exists, remove the claim. The Claude API's Citations feature returns the exact source passages for each part of an answer, so your interface can show them and your code can verify them.

5. Check the output

• Deterministic checks: numbers, ids, dates and names in the answer must appear in the sources or the database.
• Consistency: run the same question several times. Answers that disagree suggest the model is guessing (best-of-N verification).
• A second model checks faithfulness: are all claims supported by the context?

6. Design the product around it

• Show sources next to answers so users can verify.
• Use structured outputs for data, so there are no invented fields.
• Send high-stakes answers (medical, legal, financial, actions on accounts) to a person (human in the loop).

7. Measure it

Track a faithfulness score and the rate of "I don't know" on a fixed evaluation set. A falling hallucination rate with rising refusals may just mean the model got timid, so look at both. OWASP lists misinformation among its top 10 risks for LLM applications, which is why this belongs in testing, not only prompting.

What not to say

"Set temperature to 0." It makes output more repeatable, not more true, and several current models no longer accept a non-default temperature at all.

Use the provider's structured output mode with a JSON schema, which constrains generation so the output always parses and matches the schema. Then still validate the values in code, because a valid shape can hold wrong content. Prompting "reply in JSON" alone is not reliable enough for production.

Three levels of reliability

1. Prompt only: "Reply only with JSON like this example." This mostly works, but expect occasional extra text, missing fields or broken quotes, so you need parsing with retries.
2. Tool calling: define a tool whose input schema is your output, and have the model "call" it. Turning on strict: true makes the call follow the schema exactly.
3. Structured outputs: on the Claude API, output_config.format with type: "json_schema" uses constrained decoding, so the response is valid JSON that matches the schema: required fields present, types right. OpenAI offers the same idea with its structured outputs.

Design the schema well

• Keep it flat and small, with clear field names and descriptions.
• Use enums for fixed choices ("status": "approved" | "rejected" | "needs_review").
• Mark fields required, and set additionalProperties: false.
• Give the model a way out: a nullable field or a "cannot_determine" value, so it is not forced to invent one.
• Put a reasoning or notes field before the answer fields if the task needs thought.

Know the limits (Claude API)

• Some JSON Schema features are not supported, such as minimum/maximum and string length limits. Check those in your own validation.
• A refusal or hitting max_tokens can produce output that does not match the schema. Check the stop reason.
• Enum values may come back in different capitalisation, so compare case-insensitively.

Validate in code anyway

Parse the result into a typed model (Pydantic, Zod) and check what the schema cannot: ranges, ids that must exist, totals that must add up, dates in the right order. On failure, retry once with the error message, then send the item to review. OWASP's improper output handling risk applies too: model output that goes into SQL, HTML or a shell must be treated as untrusted input.

Example: extracting invoice fields

Schema: vendor (string), invoice_date (date), total (number), currency (enum), line_items (array). With structured output the JSON always parses. Code then checks that the line items add up to the total and that the date is not in the future. Mismatches go to a person.

What interviewers listen for: the difference between valid format (the schema solves it) and correct content (validation and evaluation solve it).

First measure where the time goes: time to first token (TTFT, what the user feels) and total time (which grows with output length). Then stream the answer, so users see it start at once. Generate fewer tokens, since output is the slow part. Cache long repeated prompt prefixes, use a smaller or lower-effort model where it is good enough, and run independent steps in parallel.

Measure first

Anthropic's latency guide separates baseline latency (the model's processing time) from time to first token. Log both for each step of your pipeline (retrieval, reranking, each model call, tool calls). It is common to find that retrieval or a second model call, not generation, is the slow part. The guide also says to get quality right first, then cut latency, so you know what you are trading.

The levers, roughly by impact

1. Streaming. It does not shorten total time, but users see text within the first second instead of waiting for the whole answer. It is essential for chat.
2. Shorter output. Generation speed is roughly fixed per token, so 150 tokens finish far sooner than 800. Ask for concise answers, use structured output instead of prose where code consumes it, and set max_tokens.
3. Shorter or cached input. Trim the prompt to what the task needs. With prompt caching, a long system prompt or document that repeats across calls is not processed from scratch each time, which reduces both cost and time to first token.
4. Model choice and effort. A small fast model is often good enough for classification, routing and extraction. On models with an effort or thinking setting, lower it for routine requests.
5. Fewer sequential calls. Merge steps where one call can do both. Run independent calls (retrieval and a database lookup, or several subqueries) in parallel.
6. Faster retrieval. Tune the vector index for latency, cache embeddings of frequent queries, and rerank fewer candidates.
7. Answer ahead of time. Precompute answers for the most common questions, or summaries of documents that rarely change.

Perceived latency

Show progress ("Searching 3 sources…"), stream partial results, and return the first useful piece early. Users judge waiting by what they see.

Trade-offs to state

Smaller models and shorter prompts can lower quality, and aggressive caching can serve stale answers. Run your evaluation set after each change, and watch the 95th-percentile latency, not just the average. A slow tail is what users complain about.

Put whole documents in the prompt when they fit comfortably, the same documents serve many questions (so caching makes them cheap), and the question needs the whole document (comparing sections, summarising, finding contradictions). Use RAG when the corpus is far larger than the context window, changes often, or different users may see different documents, or when cost and latency per question must stay low.

Why long context is attractive

• No retrieval step that can miss the answer.
• The model sees connections across the whole document.
• Simpler to build: no chunking, embeddings or index to maintain.

Anthropic's contextual retrieval write-up says that for a knowledge base under about 200,000 tokens you can skip RAG and put it all in the prompt, using prompt caching to keep it affordable.

Why it is not always enough

• Size. Many company knowledge bases run to millions or billions of tokens. No context window holds that.
• Cost and latency. Every question pays for every token in the prompt. Caching cuts the price of repeats, but an uncached 500,000-token prompt is slow and expensive.
• Attention. "Lost in the Middle" (Liu et al.) showed models use information at the start and end of a long context better than in the middle, even models built for long contexts. Newer models do better, but test with your own documents: put the answer at different positions and measure.
• Access control. If users may only see some documents, building a per-user prompt is retrieval anyway.
• Freshness. A changed document breaks the cache and needs re-sending. An index updates one chunk.

A middle way: retrieve documents, not chunks

Retrieve the few most relevant whole documents (or sections) and put them in full. This keeps context intact and leaves out the rest of the corpus. It often beats small chunks for long reports and contracts.

How to decide

The answer interviewers like: "I'd try the long-context version first because it's simpler, measure accuracy, cost and latency on real questions, and move to retrieval only where it fails one of those."

Treat the index as a data pipeline, not a one-off load. Detect changes at the source, re-chunk and re-embed only what changed, delete chunks of removed or replaced documents, and track a version for every chunk (source version, chunking version, embedding model). Then monitor freshness like any other pipeline.

1. Detect changes

• Change events: webhooks or change feeds from the source (a CMS, Confluence, a ticket system), or change data capture from a database.
• Scheduled diffs: compare each document's modified time or content hash with what was indexed.
• Pick a frequency that matches how stale an answer may be. A price list may need minutes, a handbook a day.

2. Update incrementally

• Store a content hash per document. If it is unchanged, skip it.
• For a changed document, delete all its old chunks and insert the new ones in one transaction, or behind a version switch. Updating chunks in place leaves orphans when the chunk count changes.
• Re-embed only the new chunks. Embedding is often the slowest and dearest step.

3. Handle deletions properly

A deleted or unpublished document must disappear from the index quickly. Otherwise the system keeps quoting it. Deletions are the most commonly missed case. Run a periodic reconciliation of the ids in the source against the ids in the index.

4. Version everything

Each chunk stores the source document's id and version, the chunker version, and the embedding model and version. When you change the chunking or the embedding model, build a new index alongside the old one, evaluate it, then switch the application over. Vectors from different embedding models cannot be mixed in one index.

5. Prefer current information in answers

• Keep effective_date and updated_at in metadata. Filter or boost by recency where it matters.
• When two versions of a policy exist, retire the old one rather than leaving both to be retrieved.
• Show the source's date next to the answer.

6. Monitor

• Freshness lag: the time from a source change to an indexed change.
• Counts per source, compared with the source system.
• Failed parses and empty documents.
• Retrieval quality on the evaluation set after each large re-index.

OWASP also advises validating and auditing data before it enters a knowledge base. A pipeline is the right place to check for hidden text, malformed files and documents from untrusted sources.

Summary line for an interview: "Hash, diff, delete-and-replace per document, version every chunk, rebuild side by side when the model changes, and alert when the index falls behind the source."

Put everything that stays the same at the start of the request and everything that changes at the end, mark the end of the stable part as a cache breakpoint, and make sure nothing inside that stable part varies between requests (timestamps, user names, shuffled tool lists). Then check the cache-read tokens in the usage data to confirm it is working.

How the cache works (Claude API)

• The cache covers the whole prefix of the request, in the order tools → system → messages, up to the block marked with cache_control.
• A later request reuses the cache only if its prefix is identical. Anthropic's documentation says a change at one level invalidates that level and everything after it: change a tool definition and nothing is reused.
• There is a minimum cacheable length, which depends on the model: for example 512 tokens on the newest Claude models and 1,024 or more on several older ones.
• Entries live 5 minutes by default, refreshed each time they are read; a 1-hour lifetime costs more to write.
• Reads cost about a tenth of normal input on most models; writes cost more than normal input, so caching pays once a prefix is reused.

Design rules

1. Order: tools, then the system prompt, then long shared documents, then the conversation, then the new user message.
2. Keep the prefix byte-identical. No "Today is 2026-10-16 14:32" at the top; put the date near the end, or at day granularity. Keep tool order fixed. Avoid per-user text in the system prompt.
3. Multi-turn chat: move the breakpoint to the latest turn each time, so each request reuses all earlier turns.
4. Several breakpoints when parts change at different rates: tools and system (rarely change), a document (changes per session), the conversation (grows every turn).
5. Match the lifetime to traffic. If requests for the same prefix come less often than every 5 minutes, the default cache expires between them; consider the 1-hour option or batching related requests.

Check it

Each response reports cache-creation and cache-read input tokens. If cache reads stay at zero, something in the prefix is changing. Compare two raw requests to find it. Track the cache-hit share as a production metric.

Where it pays most

• Agents, which resend tools and a growing history every turn.
• Question answering over the same long document.
• Long system prompts with many examples.

Common mistakes: a timestamp or request id in the system prompt, tools generated in a different order each time, and a prefix shorter than the model's minimum, which silently is not cached.

Summary: "Stable first, variable last, nothing random in the prefix, and I check the cache-read numbers instead of assuming."

A follow-up like "what about the second one?" means nothing on its own, so searching with it retrieves the wrong passages. Rewrite the question into a standalone search query using the conversation ("What is the return policy for the Pro plan?"), retrieve with that, and answer with both the recent conversation and the new passages in the prompt. Then evaluate on real multi-turn conversations, not only single questions.

The problem

Retrieval matches the query against your documents. "Does it work offline?" has no product name; "and in Germany?" has no topic. The model generating the answer can see the conversation, but the search index cannot.

The fix: query rewriting

1. Take the last few turns and the new message.
2. Ask a small, fast model to write a standalone question: "Given this conversation, rewrite the user's last message as a self-contained search query. If it is already self-contained, return it unchanged."
3. Retrieve with the rewritten query (hybrid search and reranking as usual).
4. Generate the answer with the original message, the recent conversation and the retrieved passages.

Log the rewritten query with every answer: when an answer is wrong, it is often the first thing to check.

Variations

• Several queries for broad questions ("compare plan A and plan B" → one query per plan), then merge the results.
• Skip retrieval when the answer is already in the conversation ("can you shorten that?"). A small classifier or the rewriting step can decide.
• Keep earlier passages for a few turns, so a follow-up about the same document does not need a new search.

Managing the conversation

• Keep the last few turns verbatim and summarise older ones, so the prompt does not grow without limit.
• Drop old retrieved passages that are no longer relevant.
• Keep the user's permissions applied to every retrieval, not only the first.

Evaluating it

Single-question test sets miss this entirely. Add conversations of 3 to 5 turns with follow-ups, pronouns and topic changes, and check:

• the rewritten query (does it contain the missing subject?),
• retrieval recall for the rewritten query,
• the final answer.

Cost and latency

Rewriting adds a model call. Use a small model, run it in parallel with other preparation where possible, and skip it when the message is clearly self-contained.

What interviewers look for: knowing that retrieval needs a standalone query, and that multi-turn behaviour has to be tested on its own.

Shortlist a few models using public benchmarks such as MTEB, filtered by what you need (languages, domain, input length, vector size, price, and where it can run), then test them on your own documents and questions, measuring recall@k. Pick the cheapest model that reaches your quality bar, and record which model made every vector, because changing it later means re-embedding everything.

Start from public benchmarks, but do not stop there

The Massive Text Embedding Benchmark (MTEB) covers 8 kinds of embedding task across 58 datasets and 112 languages. Its authors found that no single embedding method was best at every task. That is the key lesson: look at the retrieval scores, in your languages, and expect your own data to rank models differently again.

What to filter on

• Task fit: retrieval scores, not clustering or classification.
• Languages: multilingual models if users ask in one language and documents are in another.
• Domain: some models are trained for code, law, finance or medicine.
• Input length: the model's token limit decides your largest chunk.
• Vector size: longer vectors can be more accurate but cost more storage, memory and search time. Some models let you shorten vectors with a small loss.
• Query vs document input: some models expect a different input type or prefix for queries and documents; get this wrong and quality drops quietly.
• Hosting and data: a hosted API (check its data terms and region) or a model you run yourself.
• Price and speed: for the initial embedding of the whole corpus and for every query.

Test on your data

1. Build 100 or more real questions, each labelled with the passages that answer it.
2. Embed the corpus with each candidate, using the same chunking.
3. Measure recall@k and ranking quality, overall and per category (product names, long questions, other languages).
4. Add hybrid search and reranking to the comparison: a cheaper embedding plus a reranker can beat an expensive embedding alone.

Plan for change

• Store the model name and version with each vector.
• Changing the model means re-embedding the whole corpus; build the new index alongside the old one, evaluate, then switch.
• Vectors from different models cannot be mixed in one index.

What interviewers want: benchmarks to shortlist, your own evaluation to decide, and awareness of the hidden costs (re-embedding, storage) of the choice.

Decide what the feature needs (quality bar, latency, cost per request, context length, input types, data and hosting rules), shortlist two or three models that meet the hard constraints, and run your evaluation set on each. Pick the cheapest and fastest one that clears the quality bar, and keep the choice easy to change, because models are replaced every few months.

Hard constraints first

• Data and hosting: may the data go to this provider? Is it available in the cloud and region you need (the provider's API, Bedrock, Vertex AI, Azure)?
• Inputs: images, PDFs, audio, very long documents.
• Features: tool use, structured outputs, citations, batch processing, caching.
• Context window: enough for your largest prompt plus output.

Then the trade-off: capability, speed, cost

Anthropic's model-choice guide frames it the same way and offers two starting points:

• Efficiency-first: start with a fast, cheap model and move up only where your evaluations show it falls short. Good for high-volume, simpler tasks (classification, extraction, routing).
• Capability-first: start with the most capable model to learn what is possible, then try cheaper models and keep them where quality holds.

The same guide notes that adjusting the effort setting within one model is often a better lever than switching models.

Measure on your own task

• Public benchmarks and leaderboards help you shortlist; they do not tell you how a model does on your documents and users.
• Compare quality per category, cost per successful task (retries and escalations included), and p95 latency.
• Include safety and refusal behaviour, and output length, which changes cost.

Tecyfy's model finder and cost calculator compare current prices and capability scores to help with the shortlist.

Use more than one model

Many products route: a small model for classification and easy questions, a larger one for hard cases. The router's mistakes are part of the evaluation.

Keep it changeable

• Call models through one small interface, with the model id in configuration.
• Pin versions, and record the model with every output.
• Providers retire models on a schedule, so re-running the evaluation on new models is routine work.

What interviewers want: constraints, then evidence from your own evaluation, then the cheapest option that works, not "the best model on the leaderboard".

Expect errors and design for them: retry temporary failures with exponential backoff and jitter (honouring the retry-after header), don't retry errors that will not change (bad requests, spend limits), control your own request rate so you stay under the provider's limits, and have a fallback for when the model is unavailable: a queue, another model or region, or a hand-off to people.

Know the error types (Claude API as an example)

Anthropic's rate limit documentation adds two details worth knowing. Limits are measured in requests, input tokens and output tokens per minute for each model class, using a token bucket that refills continuously, so short bursts can fail even when the minute's average is fine. And a 429 caused by a spend limit has no retry-after header and will keep failing until access returns, so code must tell the two apart rather than retrying forever.

Retry well

• Exponential backoff with random jitter, so many clients do not retry at the same moment.
• A cap on attempts and on total time, then fail clearly.
• Make retried actions safe to repeat (idempotency keys for anything with side effects).
• Most official SDKs retry some errors automatically; know their defaults so you do not double them.

Avoid hitting limits

• A client-side rate limiter or a queue that smooths bursts.
• Batch APIs for work that can wait.
• Prompt caching and shorter prompts, which cut input tokens per minute.
• Ask for higher limits before a launch, with your traffic estimate.

Fallbacks

• Another model or region for the same task, tested on your evaluation set in advance.
• Degrade gracefully: queue the request and notify later, show search results without a generated answer, or hand off to a person.
• A circuit breaker, so a provider outage does not tie up all your workers.

Monitor

Error rates by type, retries, time spent waiting, and how close you are to each limit. Alert before you hit limits, not after.

Summary: "Retry what's temporary, never what's permanent, keep my own traffic under the limits, and always have a plan B."

Log why each response ended, then handle each reason. On the Claude API every response carries a stop_reason, and a cut-off answer almost always has one of a few causes: it hit the max_tokens limit, it filled the context window, the stream broke partway, or your own code cut it (a timeout, a length limit in the UI, a parser). The fix is never one setting; it is making the application check how each answer ended before showing it.

1. Find which cut it

Log stop_reason, output tokens, the request's max_tokens and whether the stream completed, for every call. Then look at the truncated answers:

When streaming, the stop reason arrives near the end in the message_delta event, so a client that stops reading early never sees it. That alone is worth checking.

2. Fix by cause

• max_tokens. Raise it for the features that need long answers, or continue the response: send the partial text back as the assistant turn with a short user message asking to continue, and join the parts. Also ask whether the answer should be that long; a tighter prompt is often the better fix.
• Context window full. The input is too big: long chat history, too many retrieved chunks, large tool results. Trim or summarise history and cap what goes into each request.
• Broken streams. An error can arrive mid-stream after an HTTP 200, so your normal error handling never runs. Handle error events in the stream, retry or show a clear failure, and never treat partial text as a finished answer. For long non-interactive work (over 10 minutes), use streaming or the Batch API, since idle connections can be dropped by networks.
• Your own code. Check gateway and load balancer timeouts against your longest responses, and any length limits in storage or the UI.

3. Never show a silent cut

Whatever the cause, the reader should not get half an answer that looks complete. If a response ends for any reason other than a natural finish, either continue it, retry it, or show it with a clear note that it is incomplete. The same check also catches refusal, which returns HTTP 200 with no error, and pause_turn in server-tool loops.

4. Watch it

Track the share of responses per stop reason, per feature. A rise in max_tokens stops after a prompt change tells you the new prompt makes answers longer, before users notice.

What interviewers want: checking stop_reason on every response, the difference between the token limit, the context window, a broken stream and your own code, handling mid-stream errors, and never presenting a cut answer as complete.

Split the pipeline and test each half on its own. For each wrong answer, check first whether the right passage was retrieved at all. If not, it is a retrieval problem (parsing, chunking, search, filters). If it was retrieved and the answer still contradicts it or adds claims, it is a generation problem (prompt, context order, model). Logging the retrieved chunks with every answer makes this a 5-minute check instead of a guess.

Step 1: log everything

For each question, store the query sent to search (after any rewriting), the retrieved chunk ids with scores, the final prompt and the answer. Without this you cannot tell the halves apart.

Step 2: sort the failures

Take 30 to 50 wrong answers and label each:

Step 3: measure each half

• Retrieval: recall@k against labelled passages. Ragas calls the related metrics context recall (was the needed information retrieved?) and context precision (were the relevant chunks ranked high?).
• Generation: faithfulness, the share of the answer's claims that the retrieved context supports. A low score with good recall means the model is adding things.
• End to end: correctness against a reference answer.

Things that look like generation faults but are not

• Position. "Lost in the Middle" (Liu et al.) found models use information at the start or end of a long context better than in the middle. Put the best chunks first, and do not pad the prompt with weak ones.
• Query mismatch. Users ask "can I get my money back?" while documents say "refund policy". Query rewriting or hybrid search fixes this. It is retrieval, not the model.
• Access filters. The right document exists, but a permissions filter removed it. That is correct behaviour, so the answer should say so.

Step 4: lock it in

Add every fixed failure to the evaluation set, and run it on each change of prompt, chunking or model, so the fault cannot come back unnoticed.

A strong closing line: "I never tune the prompt until I've confirmed the right passage reached it."

Find which calls cost the most, then attack the biggest line first. Cache repeated prompt prefixes, batch anything that can wait, route easy requests to a cheaper model, send fewer tokens in and out, and put limits on every user and feature so no single path can run away.

1. Measure cost per feature

Log input, output and cached tokens, the model and the feature for every call. Usually a few prompts make up most of the bill, and often input tokens dominate (long system prompts, big retrieved contexts, chat history resent each turn).

2. Prompt caching

If many requests share a long prefix (instructions, tool definitions, a document), mark it for caching. On the Claude API a cache read costs 0.1 times the normal input price on most models. Writing costs 1.25 times (5-minute lifetime) or 2 times (1-hour lifetime), so caching pays off once a prefix is reused even once or twice. Keep the cached part stable and put anything that varies (the user's question, timestamps) after it.

3. Batch what can wait

Nightly classification, document summaries and evaluations do not need an answer in seconds. Anthropic's Message Batches API halves the price, and most batches finish within an hour. Other providers offer similar discounts.

4. Route by difficulty

Send classification, extraction and simple questions to a small model, and only the hard ones to a large model. Route by rules (request type) or with a cheap classifier, and confirm on your evaluation set that quality holds for each route.

5. Fewer tokens

• Retrieve fewer, better chunks (reranking helps here).
• Summarise or trim long chat histories.
• Ask for shorter outputs, or structured output instead of prose.
• Remove instructions the model no longer needs. Long prompts tend to grow by accretion.

6. Do not call the model at all

Cache whole answers for repeated questions (exact match, or carefully thresholded semantic match). Use plain code for what does not need a model (regex, lookups, rules).

7. Limits

OWASP lists unbounded consumption among its top 10 LLM risks: per-user rate limits, quotas, input size limits and per-request max_tokens, with alerts on spend. One abusive user or one looping agent should not be able to multiply the bill.

Judge by cost per successful task

A cheaper model that fails more often, triggering retries or human escalations, can cost more overall. Compare options on the same evaluation set, by cost per correct result.

A worked example to offer: "A support bot resent a 6,000-token policy on every turn. Caching it and moving intent classification to a small model cut our input cost by about two thirds with no change in answer quality on our test set." Use your own numbers if you have them.

Enforce permissions in retrieval, before anything reaches the model. Store each chunk with the access rules of its source document, filter every search by the current user's identity, and never rely on the prompt ("don't reveal confidential documents") to protect data. If a chunk is in the prompt, assume the user can see it.

Why it matters

A RAG system flattens many sources into one index. Without filters, an intern's question can retrieve a passage from the board's minutes, and the model will happily summarise it. OWASP lists this under vector and embedding weaknesses (LLM08:2025): unauthorised access, and leaks between tenants sharing one vector store.

How to build it

1. Copy the permissions at indexing time. Each chunk carries metadata: document id, tenant, owner, groups or ACL entries, classification level.
2. Filter at query time. Look up the user's groups from your identity provider and apply them as a filter in the search itself: WHERE tenant_id = :t AND acl && :user_groups in pgvector, or a metadata filter in a vector database. Filter both the vector and keyword searches before merging.
3. Partition hard boundaries. Separate indexes or namespaces per customer in a multi-tenant product, so a missing filter cannot cross tenants.
4. Keep permissions in sync. When a document's sharing changes, update its chunks' metadata. When access is revoked or a document deleted, remove or update its chunks quickly. Stale permissions are a common hole.
5. Check again after retrieval. For sensitive systems, re-check each retrieved document against the source system's permissions before building the prompt. It is a cheap second gate.

Other leak paths

• Caches. An answer cached for one user must not be served to another with different rights. Key caches by permission set, or do not cache sensitive answers.
• Logs and traces. Prompts in logs contain the retrieved text. Restrict who can read them.
• Embeddings themselves. OWASP notes that embeddings can sometimes be inverted to recover text, so protect the vector store like the documents.
• Tools. If the agent can also call APIs, those calls must run with the user's rights, not a powerful service account.

Poisoned documents

Anyone who can add a document to the index can add instructions ("ignore previous rules…"). Validate what is indexed, prefer trusted sources, and treat retrieved text as data, not instructions.

How to test it

Create test users with different rights and ask questions whose answers sit in documents they must not see. The expected result is "I can't find that", every time. Keep these tests in your evaluation set, and log every retrieval with the user and chunk ids so you can audit later.

Most bad RAG over PDFs is bad parsing. Use a parser that keeps layout, headings and tables, run OCR on scanned pages, turn tables into structured text (or keep them whole), and describe charts and images in text with a vision-capable model so they can be searched. Keep a link from every chunk back to its page, so answers can show the source and, when needed, the page itself can be sent to the model.

Why plain text extraction fails

• Two-column pages come out with lines interleaved.
• Tables become streams of numbers with no column names.
• Scanned pages produce no text at all.
• Charts and diagrams, often the most important content, disappear.

A sturdier pipeline

1. Classify pages: digital text, scanned, or mixed.
2. Parse with layout: a layout-aware parser, or a document AI service, that returns headings, paragraphs, lists and tables as separate elements.
3. OCR scanned pages, and check its confidence; flag low-quality pages for review.
4. Tables: convert to Markdown or HTML with headers kept. For large tables, one chunk per row group with the column headers repeated. For numeric questions, consider loading tables into a database and answering with a query.
5. Charts and images: ask a vision model to describe each one (title, axes, trend, key values) and index the description, linked to the page.
6. Chunk by structure, with the document title and section heading in each chunk.

Sending pages to the model directly

For a small number of documents, a model that reads PDFs can take the file itself. Anthropic's PDF support processes text and each page as an image, so it can read charts and tables, within limits: the documentation lists 600 pages per request (100 when the context window is under 1 million tokens), a request size limit, and warns that dense pages can fill the context window first. A good pattern is to retrieve the right pages with search, then send those pages as PDF or images for the final answer.

Evaluate on the hard parts

Build questions whose answers sit in tables, charts and scanned pages, and measure retrieval and answer accuracy for each kind separately. Overall scores hide the fact that every chart question fails.

Operational notes

• Keep the original files and page numbers; show the page in the answer's source link.
• Re-parse when you change parsers, and compare retrieval before switching.
• Watch costs: vision descriptions and page images use many tokens, so do them once at indexing time.

Summary: "Parse for structure, describe what's visual, keep page links, and test tables and charts on their own."

Give the model the schema and business definitions it needs, have it write a query, and then never trust the query: run it with a read-only role that can see only allowed tables and rows, with timeouts and row limits, after validating it in code. Show users the query or a plain explanation of it, and measure accuracy on a set of real questions with known answers.

Why it is hard

• Accuracy: the model must know which tables, joins and definitions to use. "Revenue" might mean gross or net, with or without refunds. Wrong but plausible numbers are worse than an error.
• Security: model-written SQL is untrusted output (OWASP LLM05). A user, or text injected into data the model reads, could ask for other tables, other customers' rows, or a DROP TABLE.
• Cost: a careless query can scan a whole warehouse.

Accuracy

1. Curated schema context: only the tables and columns needed, with descriptions, units, and how tables join.
2. A semantic layer: define metrics (revenue, active users) once, and have the model use them rather than re-deriving them each time.
3. Examples: pairs of real questions and correct SQL, chosen by similarity to the question.
4. Self-check: run the query, look at the result's shape (empty? suspicious totals?), and let the model fix errors once.
5. Clarify: if a question is ambiguous ("last quarter": calendar or fiscal?), ask.

Safety

• Read-only database role with access to approved schemas or views only. No write permission at all.
• Row-level security so each user sees only their own organisation's data, enforced by the database, not the prompt.
• Validate before running: parse the SQL, allow only SELECT, check tables against an allow-list, add a LIMIT.
• Timeouts and cost limits per query.
• Log questions, queries and results for audit.

An alternative design

For narrow use cases, skip free-form SQL: let the model fill parameters (metric, filters, date range) through structured output, and run queries your team wrote. Less flexible, much safer and easier to test.

Evaluate

A set of real questions with correct answers. Compare results, not SQL text, since many different queries are correct. Track accuracy per question type (joins, time ranges, ratios), and keep the questions that failed.

Show your work

Display the query or a plain summary ("Sum of net revenue from orders in Q3, excluding refunds"), so users can spot a wrong interpretation.

A good summary: "The model drafts, the database enforces: read-only, row-level security, limits, and validation before anything runs."

"Nobody changed the code" rarely means nothing changed. An LLM feature also depends on the model, the prompts and settings kept outside the code, the retrieved documents, the tools it calls and the questions users ask. First split the problem in two: replay last week's good requests through today's system. If the answers got worse, the system changed. If they did not, the inputs changed.

1. Pin down what "dropped" means

Which metric moved, and when? A thumbs-down rate, escalations to people, an online judge score, or complaints. Find the day it started and whether it hit every kind of request or one segment (a customer, a language, a document type, a feature). A change that began at a precise hour points at a deployment or config change; a slow slide points at data.

2. Replay to separate system from inputs

Take a sample of requests from before the drop, with the answers you logged then, and run them through today's pipeline.

• Answers changed for the worse: something in the system changed. Go to step 3.
• Answers still fine: the system is the same, and users are now asking different things. Go to step 4.

This only works if you log each request with what it used: prompt version, model ID, retrieved document IDs, tool results.

3. If the system changed, check everything that is not code

Compare traces from before and after for the same kind of request. The difference is usually visible: other documents retrieved, a tool returning empty results, a truncated answer.

4. If the inputs changed

New users, a new product line, a new language, or a seasonal question the system was never tested on. Read a sample of the failing conversations. You will usually find a cluster of questions the evaluation set does not cover.

5. Fix it and close the gap

• Fix the cause: pin the model, roll back the config, rebuild the index properly, add the missing documents.
• Add the failing cases to the evaluation set, so this failure is tested from now on.
• Monitor the inputs too: retrieval scores, the share of "I don't know" answers, request topics and lengths. Production monitoring exists to catch exactly this kind of drift.

What interviewers want: the replay test that separates system changes from input changes, a list of the parts that change without a code deploy (model, prompts, index, tools, users), and logging rich enough to compare before and after.

Break each request's time into its parts and see which one grew: your own steps before the model call (retrieval, tools, database), waiting and retries around the call, the time to the first token, and the time spent generating output. A sudden jump overnight usually has one cause, such as a deploy, a config change, a cache that stopped working, longer outputs, or retries against a busy API, and the breakdown shows which.

1. Measure the parts, not just the total

For each request you want a trace with timings for:

• Retrieval and other preprocessing.
• Each model call: when it was sent, time to first token (TTFT), time to the last token.
• Input tokens, cached input tokens and output tokens.
• Retries and their status codes.
• Tool calls, for an agent.

Compare the p95 of each part before and after the jump. If you have no breakdown yet, add it first: guessing at latency causes wastes days.

2. Read the pattern

3. The prompt cache is a common culprit

Prompt caching saves time and money on the repeated start of a prompt, but only when that start is identical. A small change can break it for every request: a timestamp or user name added to the system prompt, tools sent in a different order, or a new tool definition (changing tool definitions invalidates the whole cache). Check cache_read_input_tokens in the usage of recent responses. If it dropped to zero overnight, find what made the prefix differ.

4. Retries hide inside the average

The SDKs retry rate-limit and server errors with backoff automatically. A request that succeeds on its third try looks like one slow request in your logs unless you record retries. If 429s rose, look at traffic growth, a new batch job sharing the same key, or a lower rate limit.

5. Fix and protect

• Fix the cause: restore the stable prompt prefix, cap output length, move a batch job to its own key or to the Batch API, cache retrieval results.
• Stream responses so users see the answer start, even when the total time is long.
• Alert on p95 TTFT, output tokens per request and cache hit rate, not only on total latency.

What interviewers want: a per-request breakdown before any fix, TTFT versus generation time, the prompt cache and retries as named suspects, and monitoring on the parts.

Split the bill until the growth sits in one place. Cost is requests times tokens per request times the price of each kind of token, so ask which of those grew and for which feature, model and customer. When traffic rose 20% and cost rose 200%, the money usually went to more tokens per request, a pricier model, broken prompt caching, retries and loops, or traffic that is not your users.

1. Get the numbers to attribute cost

Tag every model call with the feature, customer or tenant, model, prompt version and environment, and log its usage: input tokens, cache write tokens, cache read tokens and output tokens. With those, the bill can be broken down like any other cost report. Without them, you are guessing, so add them first.

2. Break it down

3. The usual causes

• Growing context. A chat that resends the whole conversation every turn costs more on each turn. Average input tokens per call creep up as conversations get longer, with no change in traffic.
• More retrieved text. Someone raised the number of chunks from 5 to 20 to fix a quality issue.
• Broken prompt caching. On the Claude API a cache write costs 1.25 times the normal input price (5-minute cache) and a read about a tenth or less. If something now changes the start of the prompt on every request (a timestamp, a reordered tool list), you pay for writes and never read them back.
• Model choice. Prices between models of the same family differ many times over; Opus 5.5 lists $4 per million input tokens against Haiku 5.5 from $0.10. A config change or a fallback path can move traffic to the expensive one.
• Agents. A loop that retries a failing tool, or an agent that reads whole files, can use many times the tokens of a normal run.
• Abuse. A public endpoint without per-user limits can be used as a free model by someone else.

4. Fix and keep it fixed

• Fix the cause you found: trim or summarise history, restore the stable prompt prefix, cap retrieved text, route simple requests to a smaller model, put a step and token budget on agent runs.
• Move work that does not need an instant answer (nightly classification, backfills) to the Batch API, which is half price on the Claude API.
• Add per-user rate limits and quotas, and limits on input size.
• Set budgets and alerts per feature on daily spend and on tokens per request, so a change shows up in a day, not on the invoice.

What interviewers want: cost as requests times tokens times price, attribution by feature and model, the specific suspects (history growth, caching, model switches, loops, abuse), and alerts that catch the next spike early.