Question 1hard#System Design#RAG#LLM
Design an assistant that answers employees' questions from 2 million internal documents (wikis, tickets, PDFs, policies), where each employee may only see documents they already have access to.
Say your answer out loud first, the way you would in the room.
In shortBuild a retrieval pipeline that keeps each chunk's source permissions next to it, filters by the asking user's permissions inside the search, combines keyword and vector search with a reranker, and has the model answer only from the retrieved text with citations.
Build a retrieval pipeline that keeps each chunk's source permissions next to it, filters by the asking user's permissions inside the search, combines keyword and vector search with a reranker, and has the model answer only from the retrieved text with citations. Keep the index fresh, including permission changes, and measure retrieval and answer quality separately. Permissions are the part that cannot go wrong, so design them first.
1. Clarify
- Which sources, how often they change, and how permissions work in each (groups, spaces, per-document sharing).
- Latency target (a few seconds) and expected questions per day.
- Answers or links? Must every answer cite sources?
- Languages, file types (scanned PDFs need OCR), and how sensitive the content is.
2 million documents is far beyond what fits in a prompt, so retrieval it is.
2. Ingestion
connectors ──▶ parse and clean ──▶ chunk ──▶ add context ──▶ embed ──▶ index
(wiki, tickets, drive) (PDF, HTML, OCR) (few hundred tokens) + BM25
└──────────── permissions (ACL) per document, synced on every change ─┘
- Chunk by structure (headings, sections, ticket comments), a few hundred tokens each, with the document title and section path attached.
- Contextual chunks: a short generated note per chunk saying where it sits in the document ("Section 4 of the 2026 travel policy, on hotel limits") improves retrieval noticeably.
- Store with each chunk: document ID, source, last modified, and the list of groups and users allowed to read it.
3. Query time
- Resolve the user's groups from the identity provider (cached briefly).
- Hybrid search with the permission filter applied inside the query: vector search for meaning, BM25 for exact terms such as ticket numbers and error codes. Filtering after retrieval would leave too few results and risks leaks.
- Rerank the top 100 or so candidates and keep the best 10 to 20.
- Generate with instructions to answer only from the provided passages, cite them, and say when the answer is not there.
4. Freshness and deletion
- Incremental sync from each source's change events or modified timestamps; permission changes processed with the same priority as content changes.
- Deleted or restricted documents removed from the index within minutes, with a full reconciliation weekly.
5. Security
- Documents are untrusted input: a page can contain hidden instructions. Keep retrieved text clearly separated from instructions, and give the assistant no actions it does not need.
- Log every retrieval (user, query, document IDs) for audit.
- Test that users cannot retrieve what they cannot open in the source system.
6. Quality
- Retrieval: a labelled set of questions with the documents that answer them; track recall at the number of chunks you pass.
- Answers: faithfulness to the passages and correctness, graded by a calibrated judge and spot checks.
- Production: thumbs up or down, "no answer found" rate, and the queries with no good results, which show content gaps.
What interviewers want: permissions stored per chunk and enforced inside retrieval, structure-aware chunking with context, hybrid search and reranking, grounded answers with citations, freshness including permission changes, prompt-injection awareness, and separate retrieval and answer evaluation.