The demo was flawless. Twelve questions, twelve good answers, everyone impressed. Then it went to fifty internal users and the complaints started: confident answers citing the wrong document, questions about anything in a table returning nonsense, and one memorable case where an employee got an answer sourced from a document they were not supposed to be able to read.
Every one of those failures is a retrieval problem, not a model problem — which is why RAG interview questions are mostly search questions wearing an AI costume. Interviewers know that anyone can wire up an embedding model and a vector database in an afternoon; what they are probing is whether you understand chunking, hybrid retrieval, reranking, permissions and evaluation. This guide covers those questions with the answers that actually score.
What RAG is, precisely
The definition that scores: retrieval-augmented generation retrieves relevant documents from an external corpus at query time and includes them in the model's context, so answers are grounded in a source of truth the model was not trained on.
Then add why: it handles knowledge that changes, keeps proprietary data out of training, allows citations, and is dramatically cheaper than fine-tuning to add facts. Fine-tuning teaches behaviour and format; RAG supplies facts. Saying that distinction cleanly answers a very common follow-up.
Chunking, the most under-prepared question
"How do you chunk documents?" The weak answer is 500 tokens with 50 overlap. The strong answer starts with: it depends on the document structure and on what a good answer needs.
- Fixed-size with overlap — a fine baseline, and it destroys structure.
- Semantic or recursive splitting — split on headings, paragraphs and sentences in priority order so a chunk is a coherent unit.
- Document-aware chunking — respect markdown headings, code blocks, and especially tables, which fixed-size chunking mutilates. Tables are the single most common source of bad RAG answers in enterprise corpora.
- Small-to-big / parent-document retrieval — embed small precise chunks for matching, but return the larger parent chunk for context. This is one of the highest-value techniques to name.
- Contextual retrieval — prepend a short model-generated summary of the document to each chunk before embedding, so a chunk that says "it increased by 12%" still knows what "it" was.
- Metadata on every chunk — source, section, date, and permissions. Without permission metadata you cannot filter, which is how the employee in the opening read a document they should not have.
Embeddings and indexing
- Model choice. Domain fit matters more than leaderboard position; test on your own data. Also consider dimensionality, cost and the maximum sequence length.
- The same model for queries and documents, unless you are deliberately using an asymmetric model trained for it.
- Index type. HNSW is the common default; know that it trades recall for latency and memory, and that the tuning parameters control that trade explicitly.
- Re-embedding cost. Changing the embedding model means re-embedding the entire corpus. Mentioning this unprompted signals production experience.
- Freshness. How new documents get indexed, how updates and deletions propagate, and whether stale chunks can still be retrieved.
Retrieval: why pure vector search is not enough
This is the question that separates candidates who have shipped RAG from those who have built a demo.
Dense vector search fails on exact identifiers — product codes, error codes, names, version numbers — because embeddings capture semantics rather than exact tokens. A user searching for "error TS2345" wants that literal string.
Hybrid search is the standard answer: run dense retrieval and keyword search such as BM25, then fuse the results, commonly with reciprocal rank fusion. Add metadata filters for permissions, recency and document type — and apply permission filters at the retrieval layer, never by asking the model to ignore what it was given.
Query transformation is the other half: rewrite the user's query, expand it into several sub-queries for multi-part questions, or use HyDE, where you generate a hypothetical answer and embed *that* for retrieval. Multi-part questions failing is a very common production complaint and query decomposition is the answer.
Reranking
The cheapest large quality win in most RAG systems, and a reliable interview topic.
Retrieve broadly — say the top 50 — with a fast bi-encoder, then rerank with a cross-encoder that scores each query-document pair jointly and keep the top 5. The cross-encoder is far more accurate and far too slow to run over the whole corpus, which is exactly why the two-stage design exists. Say that reasoning out loud; it is the same candidate-generation-then-ranking pattern as any recommendation system.
Evaluation, where most candidates stall
"How do you evaluate a RAG system?" Separately, in two halves, because a bad answer has two possible causes and you need to know which.
Retrieval metrics — recall at k (did the right chunk get retrieved at all), precision, and MRR or NDCG for ranking quality. If recall at k is low, no amount of prompt work will fix the answer.
Generation metrics — faithfulness or groundedness (is every claim supported by the retrieved context), answer relevance, and context utilisation. Frameworks such as RAGAS formalise these, and LLM-as-judge is the common implementation with the honest caveat that judges are biased and need spot-checking against humans.
A golden set. Fifty to two hundred real questions with known correct answers and known correct source documents, curated from actual user queries rather than invented. Building this is the unglamorous work that separates real systems from demos, and saying so is a strong signal.
Regression testing. Every change to chunking, embedding, retrieval or prompt runs against the golden set before shipping. Our MLOps interview questions guide covers the pipeline this sits in.
Hallucination and grounding
- Instruct grounding explicitly — answer only from the provided context, and say when the context is insufficient.
- Make refusal acceptable. A system that cannot say "I don't know" will always invent, and in an enterprise setting a confident wrong answer is worse than no answer.
- Citations at the sentence level, so users can verify. This is both a trust feature and a debugging tool.
- An answerability check before or after generation, verifying the answer is supported by the retrieved chunks.
- Context ordering matters. Models attend unevenly across a long context and material in the middle is more likely to be ignored, so put the strongest chunks at the edges rather than burying them.
- Do not solve this by enlarging the context window. Stuffing a hundred chunks in degrades precision, costs more and increases latency. Interviewers ask this specifically because long-context models tempt people into it.
Production concerns
Latency budget across each stage, caching of embeddings and of frequent queries, cost per query, chunk-level access control, PII handling in both the index and the logs, and what the product does when retrieval returns nothing relevant — which should be a graceful "I could not find this" rather than an improvised answer.
Where each prep option actually helps
- Build one and break it. Index a messy corpus with tables and near-duplicate documents, and watch what fails. This is worth more than any reading.
- The RAGAS documentation — the clearest framing of the retrieval-versus-generation evaluation split.
- Anthropic's writing on contextual retrieval — a specific, practical technique interviewers increasingly expect you to know by name.
- Published work on hybrid search and reciprocal rank fusion — enough to explain why keyword search did not go away.
- ChatGPT — good for generating a golden set draft and for critiquing your chunking strategy. It will not tell you your retrieval recall is 40%, because only your evaluation set can.
- Greenroom — the spoken layer. Ari, the AI interviewer runs AI-engineering rounds out loud and asks the follow-up about evaluation, which is where most candidates stall. Honest tradeoff: Ari will not index your corpus, so build one.
Our LLM production engineer interview questions guide covers the wider role and LangChain interview questions covers the framework layer.
For the storage and retrieval layer underneath, see our vector database interview questions guide, covering index types, recall tuning and filtering.
Frequently asked questions
What is RAG and why use it instead of fine-tuning?
Retrieval-augmented generation retrieves relevant documents from an external corpus at query time and includes them in the model's context, so answers are grounded in a source of truth the model was never trained on. It is preferred over fine-tuning for facts because it handles knowledge that changes, keeps proprietary data out of training, supports citations, and is far cheaper to update. The clean distinction to state is that fine-tuning teaches behaviour and format while RAG supplies facts.
How do you decide chunk size in a RAG system?
Start from the document structure and from what a complete answer requires rather than a default of 500 tokens with overlap. Prefer recursive or semantic splitting that respects headings, paragraphs and code blocks, handle tables specially because fixed-size chunking destroys them, and consider small-to-big retrieval where you embed small precise chunks but return the larger parent for context. Attach metadata including source, date and permissions to every chunk.
Why is pure vector search not enough for RAG?
Dense embeddings capture semantics rather than exact tokens, so they perform poorly on identifiers such as product codes, error codes, version numbers and names, where the user wants a literal match. The standard answer is hybrid search — running dense retrieval alongside a keyword method such as BM25 and fusing the results, commonly with reciprocal rank fusion — plus metadata filters applied at the retrieval layer for permissions, recency and document type.
What is reranking in RAG and why does it help?
Reranking is a second stage where a cross-encoder scores each query-document pair jointly and reorders the candidates, typically taking the top 50 from fast vector retrieval down to the top 5 that go into the prompt. It helps because a cross-encoder is substantially more accurate than a bi-encoder but far too slow to run across an entire corpus, so the two-stage design buys most of the accuracy at a fraction of the cost. It is usually the cheapest large quality improvement available.
How do you evaluate a RAG system?
Evaluate retrieval and generation separately, because a bad answer has two possible causes. For retrieval measure recall at k, precision and a ranking metric such as MRR or NDCG; if recall at k is low no prompt work will fix the answer. For generation measure faithfulness or groundedness, answer relevance and context utilisation, often with an LLM judge that is spot-checked against humans. Both need a golden set of fifty to two hundred real user questions with known correct answers and source documents, run as a regression suite on every change.
How do you reduce hallucinations in a RAG system?
Instruct the model to answer only from the provided context and to say when the context is insufficient, and make refusal an acceptable outcome so the system is not forced to invent. Add sentence-level citations for verification, run an answerability check against the retrieved chunks, and order context so the strongest chunks are not buried in the middle where models attend least. Enlarging the context window and stuffing in more chunks is the common wrong answer, because it degrades precision while raising cost and latency.