RAG explained: how to make AI answer from your own documents

RAG, short for retrieval-augmented generation, is how you make an AI model answer from your own documents instead of from memory. When a question arrives, the system first searches your documents for the passages most likely to contain the answer, then gives those passages to the language model with an instruction to answer only from them and to cite them. The model never needs to be retrained on your data, the answers stay as current as your documents, and every claim can point to its source. RAG reduces made-up answers a great deal, but it does not remove them, so a good RAG system is also built to say "I could not find that".

RAG exists because of how language models work: a model knows only what it was trained on and what it is shown in the moment, as explained in what a large language model is.

How does RAG work, step by step?

RAG works in two phases, preparing the documents once and answering each question as it comes. The original research described the idea in 2020 (Lewis et al.), and production systems in 2026 usually look like this:

1. Parse: turn PDFs, Word files, web pages and emails into clean text, keeping headings, tables and page numbers.
2. Chunk: split the text into passages of a few hundred tokens that follow the document's structure.
3. Index: store every chunk twice, in a keyword index and as a vector (embedding) that captures its meaning.
4. Retrieve: for each question, search both indexes and merge the two result lists.
5. Rerank: re-score the top fifty or so candidates with a model that reads question and passage together, and keep the best five to twenty.
6. Answer with citations: give the passages to the model, ask for an answer grounded in them, and show which passage supports each claim.

Keyword search matters more than most vendors admit. It finds exact product codes, names and article numbers that meaning-based search misses, and the two approaches fail on different questions, which is why the standard pipeline uses both.

When is RAG the right choice, and when is it not?

RAG is the right choice when the answer lives in documents, the documents change, and people need to see where an answer came from. Typical cases are support answers from product manuals and policies, internal questions over procedures and contracts, and sales answers from a price list and a catalogue.

RAG is the wrong tool in three situations:

• The whole corpus fits in the model's context window. If one query needs twenty pages and they fit comfortably, sending all twenty is often simpler and better.
• The answer is a calculation over rows. "What were our returns in March?" is a database query, so let the model write or choose the query, not read spreadsheets as text.
• The problem is style, not knowledge. If the model knows the facts but answers in the wrong format or tone, examples in the prompt or fine-tuning fix that. Fine-tuning is for form, not for facts.

What breaks a RAG system most often?

What breaks a RAG system most often is not the AI model. It is that the right passage was never retrieved, or was retrieved and then ignored. The common causes are concrete:

• A table or list cut in half by a splitter that counts characters instead of following the document.
• Old documents that outrank new ones. Retrieval ranks by similarity, not by truth or date, so a well-written obsolete policy beats a short correct one. Somebody has to own the document set and retire what is out of date.
• Filters that silently cut recall, such as restricting search to one department's documents inside a vector index, which can drop results without any error.
• Nobody measuring. Without a fixed set of test questions, every change to chunking or prompts is a guess.

How big should a chunk be, and why is Greek different?

A chunk should be about 200 to 512 tokens for collections of short factual answers, and 512 to 1,024 tokens or a whole page for analytical documents, with little overlap between neighbours. Those numbers were worked out mostly on English, and Greek changes them. On OpenAI's current vocabulary Greek needs 2.06 times the tokens of English, and on older vocabularies still used by some embedding models it needs 5.12 times, as I measured in tokenlab. A 512-token chunk therefore holds roughly half as much Greek text as English on a modern vocabulary, and far less on an older one. Size chunks by counting tokens on your real Greek documents, not by copying a number from an English tutorial.

What does contextual retrieval add?

Contextual retrieval adds a short, generated sentence to each chunk before indexing, saying which document it comes from and what it is about, so a passage like "the fee is 3%" stays findable once separated from its heading. Anthropic measured 49% fewer retrieval failures in the top twenty results with it, and 67% fewer when combined with reranking, at a one-off cost of about one US dollar per million document tokens (Anthropic, contextual retrieval). It is the best-evidenced single upgrade after the basic pipeline works.

How do you know that a RAG system works?

You know a RAG system works when it passes a fixed set of real questions with known answers, and keeps passing it after every change. The method is plain:

1. Collect 30 to 100 real questions from emails, tickets or staff, and write down which document answers each one.
2. Measure retrieval separately from answering: did the right passage appear in the top results?
3. Measure the answers for grounding: is every claim supported by a retrieved passage?
4. Re-run the same set after every change to chunking, search or prompts.
5. In production, watch the share of "not found" answers and the questions that get no good passage, because those show where the documents themselves are missing.

Does RAG stop AI hallucinations?

RAG reduces hallucinations but does not stop them. Even commercial legal research tools built on retrieval produced incorrect or misleading answers on roughly one in six benchmark queries or more in a Stanford evaluation (Stanford HAI). The fixes are design choices: require citations, check that each cited passage actually supports its sentence, and give the system an explicit "not in the documents" answer that it is allowed to use.

Frequently asked questions

Do I need a vector database for RAG?

Not always. Keyword search is a strong baseline on its own, and a small or medium document set often works well with a keyword index plus reranking. A vector index helps when people ask in different words from the documents. Measure both on your own questions before buying infrastructure.

Can RAG work with Greek documents?

Yes. Normalise the text consistently, size chunks in Greek tokens, and choose a multilingual embedding model, since a 2026 evaluation found multilingual models ahead of Greek-specific encoders on Greek retrieval, with hybrid keyword and vector search ahead of any single model.

Is RAG better than fine-tuning?

For facts that change, yes. RAG keeps knowledge in documents you can update and cite, while fine-tuning bakes behaviour into the model. Use RAG for what the system must know and fine-tuning, if ever, for how it must answer.

If your team answers the same questions from the same documents every day, send me three of those questions and the documents they come from. The customer-facing version of this is on AI for customer support.