What Is RAG? How AI Answers From Your Own Documents

Ask a chatbot about your company's refund policy and you'll get a confident answer that has nothing to do with your company. The model has never seen your policy: it learned from a snapshot of public text that stops at a cut-off date. Retrieval-augmented generation, almost always shortened to RAG, is the standard fix. Before the model answers, a search step finds the relevant passages in your own documents and hands them over with the question. This guide explains what RAG is in plain English, how it works, when it beats fine-tuning or pasting everything in, how to use it without code, and where it still goes wrong.

What RAG means in plain English

RAG is an open-book exam for an AI model. Without it, the model answers from memory: whatever patterns it absorbed during training. With it, the model is first handed a few pages from the right book, then asked to answer using those pages. The name lists the three parts in order: retrieval (find relevant text), augmented (add it to the prompt) and generation (write the answer).

The term comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, who paired a language model with a searchable index of Wikipedia and got more accurate, more specific answers than from the model alone. The idea spread because it solved three problems at once:

• Private knowledge: the model can use your manuals, contracts and wikis without being retrained on them.
• Fresh knowledge: update a document and the next answer reflects it.
• Traceable answers: because the answer comes from specific passages, the tool can show you which ones.

It also helps with AI hallucinations. A model with the facts in front of it has less reason to invent them, although, as you'll see below, "less" is not "never".

How RAG works, step by step

Every RAG system follows the same pipeline: part of it happens in advance, part every time someone asks a question.

Preparation (done once, then kept up to date):

1. Collect the documents: PDFs, web pages, help articles, transcripts.
2. Split them into chunks: passages of a few hundred words, because a model works better with five relevant paragraphs than with fifty whole files.
3. Create embeddings: each chunk becomes a long list of numbers representing its meaning. Passages about similar things get similar numbers, even when they use different words.
4. Store them in an index: usually a vector database, often alongside a classic keyword index.

At question time:

1. Search: the question is turned into an embedding too, and the system finds the closest chunks. Many systems also run a keyword search, which catches exact terms such as product codes.
2. Select: the best matches, often 5 to 20 chunks, are picked, sometimes re-sorted by a second model called a reranker.
3. Build the prompt: the chunks are inserted into the prompt with instructions such as "answer using only these sources".
4. Generate: the model writes the answer, ideally citing the chunks it used.

The hidden prompt the model receives looks something like this:

That's the whole trick. RAG doesn't change the model at all; it changes what the model sees. That's why it's usually described as one part of context engineering: deciding what goes into the model's context window for each request.

RAG vs fine-tuning vs a long context window

RAG isn't the only way to give a model your knowledge, and each option suits a different job.

• Fine-tuning retrains the model on examples. It's good for teaching a behaviour, such as a house style or a fixed output format. It's poor at teaching facts: they get blended into the model, can't be cited and go stale the moment something changes.
• A long context window means pasting everything in. For a small, stable set of documents this is often the simplest option. Anthropic's guidance is that if a knowledge base is under about 200,000 tokens (roughly 500 pages), you can include all of it and use prompt caching to keep the cost down.
• RAG retrieves only what's relevant to each question. It wins when the material is too big to paste in, changes often, needs per-user permissions or must come with citations.

A rule of thumb: fine-tuning for how the model should answer, RAG for what it should know, and a long context for small jobs. Not sure how big your material is? Paste a sample into our token counter and multiply.

Where you're already using RAG

You've probably used RAG this week without seeing the label. Any AI feature that reads something before answering is doing a version of it:

• AI search: Perplexity, ChatGPT search and Google's AI Overviews search the web, read the top results and write an answer with links. Our guide on how to use Perplexity shows how to get the most from it.
• Chat with your files: uploading PDFs to a ChatGPT or Claude project or a Gemini Gem. With large files, the tool searches for the relevant parts rather than reading every page each time.
• NotebookLM: Google's research notebook answers only from the sources you add and shows a citation for every claim. It's the clearest consumer example of RAG.
• Work assistants: Microsoft 365 Copilot and similar tools retrieve from your email, files and chats, filtered by what you're allowed to see.

So when an AI search tool gets something wrong, the culprit is often the retrieval (a weak or outdated page), not the model's reasoning. Check its sources first.

How to set up RAG without writing code

For most people, "doing RAG" doesn't mean building a vector database. It means choosing a tool that does the retrieval for you and feeding it good sources.

• Pick the right container: NotebookLM for research across many documents (our NotebookLM guide walks through it), a project in ChatGPT or Claude for ongoing work, or a custom GPT with knowledge files if you want to share the assistant.
• Curate, don't dump: ten current documents beat 200 that include three versions of the same policy. Delete superseded files; the system can't tell which version is official.
• Make documents easy to search: clear headings, one topic per file, descriptive names ("2026 expenses policy – UK", not "final_v3.pdf") and dates inside the text. Scanned pages without readable text are invisible to most tools.
• Write instructions for the assistant: tell it to answer only from the files, cite them and admit gaps. Our system prompt generator gives you a structure to start from.

A solid set of instructions for a document assistant:

Prompts that get better answers from RAG tools

Retrieval is driven by your question: the system searches using what you type, so a vague question retrieves vague passages. A few habits help:

• Use the documents' vocabulary: if the handbook says "annual leave", that term retrieves more reliably than "vacation days".
• Ask one thing at a time: a question with three parts splits the search three ways. Three questions get three good retrievals.
• Ask for quotes: requesting the exact sentence makes the answer easier to verify and harder to embellish.
• Ask what's missing: a good RAG answer tells you when the sources are silent. Make that explicit.

This prompt works in almost any tool that answers from your files or the web:

Why RAG answers still go wrong

RAG reduces errors but adds a new place for them to happen: the search. A wrong answer is usually one of these:

• The right passage wasn't retrieved: the search missed it, so the model answered from a near miss or from memory. This is the most common failure.
• The passage was cut in half: "Refunds are available within 30 days" landed in one chunk, "except for sale items" in the next.
• The source was outdated: an old version was retrieved and quoted faithfully.
• The model blended sources: it mixed retrieved facts with its training data, or merged two sources into one claim.
• The citation doesn't support the claim: the linked passage says something slightly different. Citations show where to check; they don't prove the answer is right.

When something looks off, ask the tool to show its working:

Building a RAG app: the decisions that matter

If your team is building a RAG system, the model is rarely the weak link. Quality comes from a handful of unglamorous choices:

• Chunking: too small and chunks lose context, too big and they dilute the match. Splitting by headings and sections usually beats splitting every N characters, and a little overlap stops rules being separated from their exceptions.
• Context for each chunk: "the limit is £500" is useless without knowing which policy it belongs to. Anthropic's contextual retrieval adds a short note on where each chunk sits in its document before indexing. In their tests, combined with keyword search, it cut failed retrievals by 49%, and by 67% with reranking.
• Hybrid search: embeddings handle "how do I get my money back?"; keyword search handles "error code E-4012". Use both.
• Metadata and permissions: store dates, owners and access rights with every chunk, so you can filter by recency and never show someone a passage they shouldn't see.
• Evaluation: collect 50 to 100 real questions with known answers and measure two things separately: did retrieval find the right passages, and did the answer stay faithful to them?

More advanced setups let the model decide when and what to search, or run several searches in a row. That's where RAG starts to overlap with AI agents: retrieval becomes one tool among others rather than a fixed step.

What RAG can't do

RAG is the most practical way to connect AI to real knowledge, but it's often oversold. Limits worth knowing:

• It doesn't eliminate hallucinations: it reduces them. Models still misread and overstate sources, especially when retrieval comes back weak.
• It can't answer questions about everything at once: "How many of our 3,000 contracts have a penalty clause?" needs every document read, not the top ten passages. Counts, totals and complete lists need a database query or a document-by-document pass.
• Your documents set the ceiling: wrong or outdated sources produce wrong or outdated answers, delivered with the same confidence.
• It supplies facts, not skills: retrieved text won't change how the model writes or reasons. Style comes from instructions, examples or fine-tuning.
• Retrieved text can carry attacks: a web page or shared file can hide instructions aimed at the model, known as prompt injection. Treat retrieved content as data, not commands.
• It needs upkeep: documents change and indexes go stale. A RAG system is a library that needs a librarian, not a one-off install.

The core idea is worth keeping even if you never build one: a model answers well when the right facts are in front of it, and RAG is the machinery for finding them. Give it good sources, ask precise questions and check the citations on anything that matters.

References

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv)
• Anthropic — Introducing Contextual Retrieval
• Prompt Engineering Guide (DAIR.AI) — Retrieval Augmented Generation (RAG)