What is RAG (retrieval-augmented generation), and how does it work?

Retrieval-augmented generation (RAG) is a method where a large language model first retrieves relevant passages from an external collection of documents, then generates its answer using those passages as context. Instead of answering only from what the model memorized during training, RAG looks up real source text at query time and writes its response from that text. This grounds answers in documents you control, lets the system cite where each claim came from, and keeps knowledge current without retraining the model. RAG is the architecture behind most trustworthy document question-answering systems, including Samvad AI, which answers only from your own sources and cites the exact passage for every answer.

What is RAG (retrieval-augmented generation)?

RAG (retrieval-augmented generation) is a two-part design: a retriever that finds relevant text and a generator (the language model) that writes the answer. The defining idea of RAG is that the model does not answer from memory alone. It answers from a small set of passages pulled from your documents and placed in front of it at the moment you ask the question.

This matters because a standalone language model has two weaknesses. First, the model only knows what was in its training data, which has a cutoff date and never included your private files. Second, when the model does not know something, it tends to produce a fluent but invented answer. RAG addresses both: it supplies fresh, specific, approved material to answer from, and it makes the source of each answer checkable.

How does retrieval-augmented generation work?

Retrieval-augmented generation works in two phases: an offline indexing phase that prepares your documents, and an online query phase that runs each time someone asks a question.

Indexing phase (done ahead of time):

1. Chunking. Each document is split into smaller passages, often a few hundred words each, so retrieval can return precise sections instead of whole files.
2. Embedding. Every chunk is converted into a vector (a list of numbers) using an embedding model. Chunks with similar meaning end up close together in vector space.
3. Indexing. The vectors are stored in a vector index (a database built for fast similarity search), with a link back to the original chunk text and its source.

Query phase (done per question):

1. Embed the question. The user’s question is converted into a vector with the same embedding model.
2. Vector search. The system finds the chunks whose vectors are most similar to the question vector. These are the passages most likely to contain the answer.
3. Prompt assembly. The top retrieved chunks are inserted into a prompt template along with the question and an instruction such as “answer using only the passages below, and cite them.”
4. Generation. The language model reads that assembled prompt and writes an answer grounded in the supplied passages, ideally with citations pointing to the exact source.

The whole query loop is retrieve, then generate. The model never sees your full document store; it sees only the handful of passages retrieval judged relevant.

Why does RAG ground answers in real documents?

RAG grounds answers because the model is told to write from passages that actually exist in your collection, not from its general training memory. When the retriever does its job, the model has the right facts sitting in its context window and can quote or paraphrase them. That is what makes a source citation possible: the system already knows which chunk produced the answer, so it can show it.

Grounding and citation are related but not identical. Grounding means the answer is built from retrieved source text; citation means the system shows you which source. A citation is only trustworthy if it points to the passage that genuinely supports the claim. We cover that distinction in depth in what source-cited answers actually mean, which explains the philosophy behind verifiable answers rather than this pipeline mechanics primer.

Does RAG stop hallucinations?

RAG reduces hallucinations but does not fully stop them. By placing correct, relevant source text in the prompt, RAG removes the most common cause of invented answers: the model having no real material to work from. But several failure modes remain:

• Bad retrieval. If vector search returns the wrong chunks, the model answers from the wrong material.
• No relevant chunk. If nothing in your documents answers the question, a poorly designed system may answer anyway instead of saying so.
• Misreading. The model can blend two passages or overstate what a passage says.

The fixes are architectural: retrieve well, force the model to answer only from retrieved text, cite the exact passage so a human can verify it, and have the system say plainly when the answer is not in the sources. Samvad AI is built this way. For broader techniques, see our guide on how to reduce LLM hallucinations.

RAG vs fine-tuning: which should you use?

RAG and fine-tuning solve different problems and are often combined. RAG adds external documents to the prompt at query time; fine-tuning changes the model’s internal weights through training. The table below compares them on the dimensions that matter for document question answering.

For knowledge that changes, must be traceable, or must be cited, RAG is usually the right choice. Fine-tuning is better when you need the model to consistently behave or write a certain way. Many production systems use both: fine-tuning for behavior, RAG for facts.

How does Samvad AI use retrieval-first generation?

Samvad AI is a secure, source-cited RAG assistant that answers only from your own documents, deployed on-premise, air-gapped, or hybrid by configuration. Every answer from Samvad AI cites the exact source passage, and the assistant says plainly when an answer is not in your sources. Samvad AI is permission-aware at the role and row level, multilingual (English, Hindi, and more), and keeps a tamper-evident audit log, which is why banks, hospitals, universities, and governments use it for grounded document question answering. To see how this fits a security-first deployment, explore Samvad AI and our wider services.