How to evaluate RAG accuracy: metrics and methods for enterprise
Evaluate RAG accuracy with retrieval metrics (precision, recall, hit rate) and generation metrics (faithfulness, citation correctness) against a golden test set.
Evaluating RAG accuracy means measuring two things separately: whether the system retrieves the right source passages, and whether it generates a faithful, well-cited answer from them. You measure retrieval with precision, recall, context relevance, and hit rate. You measure generation with faithfulness (groundedness), answer relevance, citation correctness, and refusal-when-unknown. Both halves are scored against a golden test set of real questions whose correct sources you already know. This guide is about measurement and QA, not mitigation: for fixing failures once you find them, see how to reduce LLM hallucinations.
What does “RAG accuracy” actually mean?
RAG accuracy is not a single number. A retrieval-augmented generation (RAG) system has two stages, and a wrong answer can come from either one, so you must score them independently. If retrieval fails, the model never sees the right text and cannot answer correctly no matter how capable it is. If retrieval succeeds but generation is sloppy, the model can still hallucinate, misattribute a citation, or answer a question your sources do not cover. Measuring only the final answer hides which stage failed. If you need the underlying mechanics, start with what RAG is.
How do you build a golden test set for RAG?
A golden test set is a curated list of real user questions, each paired with the known-correct source passages and an ideal answer. It is the ground truth that every other metric is measured against, so build it before you measure anything.
- Collect real questions. Pull from actual user logs, support tickets, and subject-matter experts, not invented examples. Include edge cases and questions your documents do not answer.
- Label the correct sources. For each question, have an expert mark the exact passages that contain the answer. These labels drive retrieval recall and precision.
- Write an ideal answer. Record the correct, grounded answer and the citations it should carry.
- Add “unanswerable” cases. Include questions with no supporting source. The right behavior is a clear refusal, and you cannot measure refusal without them.
Aim for breadth across the topics, phrasings, and languages your users actually use. A few hundred well-labeled items beats thousands of careless ones.
How do you measure retrieval quality?
Retrieval quality measures whether the system fetched the passages that contain the answer, before any text is generated. The core retrieval metrics are:
- Precision: of the chunks retrieved, how many are actually relevant. Low precision means the model is fed noise.
- Recall: of all relevant chunks that exist, how many were retrieved. Low recall means correct sources were missed and the answer is built on incomplete evidence.
- Hit rate: the fraction of questions for which at least one correct passage appears in the top-k results. A blunt but useful first signal.
- Context relevance: how on-topic the retrieved set is for the specific question, often scored by an LLM-as-judge.
There is a real tradeoff. Pushing recall up (retrieve more chunks) usually pushes precision down (more irrelevant chunks). In regulated domains, missing the correct source is often the worse failure, so teams lean toward high recall and then rely on the generation stage to ignore noise.
How do you measure generation quality?
Generation quality measures whether the answer is faithful to the retrieved sources and useful to the user. Score these on top of retrieval:
- Faithfulness (groundedness): every claim in the answer is supported by the retrieved passages. Score as supported claims divided by total claims. This is the single most important metric for regulated use.
- Answer relevance: the answer actually addresses the question asked, not a related but different one.
- Citation correctness (attribution): each citation points to a passage that genuinely supports the claim it is attached to. A confident answer with a wrong citation is a serious failure.
- Refusal-when-unknown: for questions outside the sources, the system says so plainly instead of guessing.
This is where citing behavior matters for testing, not just for trust. Samvad AI cites the exact source passage for every answer and states plainly when an answer is not in your sources, which makes faithfulness and citation correctness directly checkable: a reviewer (or an automated judge) can compare each claim to its cited passage line by line. For why exact-passage citation differs from a vague reference, see what source-cited answers actually mean.
Automated metrics vs human and LLM-as-judge evaluation
Use three evaluation methods together, because each covers the others’ blind spots.
| Method | What it scores well | Limitations |
|---|---|---|
| Automated metrics | Retrieval precision, recall, hit rate against labeled sources | Cannot judge nuanced faithfulness or tone |
| LLM-as-judge | Faithfulness, relevance, citation correctness at scale | Can be biased or inconsistent; must be spot-checked by humans |
| Human review | Ground-truth judgment, edge cases, domain nuance | Slow and expensive; not feasible on every run |
A practical pattern: automated metrics on every build, LLM-as-judge on each release for the generation metrics, and human review on a sampled subset plus all high-risk questions. Calibrate the LLM judge against human labels first, then trust it within known limits.
What do good RAG numbers look like?
Good numbers depend on your risk tolerance, not a universal benchmark. In regulated settings, teams typically target high recall (correct sources rarely missed), faithfulness near 1.0 (almost no unsupported claims), and high citation correctness, while accepting lower precision as the cost of recall. Treat any vendor’s single headline accuracy figure with suspicion: ask which metric, on whose test set, and how unanswerable questions were handled.
How do you monitor RAG accuracy in production?
Offline scores drift once real traffic and new documents arrive, so monitoring is continuous, not a one-time gate. Watch for new-document coverage (questions about freshly added content that retrieval has not indexed well), query drift (users asking new kinds of questions your golden set never covered), and citation or refusal rate changes that signal degraded grounding. Sample live answers for human review, feed real failures back into the golden test set, and re-run evaluation after every index or model change. RAG accuracy is a measurement you maintain, not a number you achieve once. To discuss evaluating a deployment in your own environment, contact Teclops AI.
Frequently asked questions
How do you evaluate a RAG system?
Evaluate a RAG system in two halves: retrieval quality (precision, recall, context relevance, and hit rate, measuring whether the right passages were fetched) and generation quality (faithfulness, answer relevance, citation correctness, and refusal-when-unknown, measuring whether the answer is grounded in those passages). Run both against a golden test set of real questions with known-correct source passages.
What are the most important RAG evaluation metrics?
The core RAG evaluation metrics are retrieval precision, retrieval recall, hit rate, and context relevance on the retrieval side, plus faithfulness (groundedness), answer relevance, and citation correctness on the generation side. Faithfulness and citation correctness matter most in regulated settings because they measure whether each claim is supported by a cited source.
How do you measure RAG faithfulness?
Measure RAG faithfulness by checking whether every claim in the answer is supported by the retrieved passages, scored as supported claims divided by total claims. Human reviewers or an LLM-as-judge break the answer into claims and verify each against the cited source; a system that cites the exact source passage per answer makes this directly checkable.
What is a golden test set for RAG evaluation?
A golden test set is a curated list of real user questions paired with the known-correct source passages and an ideal answer. It is the foundation of RAG evaluation: retrieval metrics compare fetched chunks against the known sources, and generation metrics compare the produced answer against the ideal answer and its citations.
What do good RAG accuracy numbers look like?
Good RAG numbers depend on your risk tolerance, but in regulated settings teams typically target high retrieval recall (so correct sources are rarely missed), faithfulness near 1.0 (few unsupported claims), and high citation correctness. There is a tradeoff: maximizing recall can lower precision, so tune to your domain rather than chasing a single headline score.