Your Retrieval Eval Passed. Your Users Still Got Lied To.
The definition in your index was eight months out of date. Every metric looked healthy. The answer was wrong in the way that matters.
What This Post Covers
What eval metrics actually measure — and the gap between that and what users experience.
The four ways a 0.95 faithfulness score can accompany a wrong answer — specifically, concretely, with the mechanism.
The 47–67% finding: the most uncomfortable number in RAG evaluation research, and what it means for pipelines that score well.
Why evaluation is necessary and not sufficient — the precise definition of that gap.
What it means to build for user truth, not metric performance — the shift in discipline that closes the gap.
Here is the scenario that should end the “our eval looks great” conversation.
Your RAG system retrieves a glossary definition of “monthly active users” and generates a response that faithfully reflects that definition.
Faithfulness scores 0.95. Answer relevance scores 1.0. Context precision and context recall both score 1.0. Every metric on the dashboard is clean.
But the definition in the index was last updated eight months ago. Before the product team redefined the metric to exclude trial users.
The analyst who asked the question used that answer in a board presentation.
The board made a resource allocation decision based on a user count that was inflated by thirty percent because the system faithfully retrieved and accurately summarized a stale definition of a changed business metric.
Every eval passed. The user got lied to. Your dashboard registered nothing.
This is not a story about a broken evaluation framework.
RAGAS is a well-designed tool. Faithfulness is a real metric measuring a real property.
The story is about what happens when you treat eval scores as a proxy for the thing users actually need — correct answers — rather than as narrow measurements of specific pipeline properties.
Eval scores and user truth are not the same thing. In the specific ways they diverge, your users get wrong answers with excellent scores.
What Eval Metrics Actually Measure
Start with precision, because precision is what the evaluation conversation is missing.
Faithfulness measures whether the generated answer only makes claims that are supported by the retrieved context. It does not measure whether the retrieved context is true, current, or complete. A system that faithfully summarizes a stale document, a wrong document, or a document whose meaning has changed since indexing will score 1.0 on faithfulness while delivering a wrong answer. Faithfulness is a property of the relationship between generation and retrieval. It is not a property of the relationship between retrieval and reality.
Context precision measures whether the retrieved chunks are relevant to the query. It does not measure whether they are correct, authoritative, or current. A chunk that is topically on-point but factually outdated scores high on precision. A chunk that is semantically adjacent to the query but describes a policy that was superseded six months ago scores high on precision. Precision is relevance. Relevance is not accuracy.
Context recall measures whether the retriever found all the chunks needed to fully answer the query. It does not measure whether those chunks, collectively, represent the true answer. A retriever can achieve perfect recall of an outdated corpus. It can retrieve every relevant document from an index that hasn’t been refreshed in three months. Recall is completeness within the index. Completeness within the index is not completeness within the world.
Answer relevance measures whether the answer addresses the question that was asked. It does not measure whether the answer is correct. A response that addresses the question fluently and authoritatively from stale, incomplete, or wrong evidence scores high on answer relevance. Relevance is topical alignment. Topical alignment is not truth.
These are not flawed metrics.
They are precisely defined measurements of specific pipeline properties — the relationship between the generator and the retrieved context, the quality of the retrieved context relative to the query, the completeness of retrieval within the corpus. They are accurate and useful for what they measure.
What they do not measure is whether the user received a correct answer. That gap — between what the metrics measure and what users need — is the gap your dashboard shows as passing while your users get lied to.
The Four Ways a 0.95 Faithfulness Score Can Accompany a Wrong Answer
Each of the following is a distinct mechanism. Each produces a wrong answer with healthy eval scores. Each requires a different fix.
1. Faithful summarization of stale content.
The index contains the truth as of eight months ago.
The world changed. The product metric was redefined.
The policy was updated.
The regulation was amended. The pricing was revised.
The retriever faithfully returns the best available document for the query — the document is semantically correct, well-matched, high precision — and the generator faithfully summarizes it.
Faithfulness 0.95.
The answer reflects a reality that no longer exists.
This is not a retrieval failure.
The retriever did its job correctly given the corpus. It is not a generation failure.
The generator summarized accurately. It is a corpus freshness failure — and none of the four standard metrics measure corpus freshness, because all four of them assume the retrieved content is trustworthy.
That assumption, stated precisely, is: the index reflects current truth.
When it doesn’t, every metric that takes the index as given inherits the staleness.
2. Faithful summarization of incomplete evidence.
The retriever returns the relevant chunk but not the qualifying chunk — the clause that limits the rule, the amendment that changes the conclusion, the footnote that inverts the interpretation.
The generator summarizes what it received faithfully.
Faithfulness: 0.91. Context recall: 0.62.
The answer is a correct summary of incomplete evidence — which is the same as a misleading summary of the full picture.
The legal RAG that missed the second statute on every multi-hop question for three weeks scored 0.91 on faithfulness throughout.
The user received an answer that accurately represented what was retrieved. What was retrieved was not what was needed.
The gap between context recall and faithfulness is where that failure lived, invisible on the dashboard because only one was being watched.
3. Faithful summarization of the wrong document.
The retriever returns a document that is semantically similar to the query but not the authoritative source for the answer.
A product FAQ chunk that uses the same vocabulary as the specification sheet but was written by marketing rather than engineering.
A legal summary that paraphrases the regulation rather than citing it directly.
An HR policy from the intranet that shadows but doesn’t match the official employee handbook.
The generator summarizes faithfully. The cited source does not actually support the claim in the way the user needs it to be supported.
This is what the research calls groundedness failure — distinct from faithfulness failure.
Faithfulness measures whether the answer matches the retrieved context. Groundedness measures whether the retrieved context is the authoritative source for the claim.
A faithful answer from a non-authoritative source is an answer the user cannot rely on, even though it scored well. Citation accuracy — does the cited source actually contain and support the specific claim — is the metric that catches this. It is absent from most evaluation pipelines.
4. The generator ignored what the retriever found.
The most uncomfortable finding in 2026 RAG evaluation research: in empirical analysis across multiple benchmark datasets, 47.4–66.7% of queries had the generator ignoring the retriever’s top-ranked documents.
Not failing to retrieve the right documents. Retrieving them, and then not using them.
The generator has parametric knowledge from pretraining.
When the retrieved evidence conflicts with that parametric knowledge — or when the generator’s training distribution biases it toward a particular answer — it can produce a fluent, confident response that contradicts or bypasses the retrieved context.
Standard faithfulness metrics catch this when it’s extreme — a response that adds claims not in the retrieved context scores low. But when the generator produces a response that partially uses the retrieved context and partially draws on parametric knowledge, the faithfulness score degrades subtly rather than obviously. It may still score above the threshold. It may not reflect what the retriever actually found.
Standard RAG metrics would score these outputs as passing.
A domain expert would not.
The user receives an answer shaped by the model’s training distribution, not the organization’s current knowledge base — and the metrics, measuring the relationship between answer and retrieved context, register a partial match rather than a failure.
The Precise Definition of “Necessary But Not Sufficient”
The phrase gets used often and understood rarely. Here is its precise meaning in the context of RAG evaluation.
A retrieval eval that passes is necessary for user truth — a system that scores 0.4 on faithfulness is reliably producing wrong answers. A passing score is a required property of a system worth shipping.
A retrieval eval that passes is not sufficient for user truth — a system that scores 0.95 on faithfulness may still be producing wrong answers through any of the four mechanisms above. A passing score does not guarantee correct answers.
The sufficiency gap is the gap your eval is not covering. It contains: corpus freshness, source authority, citation accuracy, generator utilization of retrieved context, coverage of the real query distribution, and the completeness of retrieved evidence relative to the full answer.
None of those properties appear in the standard four metrics. All of them determine whether the user received a correct answer.
Knowing the precise location of the sufficiency gap is what lets you close it — not by replacing the eval metrics you have, but by adding the coverage they don’t provide.
The Missing Metrics
Each gap has a metric. The metrics exist. They are not being run.
Context trustworthiness. Every chunk carries an indexed_at timestamp. The eval checks whether retrieved chunks are within the declared freshness SLA for their document class. A 0.95 faithfulness score on content that is 240 days past its declared freshness window is a passing faithfulness score on stale content. Trustworthiness is the metric that surfaces the staleness the other four assume away.
Citation accuracy. Beyond faithfulness — does the cited source actually contain the specific claim the answer makes? Not does the answer match the retrieved context, but does the context actually warrant the claim at the precision the answer implies? Citation accuracy requires checking the cited chunk against the generated claim at the statement level, which is a harder and more expensive eval than faithfulness. It catches the groundedness failure that faithfulness misses.
Context utilization. Of the chunks assembled into the context, how many contributed to the generated answer? A system that retrieves twelve chunks and generates from two of them has a context utilization problem — and a potential for the generator to be drawing the other ten percent from parametric knowledge rather than retrieved evidence. Utilization is the metric that catches the 47–67% ignoring-the-retriever finding before it becomes a user complaint.
User correction rate. The metric that bypasses the entire technical stack and asks the user directly. When users correct the system, edit its answers, rephrase its outputs, or explicitly mark answers as wrong — those events are signals about the gap between eval truth and user truth. They are the most direct measurement available of whether the system is serving the user. Most RAG systems do not collect this signal. Those that do find it the most predictive single metric of actual user satisfaction.
What Building for User Truth Looks Like
The shift is from “does the eval pass” to “is the user receiving a correct, current, trustworthy answer.”
Those are different optimization targets, and optimizing for the first without measuring the second produces exactly the scenario at the top of this post.
Building for user truth requires three things the standard evaluation discipline doesn’t enforce.
Source authority as a retrieval property, not an assumption. The retriever should distinguish between authoritative sources — the official specification, the current policy document, the canonical definition — and derivative sources — the FAQ, the blog post, the cached summary. High-precision retrieval of a non-authoritative source is not high-quality retrieval for decisions that depend on authority. Tagging sources by authority level at ingestion and filtering retrieval by that tag for queries where authority matters is a design choice. Most RAG systems do not make it.
Freshness as a first-class eval dimension. Every document class has a truth horizon — the length of time over which the document can be assumed to reflect current reality. Operational metrics: days. Policy documents: weeks. Static reference: months. The eval should check retrieved content against these horizons and penalize freshness violations as quality failures, not infrastructure failures. A faithfulness score of 0.95 on a 240-day-old document in a class with a 30-day truth horizon is not a passing score on user truth.
Continuous production eval as the primary signal. The offline golden set is a necessary baseline. It is not the primary quality signal for a deployed system. Sampled production traces, user correction events, and explicit helpfulness feedback are the signals that measure whether the system is serving users — not whether it performs well on questions you already know the answers to. The team that treats production eval as the real signal and offline eval as the calibration tool is building for user truth. The team that treats offline eval as the product gate and never looks at production signals is building for metric performance.
The Bottom Line
Your eval suite is measuring real things.
Faithfulness is real. Context recall is real. Answer relevance is real.
The problem is not that your metrics are wrong.
The problem is that your metrics are measuring a system that exists on paper — a pipeline in which the retrieved content is trustworthy, the retriever found everything, the generator used what it found, and the query distribution resembles your golden set.
The system your users interact with is different.
The corpus ages.
The generator has its own priors. The real queries are harder and stranger. The authoritative source and the topically adjacent source look the same to a similarity function.
Your eval passed. Your users got lied to.
The distance between those two facts is not a measurement error. It is a measurement gap — a set of properties that determine user truth and are absent from your eval suite.
Name the gap. Measure it. The metrics exist.
An eval that passes is the minimum bar, not the guarantee. Build for what it doesn’t cover.