You're Evaluating RAG on Questions You Already Know the Answer To
A 0.91 faithfulness score on your golden set. A 34% user dissatisfaction rate in production. Both numbers are true at the same time. Only one of them matters.
What This Post Covers
The golden dataset selection bias — why the very act of building a golden set makes it easier than production.
The Coverage Illusion — the gap between synthetic query distributions and real user queries, and why it’s undetectable from the query alone.
Three failure modes that score well and fail silently — faithfulness, coverage, and the unanswerable query.
Why LLM-as-judge compounds the problem — the evaluator has the same biases as the system it’s grading.
What a representative eval set actually looks like — and the discipline that keeps it honest over time.
The metric the industry is missing: coverage, not accuracy.
Your RAG evaluation looked great.
The faithfulness score was 0.91. Context precision was solid.
The team signed off, the system shipped, and three weeks later users were complaining that one in six answers was wrong.
The team checked faithfulness in the dashboard: still 0.91.
Everything looked fine. Until someone thought to check context recall, which came in at 0.62.
The retriever was missing the second statute on multi-hop questions, the generator was answering coherently from the partial context, faithfulness stayed high, and not a single metric on the dashboard had surfaced the regression.
The evaluation was not lying. It was answering a different question than the one the users were asking.
This is the central failure mode of RAG evaluation in 2026, and it is almost never described plainly: your eval suite is measuring a system that is easier than the one in production.
The golden dataset was built from questions you already knew the answer to.
The production query distribution is not those questions. And a system that scores 92% on a curated eval set can run at 0.78 on real traffic — not because the eval was broken, but because the eval was optimized for questions that aren’t representative of the problem it claims to measure.
You have been grading your system on its best day against its easiest questions. That is not an evaluation. That is a rehearsed performance.
The Selection Bias Baked Into Golden Datasets
A golden dataset is built by humans.
Humans doing that work follow a specific, rational process: they select questions, identify the source chunks that contain the answer, write the ground-truth answer from those chunks, and label edge cases.
The process is sound. It contains a structural bias nobody talks about.
The questions that make it into a golden dataset are questions where the answer is findable. You pick a question, you look for the answer in the corpus, you find it, you write the ground truth.
If you can’t find the answer — if the corpus doesn’t contain it, if the answer requires synthesis across six documents in a non-obvious way, if the query is ambiguous enough that no single answer is correct — that question tends to get dropped. It’s too hard to label.
The ground truth is unclear. It makes the eval harder to build without making it more informative. So you drop it and move on.
The result: your golden set is systematically biased toward questions your corpus can answer well, asked in a form that maps cleanly to your chunks. Every question in it was selected because it has a right answer you could identify. Every question your users actually ask that doesn’t meet that bar is absent.
This is not an oversight. It’s the rational behavior of people doing hard, time-consuming labeling work. But the consequence is that you have built an evaluation for a retrieval-friendly distribution that may not resemble your production query distribution at all.
Your golden set is not a sample of your query distribution. It is a sample of your query distribution filtered to the questions your system was already built to handle.
The Coverage Illusion
A 2026 paper studying a production RAG system named this problem directly, and the name earns its keep: the Coverage Illusion.
The finding is specific.
Synthetic queries (the kind generated by tools like RAGAS that turn your corpus into a golden dataset automatically) tend to be contextually verbose and grammatically complete. They sound like questions a careful, informed person would write after reading the document they’re asking about.
Real user queries are characterized by brevity and extreme vocabulary sparsity.
They look like search box inputs, not essay prompts. “what’s the refund window for enterprise plans” not “According to our enterprise billing documentation, what is the policy regarding the timeframe for refund requests on annual enterprise subscriptions.”
That gap is not cosmetic.
The retrieval system that scores well on verbose, corpus-derived queries may fail significantly on sparse, abbreviated real queries — because the embedding that represents the synthetic query has a strong semantic signal, and the embedding that represents the real query may not overlap well with any chunk in the index.
The system looks coverage-complete on the eval. It fails on a large fraction of real traffic.
Here is the precise consequence from the production study: the coverage gap is undetectable from the query alone — you only discover it after searching the index and finding nothing.
A pre-retrieval classifier cannot predict which queries will fail coverage. You find out at query time, when the retriever comes up empty, and the model hallucinates over the hole.
The same study found that 72% of real user queries could be served without any LLM augmentation — implying that the synthetic eval set had been systematically overestimating how complex the real retrieval task was, and therefore optimizing a system for a harder problem than users were actually bringing.
Three Failure Modes That Score Well and Fail Silently
The Coverage Illusion is one family of the underlying problem.
There are two others that show up in production evaluations, each with its own signature.
Faithfulness without recall.
The legal RAG story from the top.
Faithfulness measures whether the generated answer is grounded in the retrieved context — did the model say things the chunks support?
It is a generation metric. It has no opinion about whether the retrieved context was complete.
A system that retrieves half the relevant information and generates a perfectly faithful answer from that half scores 0.91 on faithfulness and fails the user, because the answer was incomplete in a way the metric cannot see. Context recall — did we retrieve the chunks that actually contained the answer — is the missing metric, and most teams measure faithfulness because it’s easier to compute and the number is better.
The fix is not to replace faithfulness with recall. It is to measure both and to understand that high faithfulness with low recall means your generator is writing coherent fiction, from a deliberately small evidence base, that your eval calls success.
The unanswerable query problem.
Your corpus does not contain every answer your users will seek. Users ask about things that aren’t documented, events that postdate your index, edge cases nobody wrote down, and products you don’t sell.
The correct behavior for these queries is to say “I don’t have information about that” — or to give the partial answer clearly marked as partial.
A golden dataset almost never includes these queries.
The labeling process selects for questions with findable answers. So the eval grades the system on its coverage of the corpus, while the system ships into a world where a meaningful fraction of queries are outside that corpus.
If the system is not evaluated on unanswerable queries, it has no pressure to develop good abstention behavior. And an LLM with no abstention pressure answers everything confidently, including the things it should decline.
One 2026 audit explicitly flagged this: label edge cases — questions where the knowledge base does not contain the answer, and the correct behavior is to say so.
Most teams don’t.
So they ship systems that confidently hallucinate over the questions your corpus can’t answer, and those questions never appeared in the evaluation that said the system was production-ready.
The distribution drift problem.
Your golden set was built from your corpus and your query distribution in the month you built it.
Six months later, the corpus has changed — documents updated, policies revised, products deprecated. And the user query distribution has drifted too — new features surface new questions, user behavior adapts, seasonal patterns shift what people ask.
The golden answers labeled six months ago may now be wrong.
A frozen golden set that isn’t regularly refreshed with production queries becomes a measurement of historical performance on a historical query distribution — which is no longer the system in production.
The faithfulness score on the offline eval stays 0.91.
The faithfulness on live traffic, measured against real answers to real queries, may be 0.78. Both are true. The dashboard shows the number that looks better.
Why LLM-as-Judge Compounds the Problem
Most of the metrics above — faithfulness, answer relevancy, context precision — are computed by an LLM judge, because human evaluation doesn’t scale. The judge reads the retrieved chunks, the generated answer, and the ground truth, and scores the system. This is practical and widely used.
It also introduces a second layer of evaluation bias that amplifies the first.
The judge has its own preferences: it prefers longer answers, prefers verbose hedging, prefers the model family it shares with the system under test.
A judge from the same model family as the system being evaluated tends to agree with that system’s outputs — not because they’re right, but because they share a prior on what good answers look like. This can inflate faithfulness scores by several points compared to a calibrated human rater.
The fix is specific: calibrate the judge against human raters before you trust its scores. Run a set of queries through both the judge and domain experts.
Measure agreement with Cohen’s kappa — a floor of 0.6 is the industry standard for acceptable inter-rater reliability. Below that, the judge is noise dressed up as measurement. And use a different model family for the judge than for the system. Same-family evaluation is not independent evaluation.
More fundamentally: an LLM judge inherits all of the golden dataset’s biases. It grades the system on the distribution the golden set represents — not the distribution your users bring.
A judge that awards 0.91 on your curated set is telling you something real about your curated set. It is telling you nothing about Tuesday’s queries.
What an Honest Evaluation Actually Looks Like
There is no perfect eval.
The goal is not perfection — it’s directional honesty. A few disciplines that close the gap between your eval score and your production reality.
Sample from production traffic, not just from your corpus. The most representative queries in your golden set are the ones your users actually asked. Log production queries, sample them, label the ones you can, and add them to the eval set on a regular cadence. The labeled production queries will be harder than the corpus-derived ones. That’s the point. Your eval should be at least as hard as your median production query.
Include unanswerable queries explicitly. Label a meaningful fraction of your eval set — 15 to 20 percent is a reasonable target — as queries where the correct answer is “I don’t know” or “the corpus doesn’t contain this.” The system’s score on that subset is its abstention rate, and abstention rate is a real production metric. A system that hallucinates on 80% of unanswerable queries is a system with a real problem, and you will not find that out from a golden set that contains none.
Measure coverage before you measure accuracy. Coverage answers the question: what fraction of the queries in your production distribution can this system even attempt? Not answer correctly — attempt. A system with 95% accuracy on the 60% of queries it can cover is not a system with 95% accuracy. It is a system with roughly 57% overall accuracy, and the other 43% of queries are being handled by something else — either abstention or hallucination. Know which.
Refresh the golden set on a regular schedule. Tie it to your index update cadence. When documents change, the ground truth may change. When new features ship, the query distribution shifts. Treat the golden set as a living artifact, versioned alongside the index, with a documented staleness policy.
Calibrate your judge. Before trusting LLM-as-judge scores, run a calibration pass: 50 to 100 query-answer pairs, graded by both the judge and a domain expert, kappa computed. Recalibrate every time you change the judge model, the prompt, or the evaluation criteria. The calibration is not a one-time task. It is a standing discipline.
The Metric the Industry Is Missing
The field has converged on a set of metrics — faithfulness, answer relevancy, context precision, context recall — that are well-designed for what they measure: the quality of the system on queries it can handle, graded against answers a human could verify.
The missing metric is coverage — the fraction of your real query distribution that the system can handle at all, with acceptable quality. Coverage is the outer envelope. Accuracy is what happens inside it. Evaluating only accuracy is evaluating only the best case, and the gap between your best case and your average case is the gap between your eval score and your user satisfaction rate.
Coverage is hard to measure because it requires knowing the real query distribution, which requires production traffic, which requires the system to be in production.
This is the uncomfortable circularity at the heart of RAG evaluation: you can only measure the real distribution by shipping, and you want to know the real distribution before you ship.
The closest you can get before shipping is adversarial query generation: deliberately create queries that are short, ambiguous, outside the corpus, multi-hop, temporally sensitive, or dependent on information the index doesn’t contain — and grade the system on all of them.
Not to find a number to put on a slide.
To find the failure mode that will show up on day three in production before day three arrives.
The Bottom Line
Your golden dataset is not a neutral sample of the problem. It is a sample of the problem filtered to the cases your system was built to handle, graded by a judge that shares the system’s prior on what a good answer looks like, against ground truth that was correct when you wrote it.
That is not an indictment of evaluation. It is a description of a discipline that requires active countermeasures to stay honest: production query sampling, explicit unanswerable coverage, regular refresh, calibrated judges, and coverage as a first-class metric alongside accuracy.
The 0.91 faithfulness score is real. So is the 34% dissatisfaction rate.
The gap between them is the gap between the questions you evaluated and the questions your users asked.
Close that gap before shipping, or your users will close it for you.