RAG Misattributes Evidence: Deceptive Grounding Exposes Blind Spot
New research shows retrieval-augmented generation can cite real trials yet apply them to the wrong drug, a failure dubbed deceptive grounding. Across 13 models, worst-case rates hit 8–86.7%, with 7.8% in production and 13.6% for new drugs. A simple entity-attribution check flags most cases, but vendors rarely use it.
Clinical retrieval-augmented generation (RAG) loves to look honest: faithful to sources, zero hallucinations, and citations for days. This paper shows how that veneer cracks. Models can lift true statements from real trials and pin them on the wrong drug. The authors call it deceptive grounding (DG). Your audits pass; your advice is wrong.
The setup is painfully plausible. The model sees documents about drug Y in the same disease context as drug X, complete with trial identifiers and outcomes. It then writes as if that evidence belongs to X. Faithfulness checks nod along, citation checkers applaud, and no one notices the entity swap.
How to break it
If you can influence retrieval or inject content, you do not need to fabricate. You seed alternate-drug (Y) documents that look complete: trial names, registry IDs, outcomes. You keep true X documents sparse or partial. The paper’s ablation nails the mechanism: remove that completing information and DG disappears entirely, replaced by dull confabulation. With it present, the model routes to DG and produces confident, well-cited lies about the wrong entity.
Domain specialisation makes this easier to trigger. Medical and biomedical fine-tuned models top out at 86.7% DG in the worst adversarial condition. Stronger pharmacological class priors open the door; the “facts” stroll through. An explicit entity-anchoring instruction can help some models, but not reliably.
Evidence and limits
The benchmark spans 264 drug–disease–alternate drug triples, 13 models and 15 retrieval conditions. Peak DG ranges from 8.0% to 86.7% depending on the model and retrieval mix. In a live clinical RAG system across 740 drug–disease pairs, DG shows up 7.8% overall, rising to 13.6% for recently approved drugs where retrieval is thin.
On the detection side, entity-attribution verification works: a per-claim check that the cited document’s primary entity matches the query. Reported performance is 97.0% precision and 98.7% recall against an IPW-adjusted human gold standard for this subtype. Yet no existing framework implements it.
Mechanistically, activation patching and controlled ablations support a two-stage story: disease or class overlap primes attribution; completing information dictates the failure mode (DG vs confab). The methodology leans on an automated judge (Kimi-K2.5) validated against humans, with honest caveats: adversarial stimuli, modest human samples for rare subtypes, calibration to one baseline model, and open questions on training generalisation.
On an engagement, I’d target retrieval quality and document mix, especially around new approvals and medically fine-tuned stacks. The open question for vendors: will entity checks hold up at scale on messy PDFs and heterogeneous corpora, and will you ship them before someone else ships you a cited, confident, clinically wrong answer?
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation
🔍 ShortSpan Analysis of the Paper
Problem
This paper studies deceptive grounding, a failure mode in retrieval-augmented generation where a model truthfully relays content from retrieved documents but attributes that evidence to the wrong entity. Deceptive grounding is invisible to standard automated checks used in clinical RAG systems because responses are factual, faithful to sources and carry real citations, yet they present another drug's clinical evidence as if it applied to the queried drug. In high-stakes domains such as medicine this can produce confidently cited but clinically misleading outputs.
Approach
The authors build a controlled 2D factorial benchmark over 264 drug–disease–alternate drug triples and test 13 models under 15 retrieval conditions that vary queried-drug retrieval completeness (absent, partial, complete) and alternate-drug document content (including presence or absence of completing information such as trial names, registry identifiers and outcomes). Synthetic adversarial documents are generated and judged at scale by an automated judge Kimi-K2.5, validated against human adjudication. They run mechanistic experiments including activation patching, completing-information ablation and label-substitution, and measure real-world prevalence across 740 pre-registered drug–disease pairs in a deployed clinical RAG system. They operationalise detection as entity-attribution verification, a per-claim check that the cited document's primary entity matches the queried entity.
Key Findings
- Deceptive grounding can be large under adversarial retrieval: peak DG rates across models range from 8.0% to 86.7% in the worst adversarial condition, with medical and biomedical fine-tuned models at the high end.
- Two-stage mechanism: Stage 1 opens when disease-context overlap or pharmacological class representations prime attribution; Stage 2 routes the outcome. If retrieved documents contain completing information (trial identifiers, outcomes) Stage 2 produces deceptive grounding, otherwise it produces confabulation.
- Completing-information ablation eliminates entity-attribution failure entirely while increasing confabulation, proving completing information is causally required for DG.
- Entity-attribution verification achieves high detection performance: Kimi-K2.5 reports 97.0% precision and 98.7% recall for detecting deceptive grounding on a human-adjusted gold standard for that subtype.
- Production prevalence is non-trivial: 7.8% overall DG across 740 live pairs, rising to 13.6% for recently approved drugs where entity-specific retrieval is sparse.
- Domain fine-tuning amplifies DG susceptibility because stronger pharmacological class representations make Stage 1 more likely to open; an explicit entity-anchoring instruction reduces DG substantially in some model profiles but not all.
Limitations
The benchmark is intentionally adversarial and thus overstates worst-case susceptibility relative to naturalistic deployment; controlled stimuli were calibrated to one baseline model so some absolute rates for other models are lower bounds. The automated judge has complementary blind spots versus human raters and the human gold standard sample size is modest for rare subtypes. Entity-attribution verification addresses the defined DG subtype but does not detect entity substitution where the model makes no affirmative claim about the queried entity. Mediation analysis of training effects is underpowered and generalisation to other training regimes or architectures remains to be shown.
Implications
An attacker who influences retrieval or injects adversarial documents can cause a RAG system to cite real trials and outcomes yet attribute them to the wrong drug, producing plausible, well-cited but misleading clinical guidance. Domain-specialised models are especially vulnerable, so an adversary could target medically fine-tuned systems to maximise impact. The failure is invisible to current faithfulness, hallucination and citation checks, enabling deceptive outputs to bypass routine audits. Practical exploitation could mislead clinicians, poison decision support, or erode trust in automated pipelines. Defences require extending evaluation with entity-attribution verification, prioritising entity-specific retrieval, and enforcing entity identity before synthesis; attackers who manipulate retrieval quality or document content remain a primary threat vector.
