RAG Misattributes Evidence: Deceptive Grounding Exposes Blind Spot

Problem

This paper studies deceptive grounding, a failure mode in retrieval-augmented generation where a model truthfully relays content from retrieved documents but attributes that evidence to the wrong entity. Deceptive grounding is invisible to standard automated checks used in clinical RAG systems because responses are factual, faithful to sources and carry real citations, yet they present another drug's clinical evidence as if it applied to the queried drug. In high-stakes domains such as medicine this can produce confidently cited but clinically misleading outputs.

Approach

The authors build a controlled 2D factorial benchmark over 264 drug–disease–alternate drug triples and test 13 models under 15 retrieval conditions that vary queried-drug retrieval completeness (absent, partial, complete) and alternate-drug document content (including presence or absence of completing information such as trial names, registry identifiers and outcomes). Synthetic adversarial documents are generated and judged at scale by an automated judge Kimi-K2.5, validated against human adjudication. They run mechanistic experiments including activation patching, completing-information ablation and label-substitution, and measure real-world prevalence across 740 pre-registered drug–disease pairs in a deployed clinical RAG system. They operationalise detection as entity-attribution verification, a per-claim check that the cited document's primary entity matches the queried entity.

Key Findings

• Deceptive grounding can be large under adversarial retrieval: peak DG rates across models range from 8.0% to 86.7% in the worst adversarial condition, with medical and biomedical fine-tuned models at the high end.
• Two-stage mechanism: Stage 1 opens when disease-context overlap or pharmacological class representations prime attribution; Stage 2 routes the outcome. If retrieved documents contain completing information (trial identifiers, outcomes) Stage 2 produces deceptive grounding, otherwise it produces confabulation.
• Completing-information ablation eliminates entity-attribution failure entirely while increasing confabulation, proving completing information is causally required for DG.
• Entity-attribution verification achieves high detection performance: Kimi-K2.5 reports 97.0% precision and 98.7% recall for detecting deceptive grounding on a human-adjusted gold standard for that subtype.
• Production prevalence is non-trivial: 7.8% overall DG across 740 live pairs, rising to 13.6% for recently approved drugs where entity-specific retrieval is sparse.
• Domain fine-tuning amplifies DG susceptibility because stronger pharmacological class representations make Stage 1 more likely to open; an explicit entity-anchoring instruction reduces DG substantially in some model profiles but not all.

Limitations

The benchmark is intentionally adversarial and thus overstates worst-case susceptibility relative to naturalistic deployment; controlled stimuli were calibrated to one baseline model so some absolute rates for other models are lower bounds. The automated judge has complementary blind spots versus human raters and the human gold standard sample size is modest for rare subtypes. Entity-attribution verification addresses the defined DG subtype but does not detect entity substitution where the model makes no affirmative claim about the queried entity. Mediation analysis of training effects is underpowered and generalisation to other training regimes or architectures remains to be shown.

Implications

An attacker who influences retrieval or injects adversarial documents can cause a RAG system to cite real trials and outcomes yet attribute them to the wrong drug, producing plausible, well-cited but misleading clinical guidance. Domain-specialised models are especially vulnerable, so an adversary could target medically fine-tuned systems to maximise impact. The failure is invisible to current faithfulness, hallucination and citation checks, enabling deceptive outputs to bypass routine audits. Practical exploitation could mislead clinicians, poison decision support, or erode trust in automated pipelines. Defences require extending evaluation with entity-attribution verification, prioritising entity-specific retrieval, and enforcing entity identity before synthesis; attackers who manipulate retrieval quality or document content remain a primary threat vector.