Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG

Beyond Correctness: Auditing Source-Dependence in Multi-Source Medical RAG Systems

Current AI systems are primarily judged on whether an answer is "right." However, in sensitive fields like medicine, different institutions often have different rules. This research introduces a way to test if an AI's answer changes depending on which hospital's guidebook it reads. This helps identify hidden disagreements. Most existing benchmarks assume there is one single "gold" answer for every question. This premise breaks down when moving from textbook theory to real-world institutional protocols.

The Problem

The status quo in Retrieval-Augmented Generation (RAG) evaluation relies on the "single-gold-answer" paradigm. Whether looking at MedQA or PubMedQA, the goal is to match a curated ground truth. This works for general medical knowledge. But it cannot diagnose source-dependence. This is a failure mode where a RAG system provides a confident answer contingent on which specific document the retriever pulled.

In a multi-author environment, such as a hospital system using various departmental handbooks, these documents are not interchangeable. They reflect local protocols and risk-management choices. As shown in the construction of the TransplantQA benchmark, a single patient question can be answered by dozens of different institutional handbooks.

Those answers may vary wildly. Current benchmarks miss this axis of error. They tell you if a model is "correct" relative to a static truth. They cannot tell you if your system provides inconsistent guidance because of a retrieval hiccup.

How It Works

The authors propose shifting the unit of evaluation from answer correctness to the inter-source relationship. They operationalize this through a three-stage pipeline.

1. Structured Extraction: Raw PDF handbooks are converted into structured JSON using LlamaParse. This preserves section hierarchies and page metadata. This structure is critical for hierarchical retrieval (searching through nested document levels).
2. HERO-QA Retrieval: This is the architectural core .

Instead of simple semantic search, HERO-QA uses a multi-layer orchestration strategy. It routes queries based on handbook length. It decomposes long documents into a hierarchy of sections and child chunks. It employs four parallel retrievers (dense, sparse, section-body, and title navigation). Results are fused via Reciprocal Rank Fusion (RRF, a method to combine multiple ranked lists). It then uses a cross-encoder (a model that compares two pieces of text directly) for reranking. Finally, it performs "parent-section expansion." This ensures the generator sees a coherent context rather than isolated snippets. 3. Structured Pairwise Judgment: To measure the relationship between answers, the authors use an LLM-as-judge. The judge emits a structured JSON record. This includes a classification from a 5-label taxonomy (ranging from CONSISTENT to CONTRADICTORY). It also provides clinical reasoning and a "divergence topic" to pinpoint where sources disagree.

Numbers

The scale of this evaluation is significant. The authors conducted a production run generating 48,056 grounded answers. They performed 5,730,465 pairwise comparisons. The computational cost for this reference run was approximately $1.3K–$1.8K. This utilized NVIDIA H100 80 GB GPUs on a SLURM cluster (a workload manager for high-performance computing).

The most striking finding concerns the relationship between retrieval quality and perceived disagreement. The authors compared an earlier 14B parameter run with a 32B reference run. The higher-capacity pipeline reduced the "absence rate" ($r_{abs}$, the frequency where the model finds no info) by an average of 13.6 percentage points. However, the per-pair divergence rate remained largely unchanged. Crucially, the proportion of questions showing any divergence rose by nearly 16%. This suggests that better retrieval does not reduce disagreement. Instead, it reveals that institutional divergence is more prevalent than prior estimates suggested. Improved retrieval surfaces latent differences that were previously hidden by retrieval failures.

Regarding the taxonomy, the authors report that explicit contradictions are rare. The dominant mode of disagreement is "complementary" (differing in detail) or "divergent" (substantive differences in thresholds or timelines).

What's Missing

While the framework is robust, there are gaps. First, the empirical validation is strictly confined to U.S. solid-organ transplant education. The authors argue the methodology transfers to legal and educational RAG. However, the specific taxonomy and the nuances of "clinical significance" are domain-specific.

Second, the "LLM-as-judge" component inherits standard model biases. The authors acknowledge potential issues with self-preference (favoring models from the same family). They also note length/citation artifacts. Although they achieved high judge-vs-human agreement ($\kappa = 0.842$, a measure of statistical agreement), the judge's ability to grade "clinical significance" was weaker ($\kappa = 0.385$). This means you should treat significance ratings as a population-level signal. Do not treat them as a definitive verdict on a single pair.

Finally, the paper does not explore how this pipeline handles extremely long contexts or massive corpora. In such cases, the number of pairwise comparisons would scale quadratically ($N^2$) beyond the 5.7 million handled here.

Should You Prototype This

Yes, if you are deploying RAG over heterogeneous, multi-author corpora. If your system pulls from different legal jurisdictions or varying state educational standards, "accuracy" is a dangerous metric. You need to know if your system is flip-flopping between sources.

The HERO-QA retrieval strategy is worth prototyping for long, structured documents. Simple chunking often leads to lost context. The structured-output judge is also a significant improvement over label-only evaluators. The authors demonstrate that losing "divergence topic" and "significance" metadata in a post-hoc extraction step destroys the utility of the evaluation. The artifacts are being released. The pipeline is designed to be resumable and idempotent (meaning it can be re-run without changing the result). These features make it viable for real engineering workflows.

Figures from the paper