AtomiMed: Hierarchical Atomic Fact-Checking for Universal Clinical-Aware Medical Report Evaluation

Breaking Down the Diagnostic Narrative

Current AI tools for reading medical reports often miss critical errors. AtomiMed breaks reports down into tiny, verifiable facts about diseases and their details—such as size or location. It checks if an AI's report matches the truth, much like a second doctor reviewing a case. This approach addresses a fundamental gap in how we judge the reliability of automated medical reporting.

The Architecture of a Medical Truth

The goal of Medical Report Generation (MRG) is to alleviate the heavy workload of radiologists. AI aims to transform medical images into structured textual narratives. However, evaluating these AI systems is notoriously difficult. If an AI generates a report that is grammatically perfect, it can still fail. It might accidentally state that a tumor is on the left side instead of the right. Such a mistake is a catastrophic clinical error.

Standard evaluation metrics typically rely on n-gram overlap. This calculates how many sequences of words (n-grams) the AI's text shares with a human's text. While useful for general language, these lexical metrics are "semantically blind." They cannot distinguish between "no pleural effusion" and "pleural effusion." These two phrases are nearly identical in word overlap. Consequently, a system might receive a high score while providing dangerously incorrect diagnoses.

The Limits of Current Evaluators

Researchers have developed several specialized approaches, yet each carries significant trade-offs. Structure-based metrics like RadGraph attempt to extract clinical entities and their relationships. However, they are often limited to specific domains like chest X-rays. This makes them less useful for CT or MRI scans. Other "LLM-as-a-Judge" approaches use powerful models to grade reports holistically. These often function as "black boxes." They provide a score without an audit trail to explain why a report was marked down.

The authors of the AtomiMed paper argue these failures stem from treating a report as a single, monolithic block of text. In reality, a radiology report is a composition of discrete, hierarchical units. To solve this, they propose a framework that mimics the radiological peer-review process. In this process, a second reader independently verifies the findings of the first.

Decomposing the Clinical Narrative

The AtomiMed framework operates through a two-stage process. It turns unstructured prose into a verifiable hierarchy of Atomic Clinical Facts (ACFs). As shown in, the first stage is Hierarchical Atomic Decomposition.

The researchers use Qwen3-235B-A22B as the backbone engine for this task. An instruction-tuned Large Language Model (LLM) parses a report to extract two distinct levels of information:

1. Disease-level QA: Binary questions regarding the presence or absence of a clinical entity (e.g., "Is there a pneumothorax?").
2. Attribute-level QA: Descriptive facets tied to those entities. These include location, size, morphology (shape), severity, quantity, and temporal change.

Once the reports are decomposed, the second stage begins. This is the Agentic Cross-Verification loop. Instead of comparing the two texts directly, the system treats one report as "evidence" to answer questions posed by the other. This bidirectional loop calculates precision, recall, and F1 scores (standard measures of accuracy and completeness) at both levels. This allows the framework to decouple diagnostic detection from descriptive accuracy.

To test this, the researchers curated OmniMRG-Bench. This is a massive multi-modal benchmark containing over 178,000 verified QA pairs across X-ray, CT, MRI, and Ultrasound .

Seeing Through the Errors

By shifting to atomic facts, AtomiMed reveals systematic weaknesses in current medical AI. In their empirical analysis, the authors found that many models are proficient at describing morphology. However, they struggle significantly with "severity" and "size" [Figure 4a]. For example, the model HuatuoGPT-34B scored near the floor on severity. This suggests it can identify a problem but cannot reliably communicate how dangerous it is.

Furthermore, the framework exposes anatomical biases. Many models show high performance in the respiratory system. This is likely due to heavy training on chest X-rays. However, their performance collapses when tasked with the digestive, reproductive, or urinary systems [Figure 4b].

The statistical strength of this approach is evident in its correlation with human experts. In a pairwise preference study, the metric must decide which of two models a radiologist would prefer. AtomiMed achieved 95.71% accuracy on X-ray data. It also reached a Kendall’s tau of 0.9807 (a measure of how well the ranked order of models matches human ranking) [Table 3]. Most importantly, AtomiMed maintained high correlation across all imaging modalities. This is a major improvement over previous LLM-based judges like GREEN. GREEN saw its performance degrade sharply when moving from X-rays to complex MRI scans .

The Boundaries of the Framework

Despite its strengths, AtomiMed is not a complete replacement for human oversight. The framework relies on an LLM as its engine for decomposition and verification. This introduces significant inference costs (the computational resources and time required to run the model). The authors suggest that future work will focus on creating smaller, "distilled" models to make this process more efficient.

Additionally, the framework's effectiveness depends on the underlying LLM's ability to parse complex medical terminology. Finally, the current iteration focuses on static reports. Extending this to "longitudinal" comparisons—tracking how a disease changes over multiple scans—remains an open challenge.

Figures from the paper