ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents

Even the strongest LLMs struggle with management decisions (medications/procedures) compared to diagnoses. High decision accuracy often masks inefficient or redundant information-seeking behavior.

In medical AI, evaluation is shifting from factual recall to complex agency. Researchers want to move beyond asking an LLM to pick an answer from a list. They want to test if an AI can function as a clinician. Previous benchmarks focused on medical exams or database retrieval. These failed to capture the reality of a hospital stay. In a real stay, information is gathered incrementally. Decisions are sequential and irreversible. This paper introduces CLINENV to bridge that gap. It simulates real inpatient admissions where an AI must interact with a digital hospital environment to succeed.

The Problem

Current medical benchmarks suffer from a structural mismatch with real clinical practice. Most existing evaluations fall into three traps. They are static (testing factual recall via multiple-choice questions). They are data-access oriented (treating the Electronic Health Record, or EHR, as a queryable SQL database). Or they rely on simulated patients that lack deep medical histories. None of these approaches probe the "long horizon." This refers to the ability of a model to maintain reasoning across a multi-day hospitalization.

As shown in, clinical practice requires navigating a timeline of evolving information.

Existing benchmarks might tell you if a model knows a symptom. They cannot tell you if a model knows which lab test to order to confirm a suspicion. They cannot tell if a model will redundantly order the same expensive test five times. This creates a blind spot. Outcome-only evaluations measure only if the final diagnosis was correct. This hides the messy, costly, and potentially dangerous ways a model arrives at a conclusion.

How It Works

The authors propose a paradigm called Longitudinal Inpatient Simulation (LIS). CLINENV builds an interactive, multi-stage environment using real patient data from the MIMIC-IV dataset. The architecture consists of three core pillars:

1. Automated Case Construction: A pipeline processes raw EHR admissions into ordered clinical stages. It uses a planning backbone (Claude-Sonnet-4.6) to extract physician decisions from discharge notes. It then "anchors" those decisions to specific timestamps in the patient's medical timeline .
2. Multi-Agent Interaction: At each stage, the model is not given the full medical record. Instead, it must act as an attending physician by querying four specialized agents. These include a Patient agent (providing subjective history in lay language), a Nurse agent (reporting vitals), a Lab agent (resolving test queries), and a History agent (supplying context from prior admissions).
3. Deterministic Dual Evaluation: The system does not just grade the final answer. It uses "ontology-grounded matching" (scoring against standardized medical taxonomies). It uses ATC for drugs and ICD for diagnoses. Simultaneously, it tracks "process quality" metrics. These measure how much relevant information the model covered and how much money it wasted on unnecessary laboratory tests.

Numbers

The results reveal a massive competency gap between recognizing a disease and managing it. The authors report that even the strongest model, GPT-5.4, achieves a decision F1 (a harmonic mean of precision and recall) of only 0.306. While the model is relatively good at recovering discharge diagnoses (0.511 F1), its performance on management actions collapses. Specifically, medication management drops to 0.17 F1 [Table 6]. This means models can identify a sickness but struggle to prescribe the correct treatment.

The paper highlights a critical decoupling of outcome and process. For instance, Llama-3.1-70B achieved a competitive medication score. However, it showed poor process quality. It had the lowest information coverage (0.140) and the highest laboratory waste (35.8%) [Table 5]. Conversely, smaller models like GPT-5.4-nano were more efficient. They reached comparable coverage with fewer queries and much lower waste (4.1%).

Crucially, difficulty scales with time. The authors demonstrate in that decision F1 declines as the stage index increases.

By the later stages of an admission, models issue redundant queries. These queries surface no new information. This causes "coverage efficiency" to plummet [Figure 2c]. This suggests the bottleneck is not information access. Instead, it is the cognitive load of reasoning over a growing longitudinal record.

What's Missing

While CLINENV is a significant step forward, there are several gaps:

• Concordance vs. Optimality: The benchmark scores models based on how closely they match the actual decisions made by humans in the MIMIC dataset. This measures concordance (agreement with recorded practice) rather than "optimal" care. A model might suggest a superior treatment. It would still be penalized as a "miss" if it differs from the recorded history.
• Single-Center Bias: The entire dataset comes from a single US academic medical center. This limits the benchmark's ability to account for different regional protocols. It also limits its ability to handle diverse coding standards or patient populations.
• Simulation Fidelity: The information agents are powered by LLMs (specifically GPT-5.4-mini). The "ground truth" of the environment is itself a simulation. Systemic biases in the agent models could influence the benchmark results.

Should You Prototype This

Yes, but specifically for evaluating agentic workflows. If you are building a system to interact with an EHR, you cannot rely on static datasets. You must measure the "information-acquisition gap" that CLINENV makes visible.

The benchmark is publicly available on PhysioNet (subject to credentials). The source code is reportedly available on GitHub. If your goal is to minimize "lab waste," use the metrics defined here. Specifically, look at the inverse relationship between coverage and waste shown in .

This provides a concrete engineering target. Do not just optimize for the right diagnosis. Optimize for the shortest, cheapest path to getting there.

Figures from the paper