IyawoBench v2.0: Extended Diagnostic Evaluation of Large Language Model Clinical Triage in Nigerian Primary Care

Beyond Binary Safety: Catching Silent Errors in Medical Triage

Common ways of testing medical AI can be dangerously misleading. A model might appear 100% safe because it never sends an emergency patient home. However, it may simultaneously delay critical care. It might do this by misclassifying "immediate" emergencies as merely "non-urgent." A new study introduces IyawoBench v2.0. This is a diagnostic framework designed to expose these hidden failure modes in large language models (LLMs) used for clinical triage.

The Hidden Risks of Clinical Triage

In many low- and middle-income countries, trained physicians are scarce. This leads to the deployment of LLMs as clinical triage tools. Triage is the process of determining the urgency of a patient's condition. This decision decides the next step: treating them on-site, referring them to a clinic later today, or transferring them to a hospital immediately.

The core problem is that existing safety metrics often rely on binary logic. They ask simply, "Did the model avoid sending a high-acuity patient home?" While this prevents the most obvious catastrophes, it creates a false sense of security. A model can pass this test with flying colors. Yet, it may still commit systematic errors that undermine an entire healthcare system. For example, if a model consistently downgrades "refer now" cases to "refer today," it causes fatal delays. Such delays occur in patients with sepsis or meningitis.

The Metrics of Misplaced Confidence

To understand this, we must first look at how models are traditionally evaluated. Most clinical AI benchmarks focus on accuracy. Accuracy measures how often the model's label matches the gold standard. Another common metric is "lenient sensitivity." This measures whether a model correctly identifies high-urgency cases by ensuring they aren't downgraded to the lowest level of care.

However, the authors of the IyawoBench v2.0 study argue that these aggregate scores conflate distinct types of mistakes. They build upon their previous work, IyawoBench v1.0. That work used synthetic vignettes (simulated patient stories) to test models. The new framework treats triage as an "ordinal classification problem." This means the categories have a specific, meaningful order of increasing urgency.

The researchers introduce a formal mathematical taxonomy to categorize three specific failure modes: 1. Conservative Escalation Bias (CEB): The tendency to over-refer stable patients. This could potentially overwhelm hospitals. 2. Systematic Downgrade Bias (SDB): The tendency to under-triage emergencies by one level. For instance, moving a patient from "immediate" to "later today." 3. Middle-Tier Instability (MTI): High error rates specifically among moderate-urgency patients. This is characterized by unpredictable "upshifts" and "downshifts" in urgency.

Deconstructing Model Behavior

The authors applied this framework to three frontier models: Claude Sonnet 4.6, Llama 3.3 70B, and Llama 3.1 8B. They used 200 synthetic vignettes. These were derived from 1,200 real patient encounters in Nigerian primary health centers. This ensures the test reflects actual clinical realities.

The results reveal that traditional metrics effectively hide danger. As shown in, Llama 3.1 8B scored 100% on "lenient sensitivity." This makes it appear perfectly safe.

However, when the authors applied a "strict sensitivity" metric, the performance collapsed. Strict sensitivity requires an exact match for immediate referrals. Under this metric, the model's performance fell to 23.0%. This reveals a massive 77-percentage-point gap caused by Systematic Downgrade Bias.

The models also exhibited distinct error profiles. These are visualized in the confusion matrices of .

Claude Sonnet 4.6 tended toward over-escalation. Llama 3.3 70B showed significant Middle-Tier Instability. It frequently oscillated on moderate-urgency cases .

The study also shows there is no "one size fits all" winner. By using a new "Expected Deployment Cost" (EDC) metric, the authors show the best model changes depending on the goal .

EDC translates model errors into asymmetric clinical costs. An "Emergency-Focused" system might favor a highly conservative model. Conversely, a "System-Sustainability" focused system might prioritize a model that does not overwhelm referral centers.

Rethinking AI Selection

This research shifts the conversation. We must move from asking "how accurate is this model?" to "what kind of mistakes does this model make?" We must also ask if our system can afford them. It proves that failure modes like Systematic Downgrade Bias are not necessarily tied to model size or architecture. Instead, they are properties of how the model interacts with the structure of clinical decisions.

By providing a "failure mode radar", the authors offer a way to diagnose a model's personality.

Instead of blindly trusting a high accuracy score, developers can see if a model is "too cautious" or "dangerously relaxed." This allows for more targeted mitigations. For example, one could add a human-in-the-loop to review specific types of uncertain decisions.

Limits of the Framework

While the framework is mathematically rigorous, the authors note several limitations. The evaluation is currently restricted to 200 synthetic vignettes. These focus on eight categories of febrile illness (fever-related conditions). The framework does not yet account for non-febrile presentations or pediatric-specific nuances. Additionally, the cost weights used to calculate the Expected Deployment Cost are illustrative. For real-world implementation, these weights must be carefully calibrated to local economic and clinical realities. Finally, the study only evaluates three frontier models. The behavior of other major LLMs remains unexplored.

Figures from the paper