Safety That Does Not Transfer: Cross-Lingual Clinical Correctness Drift in Deployable Medical Language Models

Safety That Does Not Transfer

Does a model that passes a rigorous safety audit in English remain safe in a rural clinic in northern Nigeria? Most current safety evaluations focus on "frontier" models (massive, high-capability systems accessed via APIs). They conduct testing almost exclusively in English. However, in many low-resource health settings, the reality is different. Practitioners rely on small, quantized models (compressed versions of models designed to run on modest, local hardware). These are queried in local languages. This paper investigates whether clinical safety established in English transfers to Hausa. It asks if there is a hidden risk that current benchmarks are structurally blind to.

Is the risk in the language or the model?

The authors ask a specific, pointed question. If a model provides correct medical advice in English but fails in Hausa, what causes the failure? Is it the linguistic complexity of Hausa? Is it the nature of the clinical tasks? Or is it the specific class of models used in real-world, low-resource deployments?

They target three high-burden medical conditions in northern Nigeria: malaria, sickle cell disease, and tuberculosis. This ensures the clinical material is grounded in unambiguous Nigerian national treatment guidelines. The core goal is to isolate the "locus of failure." They compare a suite of small, locally deployable models (4 to 9 billion parameters) against a single, high-capability frontier model. They want to see if safety degradation is a universal linguistic problem or a weakness of the smaller "deployable" tier.

The cracks in refusal-based safety

Previous multilingual safety research focused on "refusal drift" (the degree to which a model's tendency to decline harmful requests changes across languages). While guardrails can degrade outside of English, the authors argue this focus misses a dangerous failure mode in medicine.

In a clinical context, a model refusing to answer a question about a fever is a minor inconvenience. The real catastrophe occurs when the model provides an answer that is fluent, authoritative, and clinically wrong. Existing benchmarks prioritize toxicity or refusal. They are essentially blind to "correctness drift" (the change in clinical accuracy between matched prompts differing only in language). Evaluating whether a model can refuse a prompt does not reveal if it will provide life-threateningly incorrect advice when it engages.

Probing the correctness drift

To move beyond mere refusal, the researchers constructed a benchmark of matched English–Hausa question pairs. These probed four distinct modes: direct knowledge recall, emergency triage (testing if a model recognizes danger signs), leading questions (inviting contraindicated actions), and traditional-remedy probes.

The methodology is anchored in local standards. Instead of relying on subjective Western medical consensus, the authors scored all 128 responses against Nigerian national treatment guidelines. Two independent, blind Hausa-speaking raters graded responses on clinical correctness, safety hedging (the use of cautious language), and language quality.

The experimental setup reveals a stark divergence in model behavior. As shown in, the researchers tracked clinical correctness across two tiers: the locally deployable models and the frontier reference.

The results represent a fundamental shift in the nature of the outputs.

From competence to harm

The findings are striking. Among the locally deployable models, mean clinical correctness plummeted from 1.57 in English to -0.03 in Hausa. On a scale where 2 is correct and -1 is actively harmful, this represents a shift from competence to being harmful on average. This is visualized in, where correctness for all three conditions shows a consistent downward trajectory when moving from English to Hausa.

Crucially, the frontier model did not follow this trend. According to [Table 1], the frontier model moved from 2.00 in English to a still-highly-competent 1.75 in Hausa. This comparison allows the authors to make a powerful claim. The failure is not a property of the Hausa language itself. It is also not a failure of the clinical logic. The frontier model handles both with ease. Instead, the deficit is a property of the "deployable tier." These smaller models lack the capacity to maintain safety alignment across linguistic shifts.

The failure manifests in two ways, separated in .

There is "silent failure," where a model produces unusable or unintelligible text. There is also "harmful failure," where the model produces fluent, confident, but clinically incorrect content. The latter is particularly insidious. It is the failure mode least likely to be detected by a user lacking the expertise to verify the advice.

Implications for the deployment landscape

The implication chain is clear. If these results generalize, the current paradigm of "safety assurance" is incomplete. A model certified safe in English or tested at the frontier tier provides no guarantee of safety in actual deployment environments.

First, safety evaluation must be conducted at the specific tier of deployment. Testing a massive model tells you almost nothing about the safety profile of a 7B parameter model running on a local server. Second, for clinical applications, the industry must shift its metrics. We must move away from "refusal" and toward "substantive correctness." A model that refuses to answer is safe. A model that answers incorrectly with high confidence is a liability.

The paper concludes with a pragmatic procurement stance. Models below the frontier tier should not be trusted for clinical guidance in any language. This applies unless they have been specifically evaluated in that tier and in that language. If this study is representative, the "safety" celebrated in large models is a luxury. It does not necessarily trickle down to the tools reaching the front lines of global health.

Figures from the paper