BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali

The False Security of Single-Track Benchmarks

Researchers have created a new way to test if AI models lie or make mistakes when speaking Bengali. While LLMs are increasingly multilingual, the industry has largely ignored specific hallucination behaviors in low-resource languages (languages with limited training data). Existing benchmarks often focus on a single dimension of truthfulness. This fails to capture the nuance required for safe deployment in non-English contexts. This paper argues that measuring only how often a model misses a lie is insufficient. Measuring how often it flags a truth as a lie is also necessary. By introducing a dual-track metric, the authors reveal that models often fall into "label collapse" (defaulting to a single response type). Models may simply say "yes" or "no" to everything to minimize perceived error.

The Problem

Current hallucination evaluation typically relies on a single track. This involves presenting a model with a known hallucination and asking it to detect it. This is fundamentally flawed because it rewards "lazy" models. If a model learns to simply output "Yes" for every query, it will achieve a perfect score on a hallucination-only benchmark. However, such a model is completely useless in production. Conversely, a model that always says "No" might look competent on standard QA tasks. Yet, it will fail catastrophically when tasked with verifying facts.

As shown in, this creates distinct, problematic bias profiles.

For instance, the authors note that LLaMA-3.1-8B exhibits a massive Track B error (81.63%) in GQA tasks. Meanwhile, it maintains a low Track A error (13.80%). This indicates the model acts as a "hallucination detector" that is far too aggressive. It flags valid answers as lies. Standard benchmarks that do not isolate these two failure modes cannot distinguish between true understanding and biased response patterns.

How It Works

The authors propose BenHalluEval, a framework designed to force models to demonstrate calibration across two distinct dimensions. The architecture, outlined in, operates through a three-stage pipeline:

1. Seed Selection and Filtering: Researchers identify "hard" cases rather than using random samples. For Generative Question Answering (GQA), they prompt multiple models to find instances where all three fail. This ensures the benchmark tests the limits of model capability.
2. Hallucination Generation: Using GPT-5.4, the authors construct 12,000 hallucinated candidates. They categorize hallucinations into twelve task-specific types. These include "Arithmetic Slips" (subtle numerical errors in reasoning) and "Semantic Drift" (changing the goal of a math problem mid-way).
3. Dual-Track Evaluation: Every sample is evaluated on two tracks. Track A uses ground-truth correct answers where the expected verdict is "No." Track B uses the generated hallucinated candidates where the expected verdict is "Yes."

To unify these, they introduce the BenHalluScore. The metric is calculated as $\frac{1}{2} \times (\text{Track A error} + \text{Track B error}) \times 100\%$. A score of 0% represents perfect calibration. A score of 50% indicates the model is behaving like a coin flip or a biased constant.

Numbers

The results highlight a massive disparity in how models handle Bengali. The authors report that BenHalluScore varies from 7.72% to 55.42% across the tested models.

The standout performer is GPT-4.1 mini. It achieved the best BenHalluScore in GQA (15.56%) and Code-Mixed QA (21.90%). However, the most significant finding for engineers is the performance of TigerLLM-9B. Despite being a small 9B parameter model, it outperformed much larger multilingual models in GQA and summarization. The paper suggests that targeted Bengali pretraining provides a superior calibration advantage over sheer parameter scale.

On the negative end, the authors observe that Chain-of-Thought (CoT) prompting is unreliable here. CoT is a technique where the model explains its reasoning step-by-step. As visualized in, applying CoT to Mistral-nemo-12B and LLaMA-3.1-8B often failed to improve the BenHalluScore.

In some cases, CoT actually worsened the score. The model simply shifted from one bias to another. It swapped a "No" bias for a "Yes" bias without improving actual discrimination.

What's Missing

There are two primary gaps that a practitioner should consider. First, the hallucinated candidates are generated by GPT-5.4. While the authors performed manual validation, the benchmark tests how well LLMs detect the specific flavor of hallucinations produced by another model. This might create a distribution shift. The benchmark may not perfectly capture the unpredictable hallucinations seen in real-world user inputs.

Second, the BenHalluScore treats Track A and Track B with equal weight. In a production environment, the cost of these errors is rarely symmetric. A false negative (missing a hallucination) might be more dangerous than a false positive (flagging a correct answer). The current metric lacks the flexibility to weight these according to specific risk profiles.

Should You Prototype This

Yes, specifically if you are moving beyond English-only deployments. If you are shipping services in Bengali, you cannot trust standard accuracy metrics. The discovery that smaller, language-specific models like TigerLLM-9B can outperform giants in calibration is a strong signal. This is vital for anyone optimizing for local latency and cost.

The dataset and code are available at https://anonymous.4open.science/r/BanglaHalluEval-EB77. Start by implementing the dual-track scoring logic in your own eval pipeline. It is a low-effort way to catch the label-collapse failures that single-track metrics hide.

Figures from the paper