ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions

The Hidden Fragility of Multilingual Logic

Researchers have created a test called CHLOGIC to see if AI models can actually reason logically when using Chinese. They want to know if models only perform well in English. They found that even powerful models often struggle when the same logical problem is phrased in different Chinese styles. This includes colloquial or rhetorical speech. This reveals a significant gap. A model might solve a math puzzle in English but fail the exact same puzzle in a conversational Chinese tone.

The English-centric reasoning trap

Current large language models (LLMs) are increasingly proficient at standardized logical reasoning benchmarks. However, these benchmarks are overwhelmingly dominated by English. Most existing datasets rely on explicit logical connectives. These are words like "if," "unless," or "all" that act as clear signposts. While high performance on these English tests is impressive, it does not prove true reasoning.

The authors of the CHLOGIC study argue that current benchmarks ignore how different languages package information. In Chinese, logical relations are often obscured by pragmatic nuances. These include rhetorical questions, idioms, or omitted subjects. The study highlights a failure in "normalization" (the process of converting messy, natural language into a clean, stable logical structure). As shown in, a model might achieve near-perfect accuracy on a logical problem in English.

However, it suffers a massive performance collapse when that same logic is expressed through a rhetorical Chinese variant. This suggests that models fail to extract the underlying logic from the linguistic surface.

Building a controlled stress test

To isolate this problem, the researchers developed CHLOGIC using a "template-first" construction method. Instead of asking a model to reason through random text, they started with formal logical templates. These are mathematical blueprints of a logical argument. They worked backward from these templates to natural language. This ensures the underlying logic remains constant even when the wording changes.

The construction workflow, detailed in, follows three distinct stages:

1. Logical Template Design: Researchers define the core logical skeleton. They also assign a gold-standard label (YES or NO).
2. Dataset Composition: The benchmark has three subsets. There is a "General aligned set" for basic propositions. There is a "Difficult aligned set" for complex mathematical-logic problems. Finally, there is a "Chinese-only set" to test 15 specific linguistic phenomena. These include polysemy (words with multiple meanings) or ellipsis (omitting words implied by context).
3. LLM-Assisted Generation and Quality Control: Models like DeepSeek-V3 draft various Chinese realizations. These range from standard written text to colloquial speech. Humans perform the final verification. This ensures the logic hasn't drifted during translation.

By pairing one English reference with five different Chinese realizations, the authors created a controlled environment. The only variable is the linguistic "surface" of the problem.

Evidence of a persistent linguistic gap

The experimental results reveal a stark disparity between English proficiency and Chinese robustness. The paper reports that strong models like GLM-5.1 achieve 98.30% accuracy on General English questions. Yet, this drops to 78.89% when faced with rhetorical Chinese variants. The gap widens significantly in the "Difficult" set. The authors find that the Qwen3-32B model scores 96.05% on difficult English problems. However, it falls to 69.35% on difficult rhetorical Chinese. This represents a roughly 26-point drop in reliability.

The researchers also used "back-translation probes" to diagnose the cause of these errors. They translated standard Chinese back into English and re-tested the model. This helped them see if errors occurred during reasoning or language processing. On the General aligned set, back-translation frequently restores performance to near-English levels. For example, Qwen3-8B improved from 90.53% to 99.10% after being fed the back-translated English.

However, the effect is not always positive. In the Difficult aligned set, the authors observe mixed results. For models like Qwen3-32B and GLM-5.1, back-translation actually decreased performance. This indicates that for complex problems, translation might strip away helpful Chinese linguistic cues. It might also introduce new ambiguities that confuse the model.

Limitations of the diagnostic approach

While CHLOGIC provides a rigorous stress test, the authors acknowledge several boundaries. First, the benchmark is a specialized diagnostic tool. It is not a complete measure of all Chinese reasoning. Because it is driven by formal logical templates, it may not capture all discourse-level reasoning.

Second, the reliance on back-translation carries inherent risks. The authors note that back-translation is a "model-mediated transformation." This means the act of translating the text might introduce new artifacts. It might also simplify the language in ways that do not reflect real-world usage. Finally, the study is limited to the specific models tested. While trends appear consistent across the Qwen, Ministral, and GLM families, results may evolve as newer models are released.

The verdict on multilingual reasoning

Is the reasoning of modern LLMs truly robust? Based on the evidence in this paper, the answer is: not yet.

The study demonstrates that high-tier English logical performance is a poor predictor of success in Chinese. The persistent gap suggests that current models possess a "fragile normalization step." They can reason once the logic is laid bare in English. However, they stumble when extracting that logic from varied Chinese expressions. For engineers building multilingual systems, the takeaway is clear. Optimizing for English-language benchmarks will leave a significant blind spot. Your model may still struggle with real-world, non-canonical communication in other languages.

Figures from the paper