SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses

Researchers created a new test called SPLIT to see if AI can provide emotional support in both English and Ukrainian. They found that while AI can speak Ukrainian, it often fails to provide the same level of cultural understanding and natural empathy that it does in English.

The multilingual fallacy

Current Large Language Model (LLM) benchmarks focus heavily on multilingual performance. This refers to the ability of a model to process text in various languages. However, the authors of the SPLIT study argue that being multilingual is not the same as being multicultural. A model might successfully navigate the grammar of Ukrainian. Yet, it may still fail to grasp subtle emotional markers or social norms.

This gap is particularly dangerous in crisis-related contexts. These include support for stress, panic, or internal displacement. As shown in, there is often a significant downward trajectory in performance.

This occurs when transitioning from high-resource languages like English to mid-resource languages like Ukrainian. Current benchmarks fail to catch this. They prioritize linguistic fluency over "empathic grounding" (the ability to acknowledge a user's specific cultural reality).

Measuring the emotional gap

To move beyond simple translation checks, the authors developed the SPLIT benchmark. This is a collection of 500 distinct emotional support queries. These cover five psychosocial domains: Stress, Panic, Loneliness, Internal Displacement, and Tension. The methodology evaluates models across three dimensions: Empathetic Accuracy, Linguistic Naturalness, and Contextual & Cultural Grounding.

The researchers chose three architecturally diverse models to test the findings: 1. DeepSeek-V3: A Mixture-of-Experts (MoE) model. In an MoE architecture, the model routes inputs to specialized "expert" sub-networks. This is similar to how a hospital routes patients to specific specialists. 2. LLaMA-3.3-70B-Instruct: A standard dense transformer model. In this architecture, every parameter is activated for every calculation. 3. Gemini-2.5-Flash: A hybrid model. It utilizes a sparse mixture-of-experts approach to balance speed and intelligence.

The evaluation uses a dual-track system. First, an "LLM-as-a-jury" paradigm employs high-reasoning models to automate the scoring. These judges include GPT-4o, Mistral Large, and Claude 4.5 Sonnet. Second, the authors perform human validation. A native Ukrainian speaker manually grades a 10% sample of the responses .

Stability versus degradation

The results reveal a stark divide in how different architectures handle the transition to Ukrainian. The paper finds that Gemini-2.5-Flash and LLaMA-3.3-70B-Instruct suffer significant performance degradation. Most notably, LLaMA’s Linguistic Naturalness scores plummeted. They fell from 3.92 in English to just 2.16 in Ukrainian .

This represents a massive drop in how natural and idiomatic the model sounds.

In contrast, the authors report that DeepSeek-V3 remained remarkably stable. DeepSeek actually showed slight improvements in certain Ukrainian metrics. Its Empathetic Accuracy increased from 3.14 to 3.56. Its Contextual & Cultural Grounding rose from 3.54 to 3.74. The authors suggest this stability may stem from DeepSeek's MoE routing. It may also come from its "Multi-Token Prediction" training objective. This objective allows for better phrase-level planning in morphologically rich languages.

However, the most critical finding concerns the reliability of AI judges. When comparing human scores to the AI jury, the authors find a weak correlation for empathy and naturalness. There is a complete lack of meaningful alignment for cultural grounding. In fact, the AI jury showed a negative correlation ($r = -0.095$) with human judgments on cultural grounding .

This means the AI jury actually moved in the opposite direction of human intuition. An AI judge might reward a response for being grammatically correct. However, it remains blind to whether the response feels culturally authentic.

Limits of the automated jury

The study highlights a major risk for engineers building automated evaluation pipelines. The "LLM-as-a-jury" paradigm is prone to systematic leniency bias. This is a tendency where judges give higher scores than humans would. The authors report that AI judges tend to overscore responses. They reward superficial linguistic indicators like structure and politeness. They miss the lack of deep emotional resonance.

There are several caveats to consider. The human validation relied on a single native speaker. Therefore, the results may reflect that individual's specific subjectivity. Furthermore, the benchmark is specialized. It focuses exclusively on five crisis categories. It does not claim these findings apply to medical or legal communication. Finally, the study is limited to the English-Ukrainian language pair. The full scale of empathy divergence in other languages remains unknown.

The verdict

If you are developing AI for emotional support, the verdict is clear. Do not rely on automated multilingual benchmarks alone.

The authors demonstrate that producing Ukrainian text is fundamentally different from producing Ukrainian emotional support. While models like DeepSeek-V3 show promise in maintaining stability, most current models fall into a trap. They provide "robotic" empathy when they leave English-centric training data. For practitioners, "multilingual" is a baseline requirement. However, "multicultural" is the actual goal. Code and datasets for the SPLIT benchmark are reportedly available; see the paper for the canonical link.

Figures from the paper