A Komi-Yazva--Russian Parallel Corpus and Evaluation Protocol for Zero- and Few-Shot LLM Translation

Most work in machine translation focuses on scaling laws and massive datasets. It assumes that more compute and more bitext (parallel text used for training) will solve linguistic nuance. But for thousands of endangered languages, there is no massive dataset to scale. In these extreme low-resource settings, the gap between high-resource English and languages like Komi-Yazva is a total absence of digital representation.

Researchers explore whether Large Language Models (LLMs) can bridge this gap through prompting (asking a model to perform a task without updating its weights). However, evaluating these models in data-scarce environments is difficult. Without rigorous ways to prevent "leakage" (where the model sees test data during the prompting phase), benchmarks yield unreliable conclusions.

This paper introduces the first parallel Komi-Yazva–Russian corpus. It also presents a specialized evaluation protocol to stress-test LLMs in these "micro-language" scenarios. The authors find that retrieval-based few-shot prompting provides a massive boost over zero-shot attempts. However, the benefits hit a ceiling almost immediately.

The failure of naive evaluation in low-resource niches

Current approaches to low-resource machine translation (MT) often fall into two traps. First, they rely on supervised fine-tuning (updating model weights with labeled data). This requires hundreds or thousands of parallel sentences. Many endangered languages simply do not have this luxury. Second, researchers turning to prompting often use naive random sampling for few-shot examples. In a small corpus, this is dangerous. If a model receives a prompt containing sentences from the same story it is currently translating, it performs pattern matching rather than true translation.

Standard MT metrics like BLEU (a measure of n-gram overlap, or matching word sequences) are also brittle. This is especially true in morphologically rich languages (languages where words change form to show grammar). In Komi-Yazva, a single root can take many inflections. A model might produce a perfect translation but fail a BLEU test. This happens if it chooses a different, but valid, suffix. This creates a "measurement gap" where metrics contradict actual semantic accuracy.

A protocol built for leakage control and retrieval

The authors implement a disciplined evaluation pipeline with four key mechanisms:

1. Story-level GroupKFold cross-validation: The authors do not split the 457 sentence pairs randomly. Instead, they group them by their original narrative source (74 stories). This ensures every sentence from a specific story stays together. This prevents the model from "cheating" via narrative overlap in its few-shot prompt.
2. Deterministic character-n-gram retrieval: The protocol uses TF-IDF (a statistical measure of word importance) over character-window n-grams (sequences of 3 to 5 characters). This retrieves the $k$ most similar sentences from the training set. Character n-grams allow the retriever to find matches based on sub-word similarities. This is vital for morphologically rich languages.
3. Strict generation validation: The authors do not simply accept API returns. They run a validation stage to catch "failures." These include prompt leakage, source copying, or providing explanations instead of pure translations.
4. Multi-lens metric reporting: To combat BLEU's brittleness, the authors prioritize chrF (a character n-gram F-score) as their primary metric. Since chrF operates at the character level, it is more forgiving of morphological variations. This provides a more stable signal of quality.

Diminishing returns in the few-shot regime

The empirical results reveal a hierarchy in model performance. Gemini 3.1 Pro is the clear quality leader. It achieves the highest scores across LLM-as-a-judge (using a model to grade translations), BLEU, and chrF [Table 2]. However, there is tension between quality and reliability. While Gemini leads in accuracy, models like Claude Sonnet 4.6 maintain a higher rate of "successful" (valid, non-malformed) outputs.

The most critical finding involves the number of demonstrations ($k$) and performance. As shown in, there is a massive leap in quality when moving from zero-shot ($k=0$) to a minimal few-shot setup ($k=4$).

The aggregate mean chrF score jumps from 19.72 to 31.63. This represents a significant improvement in capturing correct character sequences. Yet, increasing the context to $k=8$ yields only marginal gains.

This "saturation effect" suggests that in extreme low-resource settings, volume is not the priority. Instead, the most important factor is the transition from no guidance to a small, relevant anchor. Once the model has a few correctly aligned examples, adding more adds very little new information.

The limits of the Korpus and the judge

The paper's conclusions are bounded by several constraints. Most obviously, the corpus size (457 pairs) is tiny. Even with story-level grouping, statistical power is limited. A single outlier story could disproportionately skew results for a specific model.

There is also an absence of non-LLM baselines. The authors discuss supervised fine-tuning and pivot-based methods (using a third language as a bridge) in their literature review. However, they do not test them against the LLMs. Thus, we cannot determine if a small, tuned transformer would outperform a massive prompting-based model here. Finally, reliance on "LLM-as-a-judge" remains a secondary signal. No automated metric fully replaces a native speaker. This is especially true for languages with high dialectal variation.

Verdict: A blueprint for low-resource testing

If you seek a model to translate Komi-Yazva, the evidence suggests Gemini 3.1 Pro or Claude Sonnet 4.6. Choose Gemini for peak accuracy or Claude for operational stability.

The real value of this paper is the evaluation template. The authors show that you cannot trust standard MT pipelines for "microlanguages." You must implement story-level splits to prevent leakage. You must use character-based metrics to account for morphology. You must also employ strict validation to filter out malformed outputs. The code and dataset are reportedly available on GitHub and Hugging Face. They provide a necessary, reproducible foundation for research in this field.

Figures from the paper