KletterMix: Climbing Toward High-Quality German Pretraining Data

After reading this, you will know whether translating high-quality English datasets is a viable shortcut for building specialized models in other languages. You will also see how to implement a quality-controlled translation pipeline. The catch is that while this method boosts reasoning, it can also import the cultural and topical biases of the English-speaking world into your target language.

Modern large language models (LLMs) depend heavily on the quality and diversity of their pretraining data. This is the massive corpus of text used to teach the model fundamental language patterns and facts. While English-language resources are vast and meticulously curated, many other languages suffer from a "data gap." German-language resources, for instance, are often smaller, noisier, and less documented than their English counterparts.

Current state-of-the-art approaches typically rely on crawling the native web for German text. However, raw web crawls are notoriously messy. They are often filled with spam, duplicate content, and low-quality fragments. This creates a dilemma. Do you settle for the massive scale of noisy native data, or the high quality of scarce, curated data? A new study introduces KletterMix. This approach suggests a third way. Researchers created a massive new German dataset by carefully translating a high-quality English dataset. They preserved its original structure and topical diversity. This ensures the model learns complex reasoning rather than just surface-level patterns.

Transferring Curation via Translation

The KletterMix method takes a high-quality English pretraining mixture called ClimbMix. It transforms this into a 725B-token German corpus. Given an English source corpus with rich metadata and diverse topics, the pipeline produces a functionally equivalent German corpus. It retains document boundaries, source identifiers, and topical distribution. As shown in, the process is not a simple "translate everything" command.

It is a structured pipeline. It routes documents into length-aware buckets and uses contextualized chunking to maintain coherence.

What You Need To Run It

The authors provide the dataset at https://huggingface.co/datasets/AIML-TUDA/KletterMix and the code at https://anonymous.4open.science/r/KletterMix-5F3F.

To replicate the full-scale construction, the study utilized a massive infrastructure. It consisted of 126 nodes, each equipped with 8 NVIDIA B200 GPUs. The translation itself was powered by the Qwen3.5-397B-A17B-FP8 model. For practitioners looking to run smaller-scale experiments, the paper demonstrates successful results using the Qwen3-0.6B architecture. The research is highly reproducible. The authors disclose the complete training hyperparameters. These include a global batch size of 512 sequences and a cosine learning-rate schedule (a method where the learning rate decreases following a cosine curve).

How It Works

The core innovation lies in treating translation as a dataset-construction problem. This is more than a simple augmentation step. To prevent the "lost in translation" effect, the authors implemented "document-preserving contextualized chunking." This prevents long documents from becoming fragmented or incoherent. When a document is too long for a single pass, it is split into chunks. The model is then fed a truncated window of the previous German translation as context for the current chunk. This acts like a relay race. Each runner (chunk) knows what happened in the previous leg to maintain discourse continuity.

Efficiency is managed through "length-aware routing." In this process, documents are assigned to context buckets (ranging from 4k to over 64k tokens). This minimizes wasted computation. Furthermore, the authors avoid the common pitfall of fixed-length truncation. They use a dynamic target-side budget. This formula, $\ell_{tgt}^{max} = \min (L_{max}, \lceil \alpha \ell_{src} + \beta \rceil)$, allows the output limit to expand or contract based on the source length. This accommodates the fact that German text often expands in length compared to English.

To handle the scale of 725B tokens, the authors developed a two-tier quality estimation system. First, they used COMETKiwi—a reference-free quality estimation model (a tool that scores translation quality without needing a perfect human reference)—to score a pilot subset. Second, they trained a "target-only proxy." This is a lightweight gradient-boosted regressor (a machine learning model that combines many weak predictors). It predicts quality scores by looking only at the generated German text. It also uses inexpensive metadata like character ratios and language identification confidence. This allows for massive, cheap filtering of the full corpus. It does so without needing to reload the original English source.

How To Tell If It Worked

Success is measured through two primary signals: training dynamics and downstream reasoning. The authors report that models trained on KletterMix exhibit significantly lower training and validation loss compared to established German baselines like FineWeb2-DE and GermanWeb .

Lower loss means the model is better at predicting the next token in the sequence.

More importantly, the study finds that KletterMix provides a superior "steering signal" for reasoning. In benchmark evaluations, KletterMix-trained models outperformed baselines on tasks requiring logical inference and scientific reasoning. Specifically, they excelled on HellaSwag (which tests event-level plausibility) and ARC-Challenge (which tests science-style reasoning). Even when used only for "annealing"—the final stage of training where a model's knowledge is sharpened—KletterMix improved performance on these reasoning tasks compared to using native German web data.

Gotchas

There are several ways this pipeline can fail if not monitored. One major risk is "translationese" or semantic drift. This occurs when the nuances of the original text are lost. The text might be replaced by unnatural, overly literal German phrasing. The authors note that while the proxy model is excellent at catching "broken" text, it has limits. For example, it can catch code being translated into prose. However, it cannot directly measure if the actual meaning remains accurate.

Another failure mode involves "length pathologies." As seen in [Figure 3b], some documents in long-context buckets unexpectedly result in very short German outputs. The authors identify these as suspicious cases. They might indicate truncation or dropped content. Finally, users should watch for "wrong-language" leaks. As illustrated in the qualitative examples in Appendix B.5.1, translation models sometimes default to dialectal variations (like Swiss-German). They might also accidentally translate technical identifiers (like Python variables) into German. This can corrupt the training signal for technical domains.

When This Is The Wrong Tool

KletterMix is not a magic bullet for cultural representation. Because the foundation is an English corpus, the resulting German data inherits certain biases. It carries the topical, geographic, and cultural biases of the English-speaking world. If your goal is to build a model that deeply understands German regional nuances, this may not be enough. It might lack local folklore or specific European socio-political contexts. It is a tool for scaling reasoning and breadth. It is not designed for capturing indigenous cultural depth.

Figures from the paper