OpenSTBench: Beyond Semantic Evaluation for Speech Translation

Can We Measure More Than Just Words?

Current speech translation tools are usually judged only on how well they translate words. This new framework, OpenSTBench, looks at the whole picture. It includes how natural the voice sounds, if the speaker's identity is preserved, and how fast the translation happens in real-time.

The multi-dimensional problem of speech translation

As speech translation evolves, the definition of success is shifting. It moves from simple speech-to-text (S2TT)—converting audio into translated text—to complex speech-to-speech (S2ST) systems. These newer systems generate synthetic voices. A system might produce a grammatically perfect translation. Yet, it can fail if the output voice sounds robotic. It can fail if it lacks the original speaker's emotion. It can also fail if it arrives too late for a live conversation.

The authors of OpenSTBench investigate a central question. Can we systematically compare these heterogeneous systems? These systems range from offline text generators to real-time voice cloners. They want to know how "good" manifests across different, often conflicting, dimensions.

Cracks in the semantic-only paradigm

Historically, the field has relied on machine translation metrics. One example is BLEU (which measures how many words or phrases match a reference translation). Another is COMET (a neural framework that measures semantic similarity, or meaning). These are essential for linguistic adequacy. However, they are "blind" to the acoustic reality of S2ST.

If you rely only on text-based metrics, you miss the voice quality. A monotone, lifeless voice might produce a perfect translation. But it lacks the speaker's unique timbre (the quality of a voice) or emotional urgency. Previous efforts tried to patch this. They applied speech-side measures like speaker similarity. They also used streaming measures like latency (the delay before a system responds). However, these are usually applied under separate, task-specific protocols. This fragmentation makes it hard to compare a streaming S2ST model against an offline S2TT baseline. There is no common ground to decide which system is better for a specific use case.

A unified architecture for heterogeneous outputs

To fix this, the researchers developed OpenSTBench. This framework organizes diverse outputs into a shared evaluation format. As shown in, the framework maps various inputs into a unified sample record.

These inputs include source speech, reference text, and timing logs. This allow the system to trigger different "modules" based on the output. If the model only outputs text, the framework runs translation metrics. If it outputs audio, it triggers speech-quality evaluators.

The study tested many representative systems. These included streaming models like Qwen3-LiveTranslate and GPT Realtime Translate. It also included offline models like SeamlessM4T-v2-Large and UniSS. The researchers used specialized datasets to test specific dimensions. They used LibriTTS for speaker preservation and RAVDESS for emotion. They used MSLT for general translation quality. By structuring the evaluation this way, they could map the "profile" of each system across many dimensions.

Disjointed rankings and the death of the global leaderboard

The most striking finding is that there is no single "winner" in speech translation. Instead, systems show massive, cross-dimensional trade-offs. The authors demonstrate this through normalized radar plots in .

These plots visualize how different models occupy different niches.

For instance, the paper finds that Qwen3-LiveTranslate leads in translation quality. It reported a BLEU of 43.27 for EN→ZH in [Table 5]. This means it has high linguistic fidelity. However, it does not necessarily dominate in speech-side qualities. Similarly, [Table 6] shows that speech naturalness (measured via UTMOS) and text realization (measured via CER/WER) do not always correlate. This means a model can sound natural while failing to say exactly what was intended.

Temporal quality results in [Table 7] are also critical. They reveal that streaming responsiveness is often decoupled from temporal consistency. Responsiveness is how quickly a system starts talking. Temporal consistency is how well the output matches the source duration. A system might be very fast but produce a translation that is out of sync with the original speaker's pacing. This confirms the authors' hypothesis. A single global ranking is a misleading metric for this field.

Moving toward application-oriented selection

The implications of this work suggest a shift in how we approach speech translation. If these findings generalize, the industry must move away from chasing a single "state-of-the-art" number.

First, practitioners must drive model selection by specific utility profiles. A teleconferencing tool will prioritize low latency and speaker preservation. A subtitling service for recorded media will prioritize translation accuracy and offline efficiency. The OpenSTBench framework provides the math to make these trade-offs explicit.

Second, for researchers, this highlights a clear frontier. Paralinguistic fidelity (preserving non-verbal cues like laughter or sighs) remains weak across all tested systems. This suggests that current architectures struggle to bridge the gap between semantic translation and holistic acoustic mimicry. Researchers must now optimize for these missing dimensions.

Figures from the paper