AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task

Researchers have found a way to make powerful AI models like Gemma translate speech in real-time. They do this without waiting for the entire sentence to finish. They "watch" the model's internal attention mechanisms to see if it has processed enough spoken words to safely commit to a translation.

In simultaneous speech translation, the goal is to minimize lag. This is the delay between a speaker talking and a listener seeing the translated text. Traditionally, this required specialized encoder-decoder architectures. Those models used "cross-attention" (a mechanism where the decoder looks at the encoder's output) to see which parts of the source audio were being focused on. However, the industry has moved toward massive, decoder-only Large Language Models (LLMs). These models have superior reasoning and multilingual capabilities. The question was whether we could adapt these decoder-only models to act like simultaneous translators.

The Problem

The status quo relies on waiting for full sentences (high latency) or using "LocalAgreement" policies. These policies try to predict when a model is ready to commit. However, they often introduce significant latency. They do not leverage the model's internal state effectively.

The core issue is that decoder-only LLMs treat everything as one continuous, causal string of tokens. This includes system instructions, the source transcript, and the target translation. There is no explicit "bridge" between the source and the target. Earlier AlignAtt systems relied on reading encoder-decoder cross-attention to find the "source frontier" (the limit of processed audio). Without that cross-attention, a standard LLM has no native way to signal its progress. It might guess the end of a sentence before the audio arrives. This leads to "flicker" or hallucinations.

How It Works

The authors propose AlignAtt4LLM. It adapts the AlignAtt policy to the decoder-only substrate through four technical pillars.

1. Explicit Source Spanning: To solve the lack of a cross-attention bridge, the authors structure the prompt deterministically. As shown in, the prompt is partitioned into a clear hierarchy.

It includes the system prompt, live transcript, translation instruction, and the accepted target prefix. This creates a known map from source-word indices to specific positions in the prompt.

1. Offline Head Calibration: Not all attention heads are useful for translation. The authors perform an offline calibration using a larger model (GPT-5-mini) to identify heads that track source-target alignments. They select a sparse set of the top 8 heads per language pair. These are typically concentrated in the later layers of the model .

2. Runtime Q/K Capture: Modern production environments like vLLM use fused kernels (optimized code that combines multiple operations). In these kernels, the $n \times n$ attention matrix is never visible to the user. To bypass this, the authors implement a "per-layer observer." This observer intercepts the post-normalization Query ($Q$) and Key ($K$) tensors during the forward pass .

They use a zero-valued sentinel dependency to prevent the compiler from removing this observer.

1. Selective qk-fast Replay: The system does not recompute the massive full attention matrix. Instead, it uses a "qk-fast" path. It recalculates only the specific "draft-to-source" block needed for the policy .

This block calculates two provenance features. These features measure the mass of attention on the "accessible" side of the source frontier versus the "inaccessible" side .

If the attention peak shifts too far into inaccessible territory, the policy rejects the draft.

Numbers

The authors report that AlignAtt4LLM achieves a quality boost in the low-latency regime. For English to German (EN→DE), they report a BLEU score (a metric for translation quality) of 28.76. This beats the organizer's baseline of 22.35 at roughly 2 seconds of latency. For English to Italian (EN→IT), they report a XCOMET-XL score (a metric for translation adequacy) of 0.805. This beats the 0.683 baseline.

The "qk-fast" implementation is efficient enough for production. The median inference cost per token for the vLLM implementation is 25.4 ms. This is 2.5x faster than a standard Transformers "eager" reference implementation . Using all 336 heads instead of the selected 8 is counterproductive. While quality remains similar, latency increases by up to 179.5 ms in the computational-aware (CA) setting [Table 2]. This happens because reconstructing the attention state becomes much more expensive.

What's Missing

Results for English to Chinese (EN→ZH) are mixed. The authors note that the Gemma-4 backbone struggled with Chinese. The system trailed the baseline in BLEU and XCOMET-XL scores. This suggests the policy's success depends on the underlying model's linguistic competence.

The system is also a synchronous cascade. As shown in, the ASR update and the MT request are serialized.

This isolates policy effects from scheduler noise. However, it may leave performance on the table. An asynchronous pipeline could potentially reduce latency. But it would introduce complex race conditions regarding the source frontier.

Finally, the paper recommends a 250 ms "hold-back" on the ASR tail to handle transcription instability .

This improves reliability. However, it is a manual heuristic that adds a hard floor to the achievable latency.

Should You Prototype This

Yes, if you are building real-time translation services using decoder-only LLMs. The $Q/K$ capture method is an architectural win. It avoids the overhead of materializing full attention matrices. The implementation is reportedly available at https://github.com/QuentinFuxa/AlignAtt4LLM.

Do not expect a "plug-and-play" experience for all languages. You must invest time in the offline calibration step. You need to find the right heads for your specific backbone and language pair. For non-European languages, prioritize finding a translation-optimized backbone. The Gemma-4 results suggest that backbone quality is a major limiting factor.