Comprehensive Benchmarking of Long-Form Speech Generation in Diverse Scenarios

Current AI speech generation has mastered the art of the perfect sentence. It falters once the transcript turns into a page. Researchers have developed tools that can mimic a single voice with startling fidelity. Yet these systems often lose their way when tasked with reading an entire audiobook or hosting a multi-turn podcast. They might get the words right. But the persona drifts, the room acoustics shift unnaturally, and the emotional energy flatlines into a robotic monotone.

A new paper introduces SwanBench-Speech. This is a benchmark designed specifically to expose these long-form failures. Rather than just checking if the words are right, it tests if the voice stays consistent. It checks if the audio sounds natural and shows emotion throughout a long sequence. The authors find that even the most advanced proprietary models still struggle significantly. They struggle with reverb consistency (the stability of the simulated room environment), prosodic coherence (the naturalness of rhythm and pauses), and paragraph-level expressive hierarchy (the emotional arc of a story) compared to real human recordings.

The Problem

Standard evaluation for Text-to-Speech (TTS) is broken for long-form content. Most existing benchmarks focus on sentence-level metrics like Word Error Rate (WER). WER measures how accurately a model transcribes text. Other benchmarks use simple Mean Opinion Scores (MOS) for short clips. These metrics are essentially "saturated" for modern models. They tell you the model is accurate. However, they fail to capture the temporal decay that happens over minutes of audio.

In long-form synthesis, you encounter three specific failure modes. First is acoustic drift, where the simulated room environment changes mid-speech. Second is semantic breakdown, where the model loses the thread of the narrative. Third is expressive monotony, where the "acting" becomes a flat, unchanging drone. As seen in, many models exhibit a clear performance degradation as the number of sentences increases.

Current benchmarks lack the granularity to distinguish between a model that is "accurate but boring" and one that is "accurate and engaging."

How It Works

The authors propose SwanBench-Speech. This is a hierarchical framework that decomposes long-form quality into three axes: Acoustics, Semantics, and Expressiveness .

To build a meaningful test bed, they constructed a dataset of 1,101 samples. These span 17 different scenarios, ranging from news broadcasting to dramatic storytelling .

The core innovation is the automated evaluation protocol. It moves away from coarse-grained comparisons toward seven disentangled metrics:

1. Acoustic Metrics: They measure Timbre Consistency (using WavLM embeddings to ensure the speaker doesn't "morph" over time). They also measure Reverb Consistency (using the Speech-to-Reverberation Modulation Energy Ratio, or SRMR, to check if room acoustics stay stable). Finally, they use Sound Fidelity (via SQUIM-PESQ, a reference-free way to judge audio clarity).
2. Semantic Metrics: They use ASR-based Word Error Rate (WER) for content accuracy. They also use a specialized Large Audio-Language Model (LALM) evaluator to score Prosodic Coherence. This measures the naturalness of pauses and rhythm.
3. Expressiveness Metrics: This is the most ambitious part. They use LALMs to score "Expressive Richness" (the average emotional impact of 10-second segments). They also score "Expressive Hierarchy" (the paragraph-level emotional arc).

By using LALMs as "judges," they attempt to automate the nuanced qualitative assessment. This replaces expensive human listening tests with a scalable proxy for perception.

Numbers

The results reveal a massive gap between synthetic audio and reality. The authors report that while models can match human recordings in sound fidelity, they fall significantly behind in expressiveness. Specifically, closed-source models lag behind real speech by nearly one MOS point in richness. They also lag by over half a point in hierarchy.

In the dialogue generation setting, the problem gets worse. The paper finds a marked gap in Reverb Consistency. Models scored 3.36 compared to 2.73 for real dialogue [Table 3]. This highlights the difficulty of maintaining a unified acoustic scene when multiple speakers are involved.

The trade-offs between model architectures are also quantified. Autoregressive (AR) models predict tokens one by one. They excel at prosody but suffer from error propagation. This leads to lower content accuracy as sequences grow. Non-autoregressive (NAR) models generate audio in parallel. They are much more stable for content accuracy. However, they tend to produce "over-smoothed" rhythms that lack emotional nuance .

What's Missing

The paper is thorough, but there are gaps. First, the linguistic scope is narrow. The benchmark is restricted to English and Chinese. If you are building a product for a low-resource language, these findings may not translate.

Second, the evaluation of expressiveness relies heavily on closed-source LALMs like Gemini 3 Pro. While the authors validated these models against human scores, this creates a reproducibility bottleneck. If the API's underlying weights or prompting sensitivities change, your benchmark scores might drift.

Finally, the "Expressiveness" dimension is still somewhat fuzzy. While the paper uses LALMs to score "emotional arcs," it lacks a robust automated framework for deep semantic understanding. It measures the result, but not necessarily the control over stylistic transitions.

Should You Prototype This

Yes, but with a caveat. If you are building anything beyond a simple "read this sentence" utility, you should adopt this logic. This includes automated audiobook engines or long-form virtual assistants. The metrics for Reverb and Timbre consistency are vital. They help you prevent "uncanny valley" effects in production.

The code and demo are reportedly available at https://swanaigc.github.io/#bench. Don't wait for a "perfect" model to arrive. Use this benchmark to identify exactly where your current stack is breaking. If your model's Expressive Hierarchy score is tanking as the text length increases, look at your long-term dependency modeling. You should also consider the temporal continuity of your training data.

Figures from the paper