UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception

Current audio AI models excel at either understanding words in speech or hearing general sounds like rain or music. They rarely do both well. Most systems face a trade-off. Use semantic tokenizers (compressed representations focused on meaning) that are "deaf" to everything else. Or use acoustic tokenizers (representations focused on waveform detail) that lack the semantic structure needed for an LLM to reason.

A new paper introduces UniAudio-Token. This framework aims to bridge this gap. It seeks to create a single, compact interface. This interface supports both high-fidelity speech generation and universal audio perception. It aims to "hear" a chainsaw or a heartbeat as clearly as a spoken sentence.

The Problem

The status quo in Audio-LLMs is a fragmented landscape. If you want an agent that understands speech, you typically use a semantic speech tokenizer. These models use ASR (Automatic Speech Recognition) encoders. These encoders are trained to strip away "noise." This includes vocal textures and background ambiance. This process isolates text but creates "acoustic blindness."

As shown in, current tokenizers fall into two camps.

Semantic-centric models like GLM-4-Voice-Tokenizer map distinct non-speech events to overlapping regions in latent space. Acoustic-centric models like WavTokenizer lack the semantic guidance to distinguish between meanings.

This leaves engineers with a difficult choice. You can build a speech-only model. Or you can manage a heterogeneous mess. This involves using continuous encoders for perception and discrete tokenizers for generation. Such setups add complexity to the inference pipeline. They also complicate the unified modeling paradigm that makes LLMs powerful.

How It Works

The authors refuse to choose between semantics and acoustics. They propose balancing them through two mechanisms: Semantic-Acoustic Primitives (SAP) and Semantic-Acoustic Equilibrium (SAE).

1. Structured Supervision (SAP): To solve the supervision conflict, the authors go beyond simple transcription. They use an LLM-based pipeline to generate structured JSON annotations. These "primitives" decompose audio into three layers. These are Linguistic Content (the transcript), Vocal Attributes (age, gender, emotion, etc.), and Auditory Scenes (transient or persistent environmental events). This forces the model to allocate capacity to acoustic details. It does so without muddying the semantic backbone. Examples appear in, and .

2. Adaptive Feature Fusion (SAE): Deep semantic encoders naturally discard fine-grained details in higher layers. To fix this, the authors implement the SAE module .

SAE uses a content-aware gating mechanism. It takes high-level semantic features from deep layers ($H_{deep}$). It then adaptively injects acoustic details from shallower layers ($H_{shallow}$). It uses a learnable sigmoid gate $g$ (a mathematical function that outputs values between 0 and 1).

This gate is not static. As shown in, the gate is "modality-aware." It stays suppressed during clear speech to maintain linguistic abstraction.

It ramps up activation during music or in low SNR (Signal-to-Noise Ratio, the ratio of signal strength to background noise) environments. This recovers the lost acoustic texture.

Numbers

The authors report that UniAudio-Token breaks the semantic-acoustic trade-off. In clustering analysis on the ESC-50 dataset, UniAudio-Token was the only model to achieve positive Silhouette Scores. This score measures how well items are clustered. Positive scores indicate valid cluster separation [Table 2]. Visualizations in confirm this.

They show well-separated clusters compared to the "feature fragmentation" seen in baselines.

On downstream Audio-LLM tasks, the impact is measurable. When integrated with a Qwen2.5-3B backbone, UniAudio-Token outperformed all single-codebook baselines on the MMAU benchmark. Notably, it hit 70.27% on sound tasks. This is a significant jump over the semantic-centric GLM-4-Voice-Tokenizer [Table 4].

The model also maintains speech quality. The authors report that UniAudio-Token actually improved speech reconstruction fidelity. It achieved a lower Word Error Rate (WER, a measure of transcription accuracy) of 3.47 on LibriSpeech. This outperformed WavTokenizer (5.07) and CosyVoice2 (4.25) [Table 3].

What's Missing

There are gaps that a practitioner should note. First, the paper admits a limitation in reconstruction. Waveform-level reconstruction for complex, non-speech audio still lags behind specialized, high-bitrate acoustic codecs. If your goal is professional music production, this single-codebook approach may not suffice.

Second, the evaluation scope is narrow. Training and testing center heavily on English and Chinese. We do not yet know how SAP supervision scales to other languages. We also do not know how "vocal attributes" labels generalize across different linguistic families.

Finally, the computational cost of the SAP curation pipeline is not fully quantified. The authors use heavy LLMs like Qwen3-235B to label data. While this is an offline training cost, it implies high resource requirements for scaling.

Should You Prototype This

Yes, if you are building a general-purpose Audio-LLM. If your roadmap moves from "chatting about speech" to "understanding the world through sound," this architecture offers a cleaner path. It avoids managing separate perception and generation encoders. Using a single codebook for both tasks simplifies the entire stack.

The code and model checkpoints are reportedly available at https://github.com/Tencent/Universal_Audio_Tokenizer. If you struggle with "acoustic blindness" in semantic models, the SAE gating mechanism is a worthwhile addition to test in your training loops.