X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding

Current MLLMs Are Blind to the Room

Current AI models are excellent at watching a single video. However, they struggle when watching multiple camera feeds simultaneously. Most existing benchmarks use single-stream paradigms (processing one continuous flow of data). This leaves a gap in evaluating real-time, cross-stream reasoning. This is a hurdle for live sports broadcasting, autonomous driving, or multi-screen collaboration. These tasks require coordinating info from different angles and devices.

The authors find that state-of-the-art Multimodal Large Language Models (MLLMs) are inept at this task. When acting as "multiplexers" (systems that combine multiple signals into one), they achieve only about 50% accuracy. They also fail significantly at proactive reasoning (predicting or waiting for future events).

The Problem

Standard video understanding relies on single-stream processing. The model consumes one continuous flow of tokens (the basic units of data processed by the model). But real-world perception is rarely a single line of sight .

A sports broadcaster manages dozens of cameras. An autonomous vehicle fuses front-facing cameras with side sensors. A robot might synchronize a shoulder mount with a wrist camera.

Existing multi-video datasets lack "streaming" characteristics. They lack the requirement for continuous, online inference with temporal alignment (matching timestamps across feeds). Most current benchmarks treat multi-view tasks as offline processing of complete files. They miss the core difficulty of streaming. The model must monitor incoming data in real-time. It must decide exactly when to respond to a query. Many datasets also suffer from "single-stream shortcuts." In these cases, a model answers a question by looking at only one stream .

This bypasses the need for true multi-stream cooperation.

How It Works

The authors introduce X-Stream. This benchmark treats MLLMs as "naive multiplexers." Since an MLLM handles only one token stream at a time, researchers tested three strategies to fit multiple streams into a limited token bandwidth :

1. Spatial Division Multiplexing: This method stitches multiple streams into a single frame via pixel-level concatenation. It creates a "grid" of videos. Vertical-level stitching generally outperforms horizontal stitching [Figure D14]. This is because it avoids interleaving tokens during the model's raster-scan (reading pixels row-by-row).
2. Time Division Multiplexing: Here, streams are processed as independent inputs. The model assigns them identical temporal embeddings (mathematical representations of time) to maintain synchronization. This allows the model to jump between streams.
3. Semantic Division Multiplexing: This is the most sophisticated approach. It uses a Determinantal Point Process (DPP) kernel to select tokens [Equation 1]. This tool picks a subset of items that are both relevant to a query and diverse. This ensures critical visual information is preserved under tight token budgets.

To ensure the benchmark tests multi-stream intelligence, the authors used a dual-verification pipeline. They use a "Sufficiency and Necessity" protocol. A question is included only if a model can answer it when all streams are present (sufficiency). It must also fail if any single stream is removed (necessity).

Numbers

The results show a massive performance gap. Top-tier proprietary models like Gemini 3 Pro achieve high scores on single-stream tasks. However, their performance drops to approximately 49.60% on the X-Stream benchmark [Table 2]. This means even the best models fail half the time in multi-stream settings. The leading open-source model, Qwen3-Omni-30B-A3B, reaches only a 34.28% overall score [Table 2].

Failures follow a pattern. Models struggle most with "agency" and "decision-making" tasks. These require high-level logical cognition. Foundational grounding (like counting objects) is relatively stable. However, "proactive" tasks are a major bottleneck [Table 2]. These tasks require the model to respond only when a specific event occurs.

Multiplexing strategies involve clear trade-offs. Spatial division is excellent for cross-stream referencing. But as the number of streams hits three or more, visual info becomes too blurred. In those dense scenarios, Semantic Division is superior [Table 5]. It preserves critical information under strict token constraints.

What's Missing

There are gaps a practitioner should note. First, proprietary models cannot be evaluated using Semantic Division. Researchers cannot manipulate token-level input for closed APIs. This leaves a gap in understanding how top-tier models behave with semantic pruning.

Second, the impact of audio multiplexing is limited. The authors acknowledge that audio signals have stronger "coupling" (overlap) than pixels. Spatial Division causes audio tracks to overlap. However, the paper does not provide a rigorous framework for multiplexing multi-channel audio without creating noise.

Finally, the hardware footprint of Semantic Division is not fully quantified. This includes the overhead of the DPP kernel and MAP (Maximum A Posteriori) inference. For real-time streaming, the latency of this selection might outweigh the benefits of fewer tokens.

Should You Prototype This

Depends on your use case. If you are building a single-camera assistant, X-Stream won't change your roadmap. But if you work on multi-camera surveillance or autonomous fleets, this is a mandatory benchmark.

The "multiplexer" framework is a solid architectural blueprint. If you hit OOMs (Out of Memory errors) or high latency, prototype the Semantic Division approach. It is the only strategy that scales gracefully to 3+ streams. Code and data are reportedly available; see the paper for the canonical link.

Figures from the paper