AdaCodec: A Predictive Visual Code for Video MLLMs

Instead of treating every video frame as a heavy, standalone image, AdaCodec only sends full images when something new happens. For everything in between, it sends tiny "update" packets that describe only the movement and changes. This approach allows video multimodal large language models (video MLLMs) to understand long sequences without drowning in redundant visual tokens.

The Problem

Current video MLLMs suffer from a massive temporal redundancy problem. Most models treat a video as a sequence of independent RGB frames. Each frame is encoded into its own set of hundreds of visual tokens. This means if a camera is static, the model spends nearly identical token budgets describing the background repeatedly.

As shown in, this per-frame interface forces a trade-off between coverage and detail.

You either sample sparsely to save context window space—missing fast-moving events—or you sample densely and exhaust the model's compute limits. Because token cost grows linearly with the number of sampled frames, this "coverage-detail dilemma" becomes a hard bottleneck for long-video reasoning. Existing efficiency methods typically try to prune tokens after they are already generated. However, they still rely on the inefficient premise that every frame deserves its own full visual identity.

How It Works

AdaCodec redesigns the visual interface. It treats video as a predictive code rather than a collection of snapshots. The architecture moves away from playback-oriented codecs (which optimize for human eyes) toward an MLLM-oriented interface (which optimizes for reasoning).

The mechanism operates in three main stages:

1. Adaptive GOP Construction: The system partitions the video into Groups of Pictures (GOPs). The first frame of a GOP is an I-frame (an intra-coded frame, encoded as a full RGB image). Subsequent frames are P-frames (predictive frames). Unlike fixed-interval schemes, AdaCodec uses a "predictive cost" ($p_{cost}$) threshold. If a frame cannot be accurately predicted from the previous one, the system triggers a new I-frame .

2. Motion and Residual Encoding: For each P-frame, the system does not send the whole image. Instead, it performs a local motion search to find how blocks of pixels moved from the reference frame. It then encodes two things: the motion vectors (the displacement of those blocks) and the residuals (the pixel-level difference between the prediction and the actual frame) .

These are packed into a compact 5-channel tensor. 3. Dual-Branch Tokenization: The model uses a specialized P-tokenizer to compress these motion-and-residual signals into a small number of "summary tokens." While an I-frame might yield 256 tokens, a P-frame is condensed to just 16 tokens. These compact P-tokens are interleaved with the full I-frame tokens and fed into the LLM.

Numbers

The authors report significant Pareto improvements across the efficiency-accuracy frontier. The most striking result is that AdaCodec achieves better accuracy than the Qwen3-VL-8B baseline while using only 1/7th of the visual tokens. Specifically, on long-video benchmarks like MLVU, AdaCodec at a 32k token budget outperforms the baseline at 224k tokens .

On general video-understanding benchmarks, the efficiency gains translate directly into latency reductions. The paper reports: * Token Reduction: An average of 84.6% reduction in visual tokens compared to the per-frame RGB baseline. This means you can process much longer videos within the same context window. * Time-to-First-Token (TTFT): Dropped from 9.26s to 1.62s. This drastically reduces the wait time before the model begins responding. * End-to-End Latency (E2EL): Reduced from 11.18s to 3.20s. This speeds up the entire generation process by roughly 3.5x.

Even after accounting for a "codec-build" overhead of 0.12s on a 16-core CPU, the system remains over 5x faster than the baseline [Table 4]. The memory footprint increase is negligible. It adds only about 1.9 GB (roughly 5.5%) of peak GPU memory due to the additional P-tokenizer branch.

What's Missing

There are several gaps a production engineer should note. First, the paper uses a fixed input resolution. It does not address how the system handles dynamic resolutions or high-aspect-ratio videos. Second, the P-frame token budget is currently uniform ($N_P = 16$). In complex scenes, 16 tokens might be insufficient. Conversely, in static scenes, it might be overkill. Finally, the authors do not evaluate the system in a streaming context. While the I/P structure is causal (meaning it only depends on past data), the actual throughput of a streaming implementation remains unproven.

Should You Prototype This

Yes, if you are hitting context window limits or latency walls in long-video applications. The ability to outperform a 224k-token baseline with only 32k tokens is a massive win for managing inference costs or KV cache pressure (the memory used to store previous model states). The architecture is modular. The P-tokenizer is a modified ViT (Vision Transformer) that can be adapted to different backbones. The two-stage training recipe is a clear path to implementation. Code is reportedly available; see the project page for the canonical link.

Figures from the paper