VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

Can We Shrink Visual Tokens Without Breaking the Model?

Vision-language models (VLMs) process massive amounts of visual data. This comes at a steep price. High inference latency and massive memory overhead occur from storing huge sequences of visual tokens. Researchers try to compress these tokens to summarize the image. This prevents the model from working too hard. Most attempts use simple heuristics. These fall apart when you compress too much. Other methods add heavy external modules. These require expensive retraining. The core question remains: can we leverage the intelligence already inside a pretrained VLM to compress its own visual input efficiently?

The search for an intrinsic compressor

The fundamental tension in VLM deployment is the trade-off between visual fidelity and computational cost. High-resolution images provide the detail needed for tasks like OCR (Optical Character Recognition) or fine-grained reasoning. However, they explode the number of visual tokens the LLM backbone must process. This creates a bottleneck in the self-attention mechanism (the process where tokens weigh the importance of other tokens). It also consumes enormous amounts of KV cache (the memory used to store previous keys and values during autoregressive generation).

The authors of VisCo investigate whether we can move away from "external" compression. External compression uses a separate, often heavy module to shrink tokens. Instead, they aim for "intrinsic" compression. The goal is to map dense visual information into a tiny, compact set of tokens. They use the VLM's own internal logic. This minimizes the need to fundamentally rewrite the model's weights.

Cracks in the pruning and merging paradigm

Until now, the field has largely split into two camps. Both hit walls under pressure. On one side are training-free methods like FastV or SparseVLM. These rely on attention scores or similarity metrics to prune (delete) or merge tokens. As shown in, these methods attempt to select only the "important" parts of the vision encoder's output.

However, as the compression ratio gets more aggressive, these methods suffer from "attention shift." This causes the model to lose global context or hallucinate. illustrates this failure vividly.

When restricted to just 9 tokens, SparseVLM loses the global scene. It also becomes overly dependent on the specific question asked.

On the other side are training-based methods that use external modules. While these are more robust, they often force the VLM backbone to adapt. This requires the model to learn a new way of seeing. Such methods incur massive retraining costs. They also risk stripping away the model's original multimodal priors (the pre-existing knowledge the model gained during initial training). Essentially, the field has been choosing between "fast but fragile" and "robust but expensive."

Building an asymmetric autoencoder

The authors propose VisCo. It treats compression as an asymmetric autoencoder task. An autoencoder is a network that learns to compress data and then reconstruct it. Instead of adding a new module, they introduce a small set of learnable "memory tokens" ($X_m$). These sit alongside the original visual tokens ($X_v$) during the encoding stage. Because the model uses a causal masking mechanism (a rule where tokens only look at previous tokens), these memory tokens can naturally attend to all preceding visual tokens. This allows them to aggregate information using the VLM's native attention patterns.

The clever part is how they handle information transfer. VisCo uses a "hierarchical KV-pass" mechanism. Rather than passing a single compressed vector, it passes data through multiple levels. As seen in, the encoder collects the Key (K) and Value (V) matrices produced by the memory tokens at every layer of the transformer.

These are stored in a memory bank. They are then directly injected into the decoder's KV cache. This ensures the decoder receives a multi-granularity representation. Shallow layers provide texture and local patterns. Deeper layers provide abstract semantics. To keep training lightweight, they use LoRA (Low-Rank Adaptation, a technique to fine-tune models by updating only a small subset of weights). This leaves the main backbone frozen during decoding.

Stability in the face of extreme compression

The results suggest this hierarchical approach is significantly more resilient than prior work. Testing on LLaVA-1.5-7B, the authors report high stability. At a 32-token budget, VisCo maintains 91.8% of the original model's performance. This outperforms established baselines like VisPruner. Even more striking is the performance under "extreme" compression. When forced to represent an entire image with just a single token, VisCo retains 85.3% of its original performance. In contrast, earlier methods like PruMerge+ drop to roughly 35.4% [Table 1]. This means VisCo stays much closer to the original model's accuracy even when the data is heavily squeezed.

The authors also found that these memory tokens are not just summaries. They are complementary. In an ablation study called VisCo+, they showed that combining original visual tokens with learned memory tokens actually improved the base model's performance [Table 4]. This implies memory tokens capture unique perspectives. They see things the standard tokens might overlook. Furthermore, shows that early memory tokens focus on primary objects.

Later tokens pick up background details. This explains why the hierarchical structure is vital for maintaining a complete scene description.

Implications for the prod path

If these findings hold, the implications for deploying VLMs in real-time environments are significant.

First, VisCo is a winner for multi-turn dialogue. The compression happens once at the encoding stage. The resulting KV cache is then reusable. This drastically reduces the cost of subsequent questions. As shown in, VisCo might have a higher initial latency due to the encoding step. However, it reaches a "break-even" point against methods like FastV after only three turns of conversation. After this point, VisCo becomes faster on average.

Second, this provides a blueprint for "lossless-feeling" compression. The ability to maintain high accuracy with a single token is remarkable. It suggests we can significantly slash KV cache memory requirements. We could potentially reduce them by orders of magnitude. This avoids the catastrophic performance collapses seen in previous pruning methods.

One immediate question for the community concerns quantization (the process of reducing the precision of numbers to save memory). How does this hierarchical KV-injection behave under heavy quantization, such as 4-bit or 2-bit? If the precision of the memory bank is sensitive to quantization, the memory savings might be offset. The need for higher-precision cache storage could negate the benefits. A prototype testing VisCo with FP8 or INT8 KV caches would be the logical next step.

Figures from the paper