Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs

When building AI that sees and talks, adding more "eyes"—or vision encoders—doesn't always help. The current trend in Large Vision-Language Models (LVLMs) is to fuse heterogeneous visual streams. This aims to increase performance. However, we lack a principled way to know which combination actually works before we burn thousands of GPU hours. The paper Beyond Encoder Accumulation argues that we have been looking at this problem backward. Instead of just accumulating more encoders, the authors suggest picking one strong "anchor" and a second "adaptive complement" that provides non-redundant information.

The Problem

The status quo in multi-encoder LVLM design is effectively "brute-force accumulation." Researchers typically take a massive language backbone and concatenate features from various vision encoders. They use methods like channel concatenation (joining features along the depth dimension). Common encoders include CLIP, SAM, or ConvNeXt.

When engineers want to know which encoder is doing the heavy lifting, they traditionally use Inference-time Masking (IM). This involves taking a single, fully-trained checkpoint. They silence one encoder branch at a time to see how much performance drops. This is also known as the Conditional Utilization Rate (CUR).

As the authors demonstrate, this approach is fundamentally flawed for design decisions. Because IM evaluates a fixed checkpoint, it fails to account for how encoders interact during actual training. This leads to misleading rankings. For example, [Figure 1A] shows that under IM, EVA-02 appears to be the top encoder. Under the authors' preferred Training-time Removal (TR) protocol, ConvNeXt is actually the most critical. Relying on IM can lead you to select an encoder pool that is suboptimal once you actually start the training run.

How It Works

To solve this, the authors implement a rigorous "retrain-and-remove" protocol. They spent approximately 20,000 GPU hours retraining all 31 possible non-empty subsets of five common vision encoders. This allows them to measure the true contribution of an encoder. They observe how the model performs when that encoder was never part of the training loop.

The core of their analysis rests on a two-axis decomposition of encoder contribution:

1. Capacity: This is the score an encoder achieves when trained as a singleton (alone). It represents the encoder's raw power in isolation.
2. Necessity: This is the absolute drop in performance when an encoder is removed from the full five-encoder pool. It measures how much unique, irreplaceable information that encoder provides.

By plotting these on a plane, as seen in [Figure 3A], they categorize encoders into a taxonomy. This reveals that the highest-Capacity encoders are not necessarily the best to pair together. Instead, the optimal strategy is to pair a "Universal Core" (a high-Capacity, high-Necessity anchor like ConvNeXt) with an "Adaptive Complement" (an encoder like CLIP). CLIP might have moderate singleton capacity. However, its utility expands significantly during joint training.

The authors explain this synergy through "effective rank." This is a measure of the dimensionality of the features entering the projector (the MLP that maps visual features to the language space). They found that most encoders lose rank as the pool grows. This happens because the projector has a fixed output dimension, creating a "rank budget." The best complements are those whose rank actually expands under joint training [Figure 5B].

Numbers

The headline result is a massive efficiency gain. A well-chosen two-encoder model can recover almost all the performance of a five-encoder model. Specifically, the CLIP + ConvNeXt pair closes 97% of the gap between a CLIP-only baseline and the full five-encoder pool [Table 1, Figure 4B].

From a hardware and cost perspective, this is significant. Looking at the Pareto frontier in [Figure 4A], the best-at-$k$ models (where $k$ is the number of encoders) are often dominated by smaller subsets. The full five-encoder model reaches the highest absolute score. However, it requires 2274M parameters and has the lowest throughput (2.34 tokens/sec). In contrast, the ConvNeXt singleton is a Pareto-optimal choice for many workloads. It offers a high relative score (0.980) with much higher throughput (4.89 tokens/sec) and fewer parameters (846M). For most practitioners, the marginal gain of moving from two encoders to five is not worth the hit to inference latency.

What's Missing

While the study is methodologically rigorous, there are gaps. First, the findings are tied to "channel-concatenation" fusion architectures. If you use cross-attention aggregation, the "rank budget" logic might shift. Those architectures do not necessarily force all encoders through a single shared bottleneck.

Second, the study is limited to a fixed decoder (Vicuna-7B). The effects of decoder scaling on the saturation point are not explored. Finally, the "Anchor + Adaptive Complement" rule is a descriptive heuristic for this specific pool of five encoders. The authors admit it is not yet a proven universal law for any arbitrary set of vision encoders.

Should You Prototype This

Depends on your target workload.

If you are building a general-purpose LVLM and want to minimize inference latency and GPU memory, yes, prototype this selection logic. Stop blindly adding encoders. Instead, use the Capacity-Necessity framework to identify a high-rank anchor and a non-redundant complement. You can likely achieve $>95\%$ of the performance of a massive ensemble using only two encoders. This will drastically improve your throughput.

However, if your application is strictly "Vision-Centric," the paper notes that these tasks are the exception. Such tasks may still benefit from larger pools [Figure 4B]. In that specific case, consider sticking to the full ensemble or exploring larger $k$.

Figures from the paper