ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

When teaching AI new tasks one by one, it often forgets how to format answers. It might switch from short answers to long descriptions mid-stream. This is a core challenge in Multimodal Continual Instruction Tuning (MCIT). In this setting, models must acquire new vision-language capabilities sequentially without losing old ones. Current state-of-the-art approaches try to solve this by isolating knowledge into specialized "experts." However, they often rely on a flawed assumption. They assume that if two tasks look similar, they should share the same parameters.

A new paper, ProtoAda, argues that this "format-blind" assignment causes failure. They find that even if two tasks involve the same visual concepts, they might require different response protocols. For example, one task might require predicting coordinates while another requires a short sentence. If you route them to the same expert, gradient interference (where updates for one task corrupt the weights for another) destroys the model's ability to follow instructions.

The Problem

Current MCIT methods often use sparse architectures like Mixture of LoRA Experts (MoE-LoRA). These route tasks to specific lightweight adapters based on image-text semantic similarity. The logic is that similar inputs should trigger similar weights. However, the authors show that semantic similarity is an insufficient proxy for task compatibility.

As shown in, the authors used controlled streams to test this.

In one stream, they kept visual content constant but varied the response protocol (the "format"). In another, they kept the format constant but varied the visual semantics. Both caused degradation. However, the performance drop was substantially larger when the format changed. This indicates that MLLM tuning doesn't just learn what is in an image. It also learns how to map instructions to specific output conventions. When a grounding task is routed to the same expert as a VQA task, the expert becomes confused. This leads to catastrophic forgetting of the required output structure.

How It Works

ProtoAda replaces semantic-only routing with a format-aware mechanism. The architecture relies on three main pillars:

1. Format-Aware Task Grouping: Instead of just using CLIP embeddings for vision and text, the authors construct a "task prototype." This includes response-format statistics. These include metrics like average response length, token entropy (a measure of how unpredictable tokens are), and template consistency. As visualized in, adding these statistics makes tasks with different response protocols much more distinguishable in the embedding space.

During training, they use these statistics. During inference, a lightweight predictor $f_{fmt}$ estimates the format from the input side. This allows for task-agnostic retrieval (finding the right expert without knowing the task ID beforehand).

1. Prototype-Conditioned Adaptive Growth: ProtoAda does not blindly add new LoRA modules to every layer. It selects layers based on two signals. First, it uses task sensitivity (calculated via the gradient energy of the task loss). Second, it uses historical group usage. This ensures that new capacity is only introduced where it is actually needed .

2. Geometry-Aware Consolidation: This is the mathematical core of the framework. When a task is finished, its LoRA update ($\Delta\Theta_t$) is split. Using orthogonal projection (a method to separate vectors into perpendicular components), the authors decouple the update. They create a "shared component" that aligns with existing group memory. They also create a "task-specific residual" that is orthogonal to it. The shared component is merged into the group adapter to promote parameter reuse. The residual is compressed via truncated SVD (Singular Value Decomposition, a method to reduce data dimensionality) and stored in a separate bank .

Numbers

The authors report improvements across two major benchmarks. On the TriGap benchmark, which features long, heterogeneous task sequences, ProtoAda achieves an average accuracy of 47.23%. This outperforms the previous state-of-the-art, SAME, which achieved 46.53% [Table 1].

The "stress test" on controlled format streams reveals the real utility. In a format-varied stream, standard fine-tuning sees its last accuracy plummet from 54.1% to 17.3%. Even sophisticated MoE-LoRA methods struggle. They leave a performance gap ($\Delta fmt$) of 4.7. ProtoAda reduces this gap to nearly zero (0.2). It maintains a last accuracy of 54.7% .

This means the model effectively resists forgetting its formatting rules even when new formats are introduced.

Regarding resources, ProtoAda avoids the linear parameter explosion typical of naive expansion. As shown in, cumulative parameter growth stays below the budget of typical MoE methods. This is achieved through periodic spectral promotion and residual pruning.

What's Missing

The paper focuses heavily on "response statistics" like length and entropy. It does not explore how the system handles extremely complex or nested formats. Examples might include structured JSON or multi-step reasoning chains. It is unclear if simple statistical proxies can capture those patterns.

The computational overhead of the "maintenance" phase is also not fully quantified. This includes the periodic reorganization of memory and SVD compression. In a production environment, we need to know if these steps create latency spikes.

Finally, the effectiveness of the format predictor ($f_{fmt}$) is critical. If the predictor fails during task-agnostic inference, the retrieval mechanism might pull from the wrong expert group. This could cause errors worse than those found in a standard dense model.

Should You Prototype This

Yes, if you are building specialized multimodal agents.

If your deployment involves a model that must learn a sequence of highly structured tasks, ProtoAda is a candidate. This applies if you transition from describing images to generating precise bounding box coordinates. The ability to decouple shared knowledge from task-specific residuals is a robust way to manage the stability-plasticity trade-off. However, if your tasks are primarily unstructured conversational queries, the complexity of managing a residual bank may not be worth the marginal gain. Start by implementing the format-aware prototype. It is the highest-leverage change in the architecture.

Figures from the paper