Selective Synergistic Learning for Video Object-Centric Learning

SSync: Solving the Conflict Between Sharp Edges and Stable Centers

Current AI models that try to identify objects in videos often struggle. They try to force every part of an image to match perfectly. This process spreads errors across the whole system. SSync only forces parts to match where they are most reliable. It uses sharp edges from one part to fix blurry edges in another. It also uses stable centers to clean up noisy areas.

Resolving the Encoder-Decoder Discrepancy

Modern computer vision aims for "object-centric learning." This means teaching machines to decompose videos into discrete entities like "a car" or "a person." Models group features into "slots." These slots act as individual containers for object identities.

A structural tug-of-war exists between the encoder and the decoder. The encoder produces "attention maps." These maps assign data to specific slots. Because encoders use high-capacity backbones, these maps are spatially sharp. They pinpoint exactly where an object begins. However, they are prone to "noise" (errant, incorrect assignments). The decoder takes these slots to reconstruct the video. It produces "object maps." These maps are temporally coherent (they do not flicker wildly). However, they are spatially blurry. They often fail to capture crisp object edges.

Previous methods, such as SRL, forced these two maps to agree everywhere. The authors argue this "indiscriminate alignment" is a mistake. By forcing every pixel to match, the model propagates weaknesses. It teaches the encoder to be blurry and the decoder to be noisy.

The Mechanics of Selective Synergy

The authors propose Selective Synergistic Learning (SSSync). Its core logic is a principle of selective alignment. Mutual supervision should only occur where each branch provides reliable cues. SSync identifies which module is the "expert" in a specific region.

As shown in, the framework exploits architectural complementarities.

The encoder is the expert on boundaries. SSync uses the encoder's sharp attention maps for "boundary refinement" of the decoder. Conversely, the decoder is the expert on the stable interior of an object. SSync uses the decoder's coherent object maps for "interior denoising" of the encoder.

This uses a specialized pseudo-labeling (creating temporary target labels) process. The model calculates local consistency to find reliable regions: 1. Boundary Regions ($P_{rbd}$): These are identified from the encoder. They appear when a patch shows high local disagreement (sudden changes in slot assignment). 2. Interior Regions ($P_{nbd}$): These are identified from the decoder. They appear when a patch shows high local consistency.

The model then performs asymmetric cross-distillation (transferring knowledge between different modules). It uses encoder labels to supervise decoder boundaries. It uses decoder labels to supervise encoder interiors. To prevent "over-fragmentation" (splitting one object into many redundant slots), the authors introduce "transitive pseudo-label merging." This mechanism builds a graph of overlapping slots. It then consolidates them into unified identities .

Scaling Without Quadratic Costs

SSSync offers a significant shift in computational efficiency. Previous dense alignment strategies required comparing every spatio-temporal patch to every other patch. This resulted in "quadratic complexity" $O((T \cdot H \cdot W)^2)$. This means doubling the video length or resolution quadruples the work. Such scaling makes high-resolution video processing nearly impossible on standard hardware.

The authors report that SSync uses a linear-complexity approach, $O(T \cdot H \cdot W)$. It focuses only on selected reliable patches. This avoids the heavy math of global comparisons. For a configuration with $T=4$ and a batch size of 32, SSync reduced VRAM usage (the memory used by the GPU) by approximately 60% compared to SRL. SSync remained feasible on hardware where SRL triggered "Out-of-Memory" errors [Table 5].

Improving Decomposition Quality

The practical impact of this approach is visible in object "cut-outs." The authors demonstrate that SSync achieves state-of-the-art results on benchmarks like MOVi-C, MOVi-E, and YouTube-VIS.

Qualitative comparisons in and show the difference.

Older methods like SlotContrast produce noisy, speckled masks. SRL produces smoother but ambiguous shapes. SSync produces masks with sharp edges and stable interiors. The authors quantify this using "outside leakage" (how much a predicted mask spills into the background). SSync reduces this leakage by 16.5% compared to prior methods. This effectively "deblurs" the decoder's output.

The model is also a "plug-and-play" module. It does not require extra layers or complex projectors. It can be integrated into existing architectures to improve performance immediately.

Limits of the Framework

SSync is not a universal solution. The authors identify two primary failure modes. First, in early video frames, the model may suffer from "under-fragmentation." The model relies on motion cues to distinguish objects. If two objects enter a scene from the same direction, the model might lump them into one slot. This persists until they physically move apart.

Second, the model struggles with "part-level over-fragmentation" in very large objects. This happens when objects have high visual variety. For example, a cargo truck has a cabin and a container. These parts look very different. The model might assign them to different slots. Even with transitive merging, this remains a challenge. This suggests a need for stronger "part-to-whole" priors (rules that help the model realize different parts belong to one object).

Figures from the paper