ZipSplat: Fewer Gaussians, Better Splats

Can We Build 3D Worlds Without Counting Every Pixel?

Current tools for 3D reconstruction often create a new digital point for every single pixel in a photograph. While this sounds precise, it is incredibly wasteful. A flat white wall and a complex, textured sculpture end up consuming the same amount of computational memory. They do this simply because they occupy the same number of pixels.

As we move toward instantaneous, "feed-forward" 3D reconstruction—where a single pass through a neural network generates a 3D scene from sparse images—engineers face an efficiency crisis. The industry seeks a way to decouple the complexity of a 3D scene from the resolution of the input camera.

A new study from researchers at ETH Zürich and Microsoft introduces ZipSplat. This method breaks the pixel-to-geometry bond. Instead of predicting a 3D element for every pixel, the authors report that ZipSplat uses smart grouping. This allows it to place elements only where they are actually needed.

Breaking the pixel-geometry bond

The central question driving this research is whether a 3D reconstruction model can allocate its "budget" based on scene complexity. The budget refers to the number of mathematical primitives used to represent a scene.

In the context of 3D Gaussian Splatting (3DGS), a technique that represents scenes as a collection of tiny, translucent, anisotropic (non-spherical) blobs called Gaussians, the budget is the total number of these Gaussians. Currently, most feed-forward models tie these Gaussians to the input pixel grid. This creates a fundamental inefficiency. The model's memory usage scales linearly with the number of pixels. Higher-resolution photos always lead to massive, redundant files. This happens regardless of whether the scene is a simple hallway or a cluttered room.

The inefficiency of ray-anchored models

Until now, the prevailing approach has been to anchor every predicted Gaussian to a specific viewing ray. Imagine mapping a room by placing a sticker on every point where a laser pointer hits a surface. If you shine lasers from ten different angles, you end up with ten stickers on the same spot.

The authors note that this "pixel-aligned" formulation leads to three distinct failures. First, it ignores scene content. It treats flat surfaces and intricate textures with equal density. Second, overlapping camera views produce duplicate Gaussians for the same physical surface. This causes memory to grow unnecessarily as more images are added. Finally, because every prediction is tethered to a specific line of sight, these models struggle to "fill in the blanks." They cannot easily cover areas that are occluded or hidden from the camera's view. As shown in [Figure 3a], traditional methods like YoNoSplat distribute Gaussian centers uniformly across the views. This essentially mirrors the 2D grid in 3D space.

From pixel grids to scene tokens

To solve this, the researchers investigated a "token-based" architecture. Instead of treating the input as a grid of pixels, they treat it as a collection of "visual tokens." These are compact, high-level descriptors of local image features.

The core of the ZipSplat investigation involves a two-step compression process. First, the model uses k-means clustering (a method of grouping similar data points together) to compress these dense visual tokens. This creates a smaller set of "scene tokens." This acts like a summarization step. Redundant observations of the same surface from different angles are merged into a single representative description. Second, a lightweight MLP (a basic neural network component) decodes these tokens. It produces groups of Gaussians with entirely unconstrained 3D positions.

Because the Gaussians are no longer forced to sit on a camera ray, they can "float" freely in 3D space. To prevent these Gaussians from drifting into empty space during training, the authors employ a "one-directional Chamfer loss." This is a geometric supervisor that pulls predicted Gaussians toward actual surfaces .

Crucially, it does not force the model to cover every single ground-truth point. This helps preserve adaptive density.

High fidelity with a 33x smaller footprint

The findings suggest that decoupling geometry from pixels actually improves quality. The authors report that ZipSplat achieves state-of-the-art results on the DL3DV and RealEstate10K benchmarks. It does so while using significantly fewer resources.

On the DL3DV dataset, the study finds that ZipSplat can match or exceed the quality of existing methods. It does this while using up to 33 times fewer Gaussians than pixel-aligned baselines like YoNoSplat .

Specifically, the researchers report that ZipSplat reaches a PSNR (Peak Signal-to-Noise Ratio, a metric for image reconstruction quality) of 25.24 dB with only 62K Gaussians. In comparison, YoNoSplat requires 393K Gaussians to achieve a lower 24.10 dB. A higher PSNR indicates a more accurate reconstruction.

Perhaps most interestingly, the authors demonstrate that ZipSplat provides a "continuous quality-efficiency knob." Because the clustering happens during inference (the stage where the model is actually used), a user can adjust the compression ratio. This allows them to choose between a high-fidelity, heavy model or a lightweight, fast one . Even at high compression, the model maintains sharp renders. It effectively concentrates its limited budget on the geometrically complex parts of the scene .

Implications for real-time 3D engines

If these results generalize to larger environments, the implications for the 3D web and mobile AR are significant. By drastically reducing the number of Gaussians required, ZipSplat could enable high-quality 3D environments to be streamed over low-bandwidth connections. It could also allow smooth rendering on hardware with limited memory.

The transition from "ray-anchored" to "free 3D placement" marks a conceptual shift. It suggests that the future of 3D reconstruction lies in models that reason about 3D volumes directly. This is different from merely "uplifting" 2D image data into a third dimension. This approach may also provide a more stable foundation for reconstructing dynamic scenes. There, temporal redundancy could be exploited similarly to spatial redundancy.

While the paper demonstrates strong zero-shot generalization to datasets like Mip-NeRF360 and ScanNet++, it does not explore how the model handles extreme lighting changes or highly reflective surfaces. A logical next step for researchers would be to test if this token-based compression can be extended to 4D. Such a system would compress redundant observations across both space and time.

Figures from the paper