SurGe: Improved Surface Geometry in Point Maps

Current AI models that turn single photos into 3D maps often create bumpy or wavy surfaces. This is especially true for thin objects like chair legs. While these models estimate the overall layout of a room well, they struggle to maintain local shape integrity.

In monocular geometry estimation (recovering 3D structure from one image), recent feedforward models predict "point maps." These maps assign every pixel a specific 3D coordinate. However, a gap remains between estimating coarse scene structure and capturing fine-grained surface detail. Researchers previously relied on pointwise metrics (measuring distance between individual points) to evaluate success. These often fail to penalize the small, high-frequency oscillations that make a reconstructed surface look noisy in 3D space.

The SurGe paper attempts to close this gap. It treats local surface orientation as a primary target for both evaluation and training.

The Problem

The status quo in 3D reconstruction relies on global and local pointwise metrics, such as Absolute Relative Error (AbsRel). AbsRel measures how far a predicted 3D point is from its ground-truth position. While useful for placing objects correctly, these metrics are blind to local surface quality.

As the authors demonstrate in, a surface can undergo high-frequency perturbations (tiny, rapid wobbles).

These wobbles barely move the average position of the points. Because the displacement is small, the AbsRel score remains nearly identical to a smooth surface. This creates a blind spot. Models can achieve high scores while producing geometry that looks physically impossible .

For example, streetlamps may appear to bend or chair legs may oscillate. This is not just a 2D edge-detection issue. It is a failure to enforce local 3D coherence (the logical connection between neighboring points).

How It Works

The authors introduce three interconnected components: a new metric, a new loss function, and a new decoder architecture.

1. Point Map Normal Metric (MAEnormal): Instead of measuring distance, the authors evaluate local surface orientation. They compute the cross product (a mathematical operation used to find a perpendicular vector) of neighboring point differences to derive local normals. This makes the evaluation sensitive to the ripples and bumps that pointwise metrics miss .

2. Point Gradient Matching Loss ($L_{pgm}$): To drive the model toward smoothness, the authors implement a scale-invariant loss. Traditional gradient matching often operates on log-depth (using the logarithm of distance). This does not translate well to 3D vectors. $L_{pgm}$ instead matches depth-normalized 3D finite differences (the change in value between adjacent points). By normalizing the displacement by the $z$-coordinate (depth), the loss ensures supervision is scale-invariant for every neighboring pair. This allows the model to learn local structure without being dominated by global scale errors.

3. Neighborhood Attention Decoder (NAD): Standard convolutional decoders (which use fixed-size sliding filters) struggle with high-frequency signals. Conversely, pure Vision Transformer (ViT) decoders often suffer from patch-aligned artifacts due to fixed resolutions. The NAD breaks this dichotomy. It uses a progressive multi-resolution structure. It replaces convolutions with Neighborhood Attention (NA). NA allows the model to perform content-dependent local feature mixing. This means the model chooses which neighbors to attend to. It does this without the quadratic cost of global self-attention. This architecture, seen in, helps reconstruct thin structures like furniture legs .

Numbers

The authors report that SurGe achieves a new state of the art in local geometry across eight zero-shot benchmarks. Looking at Table 1, SurGe achieves the lowest $\text{AbsRel}_{\text{loc}}$ (local point map error) on every tested dataset. Regarding surface smoothness, SurGe consistently lowers the Mean Angular Error ($\text{MAEnormal}$) compared to predecessors like MoGe-2 and InfiniDepth [Table 2].

These improvements in local detail do not degrade global accuracy. The authors report that SurGe achieves the best average rank for global $\text{AbsRel}$ across all eight benchmarks. This average rank is 1.62 [Table 3].

There is a clear trade-off regarding performance. According to the runtime analysis in [Table B], the NAD decoder is more computationally expensive than a standard convolutional stack. When paired with a DINOv2-Large backbone, the NAD decoder increases inference latency (the time taken to process one image) by approximately 1.46$\times$. Memory usage also increases modestly, rising from 1.87 GiB to 2.02 GiB.

What's Missing

While the results are compelling, a few areas remain unaddressed:

• Hardware Efficiency: The paper calls the NAD an "accuracy-oriented" design. However, it does not explore ways to mitigate the latency penalty. For real-time applications like AR/VR, this overhead is a significant concern.
• Quantization Robustness: The authors removed standard LayerNorm (a technique to stabilize neural network training) in the NAD blocks. They used QK normalization instead. While this improves accuracy, it is unclear how this "norm-free" architecture behaves under quantization (reducing numerical precision to save memory). This is vital for edge hardware.
• Scaling Limits: The training protocol uses a fixed budget of encoder tokens. The paper does not show how the Neighborhood Attention mechanism scales for ultra-high-resolution dense reconstruction.

Should You Prototype This

Yes, if your application demands geometric fidelity. If you are building systems where wavy surfaces or bent thin structures break the user experience, SurGe is a significant upgrade. The combination of the $L_{pgm}$ loss and the NAD decoder directly targets common failures in current models.

However, if you have strict millisecond-latency requirements on mobile hardware, proceed with caution. The 1.46$\times$ latency hit is a real cost. You should likely prototype the $L_{pgm}$ loss on your existing convolutional architecture first. This lets you test for surface smoothness gains without the full architectural overhead of Neighborhood Attention.

Code is reportedly available; see the paper for the canonical link at https://vision.rwth-aachen.de/surge.

Figures from the paper