HL-OutPaint: Coarse-to-Fine Video Outpainting for High-Resolution Long-Range Videos

When you want to expand a video's borders—for instance, turning a square social media clip into a widescreen cinematic shot—you face a massive consistency problem. Most generative models can fill in a single frame beautifully. However, as soon as you move to a long sequence, the background starts to flicker. Objects morph, and the sense of physical space collapses.

Current state-of-the-art methods generally pick a side. They either handle large spatial expansion through patch-based tiling (dividing the image into small squares), which destroys global structure. Or they handle long sequences by subsampling frames, which destroys local motion. This leaves a gap for real-world video production. You need both high resolution and long-range temporal stability. HL-OutPaint attempts to bridge this gap. It uses a coarse-to-fine strategy to separate global structural planning from fine-grained detail synthesis.

The Problem

The status quo in video outpainting is fundamentally fractured. Methods like Infinite-Canvas manage large spatial extensions by generating the video in local patches. While this allows for massive canvas sizes, the model lacks a holistic view of the scene. This leads to repetitive structures and a lack of global coherence. On the other end, models like M3DDM attempt to solve the long-sequence problem using sparse keyframes as guidance. However, when the input video contains rapid motion, the temporal gap between these keyframes becomes too large. This results in jarring temporal inconsistencies.

As seen in, competing methods like MOTIA and M3DDM often produce severe visual artifacts under large spatial extrapolations.

Even more advanced models like VACE struggle with long-term coherence. illustrates this clearly.

In a scenario where an object temporarily occludes a background region, these models often fail to reconstruct the background consistently. They essentially "forget" what the scene looked like once the occlusion clears.

How It Works

HL-OutPaint solves this by decoupling structural planning from detail synthesis .

1. Global Coarse Guidance (GCG) Construction: The model first builds a low-resolution "skeleton" of the entire video. This GCG captures the dominant motion and global structure. To prevent the loss of detail from aggressive downsampling, the authors introduce a global-local frame swapping mechanism. During the diffusion denoising process (the iterative removal of noise to reveal an image), the model periodically swaps latent representations between sparse global keyframes and their surrounding local temporal windows. This allows the global keyframes to "inherit" fine-grained motion cues from the local windows. This ensures the skeleton is both structurally stable and locally accurate .

For extremely long videos, they use a multi-scale approach. They iteratively insert midpoints between keyframes until the temporal density is sufficient.

1. GCG-Guided High-Resolution Outpainting: Once the GCG is established, the model moves to the second stage. It first performs temporal completion at a reduced resolution to fill the gaps between guidance frames. Finally, it performs spatial refinement. Using an SDEdit-style approach (injecting noise and then denoising to refine texture), the model restores high-frequency details. Because the full resolution is still too large for a single GPU pass, they use a spatio-temporal tiling strategy. They process overlapping chunks and blend them to maintain smoothness.

Numbers

The authors report significant gains in both quality and efficiency. In quantitative evaluations on the DAVIS dataset, HL-OutPaint outperformed baselines like VACE and Infinite-Canvas across nearly all metrics. This includes PSNR (signal-to-noise ratio), SSIM (structural similarity), and FVD (Fréchet Video Distance, a measure of video distribution similarity) [Table 1]. On the Long-Video dataset, the authors found that their method achieved an FVD of 133.2. This is a major improvement over Infinite-Canvas, which scored 275.0.

From a production standpoint, the inference efficiency is striking. Testing on a 500-frame, 720p video using an A100-80GB GPU, the authors report an inference time of 105 minutes. This is faster than VACE, which requires 143 minutes. It is also much faster than M3DDM, which takes 780 minutes [Table 5]. This suggests the hierarchical approach is more efficient. It plans with a low-res skeleton before committing to high-res pixels.

What's Missing

The paper is honest about its breaking points regarding extreme scale. If you attempt a massive expansion, such as 512x512 to 5760x5760, the system begins to fail. As shown in, the heavy downsampling required for the GCG causes a permanent loss of high-frequency information. While the original central regions can recover detail via conditioning, the outpainted regions end up looking blurry. The instructions in the GCG are simply too low-resolution to capture fine textures.

Additionally, the method is not suitable for real-time applications. It generates all frames jointly in a single inference process. This makes it an "offline" tool rather than a live filter. There is also no discussion on how the model handles variable frame rates or non-linear temporal shifts.

Should You Prototype This

If your roadmap involves building tools for professional video reframing, the answer is yes. The coarse-to-fine architecture is a proven pattern for scaling generative models. The global-local frame swapping is a clever way to fix the "sparse keyframe" problem.

Do not expect to ship this as a real-time feature. The joint generation process is too heavy. However, as an offline augmentation tool, it is highly viable. Code is reportedly available at https://koyy001.github.io/Publications/hl-outpaint. If you have a cluster of A100s or H100s, you can likely replicate their results quickly using their LoRA-based fine-tuning recipe.

Figures from the paper