Stitch4D: Sparse Multi-Location 4D Urban Reconstruction via Spatio-Temporal Interpolation

Abstract: Dynamic urban environments are often captured by cameras placed at spatially separated locations with little or no view overlap. However, most existing 4D reconstruction methods assume densely overlapping views. When applied to such sparse observations, these methods fail to reconstruct intermediate regions and often introduce temporal artifacts. To address this practical yet underexplored sparse multi-location setting, we propose Stitch4D, a unified 4D reconstruction framework that explicitly compensates for missing spatial coverage in sparse observations. Stitch4D (i) synthesizes intermediate bridge views to densify spatial constraints and improve spatial coverage, and (ii) jointly optimizes real and synthesized observations within a unified coordinate frame under explicit inter-location consistency constraints. By restoring intermediate coverage before optimization, Stitch4D prevents geometric collapse and reconstructs coherent geometry and smooth scene dynamics even in sparsely observed environments. To evaluate this setting, we introduce Urban Sparse 4D (U-S4D), a CARLA-based benchmark designed to assess spatiotemporal alignment under sparse multi-location configurations. Experimental results on U-S4D show that Stitch4D surpasses representative 4D reconstruction baselines and achieves superior visual quality. These results indicate that recovering intermediate spatial coverage is essential for stable 4D reconstruction in sparse urban environments.

• The paper introduces a unified solution for urban 4D reconstruction under sparse camera placements, addressing limited overlap and maintaining geometric consistency.
• It employs a Multi-View Bridging Module to synthesize intermediate panoramic views, thereby improving spatial coverage using a camera-conditioned diffusion model.
• The Multi-Video Joint Optimization Module fuses real and synthesized views with seam-aware loss, achieving temporal coherence and up to 15.8 dB PSNR improvements.

Authoritative Analysis of "Stitch4D: Sparse Multi-Location 4D Urban Reconstruction via Spatio-Temporal Interpolation" (2604.07923)

Problem Context and Motivation

The paper introduces the Sparse Multi-Location 4D Reconstruction (SP4DR) problem, targeting urban scene reconstruction from panoramic videos captured at spatially distributed camera locations. Urban deployments frequently suffer from constrained camera placement, leading to wide-baseline, sparse, and minimally overlapping viewpoints, which compromises the effectiveness of existing 4D reconstruction approaches that assume dense view overlap. Maintaining spatiotemporal alignment, geometric consistency, and temporal coherence in such configurations is a significant challenge, especially under dynamic urban conditions.

Methodological Innovations

Stitch4D Framework

Stitch4D resolves the SP4DR problem using a unified 4D reconstruction strategy that explicitly compensates for missing spatial coverage and strengthens global geometric and temporal constraints. The architecture comprises two principal modules:

• Multi-View Bridging Module (MVBM): MVBM synthesizes intermediate panoramic views between spatially separated camera locations via a camera-conditioned multi-view diffusion model, enhancing effective spatial coverage and view overlap. By generating bridge views along trajectories between actual cameras, it densifies spatial constraints and mitigates geometric collapse in poorly observed regions.
• Multi-Video Joint Optimization Module (MVJOM): MVJOM jointly optimizes both real panoramic observations and the synthesized intermediate views within a shared 4D coordinate frame. This module enforces inter-location geometric consistency and temporal smoothness via seam-aware cross-location loss regularization, addressing spatial boundaries between camera viewpoints.

The backbone leverages Spacetime Gaussian Feature Splatting (SpacetimeGS) [27] for representing time-varying urban dynamics using temporally modulated Gaussian primitives, enabling efficient differentiable rendering and continuous scene modeling.

Interpolation and Loss Formulation

• View Synthesis: Each panoramic video is converted into 120 perspective views by deploying virtual cameras in numerous directions, increasing spatial granularity and capturing urban structure.
• Seam-Aware Cross-Location Loss: The loss function integrates photometric reconstruction, boundary-aware weighting, and cross-location gradient penalties to regularize spatial seams and enforce geometric and appearance continuity across camera boundaries.

Benchmarking and Experimental Results

Urban Sparse 4D (U-S4D) Benchmark

The authors introduce U-S4D, a CARLA-based dataset for systematic evaluation of SP4DR, providing synchronized panoramic videos, calibrated camera poses, dynamic traffic simulation, and support for free-viewpoint rendering. U-S4D includes diverse urban areas with varying density, visibility, and agent types to challenge spatial and temporal reconstruction capabilities.

Quantitative and Qualitative Findings

• Numerical Superiority: Stitch4D exhibits strong empirical performance, surpassing baselines (4DGS [43], SpacetimeGS [27], FreeTimeGS [40]) on PSNR, SSIM, and LPIPS across both full reconstruction and temporal split settings. Notably, for trajectory interpolation, Stitch4D achieves up to 15.81 dB PSNR (full) and 15.53 dB PSNR (temporal split), outpacing SpacetimeGS by 2–3 dB and demonstrating enhanced spatial generalization.
• Spatial and Temporal Consistency: Joint optimization, coupled with interpolation bridge views, maintains geometric stability and temporal coherence even in unobserved intermediate regions, as confirmed by qualitative analysis.
• Ablation Studies: Removing MVBM and MVJOM leads to substantial drops in reconstruction accuracy, affirming their contributions. Performance advantages remain robust when the framework is swapped onto alternative backbones (e.g., FreeTimeGS), indicating generalizability.
• Failure Cases: Despite advances, reconstructing dynamic objects under very sparse inter-location observations can result in temporally inconsistent boundaries, highlighting an unresolved limitation.

Implications and Future Developments

Practical Consequences

Stitch4D advances scalable, robust urban scene modeling for applications in autonomous driving, surveillance, and city-scale analytics, where infrastructure and privacy constraints necessitate sparse camera layouts. The capacity to interpolate intermediate spatial coverage and enforce cross-location temporal regularization enables stable and high-fidelity 4D reconstructions, facilitating downstream tasks such as city-level navigation, free-viewpoint rendering, and geography-aware querying.

Theoretical Insights

The approach demonstrates the criticality of explicit spatial interpolation and joint optimization in bridging sparse spatial gaps for dynamic scene modeling. The seam-aware cross-location regularization provides a template for addressing geometric inconsistencies across observation boundaries, potentially applicable to other domains within spatiotemporal modeling.

Prospective Directions in AI

Future work may extend Stitch4D to integrate dynamic camera observations (e.g., vehicle-mounted, wearable cameras) alongside static video streams. This would further increase spatial density, expand coverage, and improve model robustness, especially for long-term urban analytics and simulation. Moreover, generalizing the framework to real-world datasets with noisy calibration or non-rigid dynamics remains a pertinent avenue.

Conclusion

Stitch4D introduces a principled and modular solution to sparse multi-location 4D urban reconstruction, leveraging spatial interpolation and joint optimization to achieve geometrically and temporally consistent representations. Through rigorous benchmarking and analysis, the paper provides compelling evidence of the importance of intermediate spatial coverage and unified optimization in stable 4D scene modeling. The proposed framework is widely extensible, offering practical and theoretical advances for urban scene reconstruction, and setting new standards for handling sparse, distributed observations in complex dynamic environments.