UniT: Unified Geometry Learning with Group Autoregressive Transformer

Abstract: Recent feed-forward models have significantly advanced geometry perception for inferring dense 3D structure from sensor observations. However, its essential capabilities remain fragmented across multiple incompatible paradigms, including online perception, offline reconstruction, multi-modal integration, long-horizon scalability, and metric-scale estimation. We present UniT, a unified model built upon a novel Group Autoregressive Transformer, which reformulates these seemingly disparate capabilities within a single framework. The key idea is to treat groups of sensor observations as the basic autoregressive units and predict the corresponding point maps in an anchor-free and scale-adaptive manner. More specifically, diverse view configurations in both online and offline settings are naturally unified within a single group autoregression process. By varying the group size, online mode operates over multiple autoregressive steps with single-frame groups, whereas offline mode aggregates a multi-frame group in a single forward pass. Meanwhile, a queue-style KV caching mechanism ensures bounded autoregressive memory over long horizons. This is enabled by reducing long-range dependencies on early frames through anchor-free relational modeling, thereby allowing outdated memory to be discarded on the fly. To improve metric-scale generalization across scenes, a scale-adaptive geometry loss is further introduced within this framework. It couples relative geometric constraints with a partial absolute scale term, implicitly regularizing global scale and inducing a progressive transition from scale-invariant geometry to metric-scale solutions. Together with a dedicated modal attention module for integrating auxiliary modalities, UniT achieves state-of-the-art performance in unified geometry perception, as validated on ten benchmarks spanning seven representative tasks.

• The paper introduces a unified framework using a group autoregressive transformer to jointly handle diverse 3D geometric tasks with bounded memory complexity.
• It incorporates a queue-style key-value caching mechanism and scale-adaptive loss, achieving state-of-the-art metrics for multi-view reconstruction and camera pose estimation.
• Empirical evaluations show robust performance on long-horizon video, multi-modal fusion, and efficient metric-scale reasoning for real-time 3D perception.

Unified Geometry Learning with UNIT: A Group Autoregressive Transformer Approach

Introduction

This paper introduces UNIT, a unified feed-forward framework for geometry perception that leverages a Group Autoregressive Transformer architecture to jointly encompass a broad spectrum of 3D geometric inference tasks—including multi-view and monocular depth estimation, camera pose estimation, multi-modal 3D reconstruction, depth completion, and long-horizon video perception. The approach addresses limitations in prior work, where essential geometry perception capabilities remain segregated across incompatible paradigms and models. UNIT unifies online and offline inference, supports flexible sensor modality integration, delivers metric-scale estimations, and guarantees bounded memory complexity across long input sequences (2605.21131).

Methodological Contributions

Group Autoregressive Transformer

At the core of UNIT is a group-wise autoregressive transformer that treats a group of sensor observations as the fundamental autoregressive unit, instead of an individual frame. This enables:

• Flexible View Configuration: The group size $G$ is adjustable from $1$ (frame-wise online inference) up to $N$ (global offline inference), enabling seamless transition between streaming and batch processing paradigms.
• Bidirectional Intra-Group and Causal Inter-Group Attention: Within a group, bidirectional attention efficiently captures spatial and viewpoint correlations, while across groups, causal masking enforces proper autoregressive dependencies.
• Arbitrary Sensor Modalities: Auxiliary cues (depth, intrinsics, extrinsics) are flexibly incorporated via a dedicated Modal Attention module that aligns and fuses spatially corresponding features.

Queue-Style Key-Value Caching

To mitigate unbounded memory growth typical in autoregressive architectures processing lengthy sequences, UNIT introduces a queue-style key-value (KV) caching mechanism:

• Anchor-Free Relational Modeling: By enforcing all geometric relationships in relative (anchor-free) coordinates, long-range dependencies on initial frames are eliminated. This allows discarding tail frames from the KV cache, achieving strictly bounded memory complexity $O(Q)$, decoupled from sequence length.
• Compatibility with Compression: The design is orthogonal to token compression or merging strategies, facilitating integration with further efficiency optimizations if desired.

Scale-Adaptive Geometry Loss

Metric-scale generalization remains notably challenging due to scale ambiguities and weak cross-scene transferability. UNIT employs a scale-adaptive geometry loss, which:

• Curriculum Transition: Regularizes model training by coupling scale-invariant (relative) geometric constraints with progressively introduced absolute (metric-scale) signals.
• Implicit Regularization: Avoids explicit global scale regression; instead, the loss guides the model to converge from relative to absolute solutions, stabilizing training and improving metric accuracy across scenes.

Training and Implementation

UNIT is trained on a hybrid aggregation of 21 public metric-scale datasets (real and synthetic, indoor/outdoor, and various sensor characteristics), enabling broad cross-domain generalization. Multi-modal augmentation is simulated extensively during training, and multiple sparse depth patterns are introduced to ensure robustness to sensor heterogeneity and sparsity.

Empirical Evaluation

Extensive benchmarks across ten datasets and seven fundamental geometry tasks demonstrate the effectiveness of UNIT:

• Multi-view Reconstruction and Camera Pose Estimation: Achieves SOTA or second-best accuracy under both scale-invariant and metric-scale settings. Notably, in metric-scale scenarios (which are more demanding), UNIT consistently outperforms all baselines—demonstrating robust 3D metric reasoning without post hoc scale alignment.
• Video and Monocular Depth Estimation: Shows strong performance, especially in metric-scale error metrics, and narrows the performance gap between monocular and multi-view settings compared to previous works.
• Long-Horizon Scalability: Outperforms baselines by supporting input sequences several-fold longer before hitting hardware limits, courtesy of bounded KV cache. Achieves this without sacrificing prediction accuracy.
• Multi-Modal Fusion and Depth Completion: Demonstrates superior adaptability to various sensor combinations (RGB, LiDAR, intrinsics, depth maps) and strong robustness under diverse depth sparsity regimes.
• Efficiency: Delivers higher accuracy at lower or comparable computational overheads (parameter count, FPS, GPU memory) versus established baselines.

Key numerical results (based on benchmark tables and discussions):

• Multi-view metric-scale Accuracies: Outperforming MapAnything, DepthAnything3, and CUT3R on 7-Scenes/DTU/NRGBD
• Camera Pose ATE: Best-in-class absolute errors—e.g., UNIT achieves $0.047$ m (7-Scenes) versus $0.070$ (MapAnything)
• Memory/Bandwidth: Maintains FPS at $1.18$ and peak memory $6.7$ GiB in online settings, outperforming or matching established models with superior scalability

Theoretical and Practical Implications

The group autoregressive paradigm resolves the longstanding fragmentation between online/offline and uni-/multi-modal 3D perception models, offering a general solution platform for a wide range of application domains. The anchor-free, group-wise, relational modeling approach eliminates reference bias and enhances incremental learning capabilities. Scale-adaptive training reduces convergence difficulties and improves transfer to novel metric-scale domains, suggesting a promising direction for future 3D foundation models.

For robotics, AR/VR, and autonomous systems, this enables real-time, long-term environment perception and mapping from streaming, heterogeneous sensors—enabling deployment in dynamic, unstructured, and resource-constrained settings.

UNIT’s unified design provides a blueprint for future geometric perception models, emphasizing:

• Generalization over input configuration variability (number, type, and nature of sensor streams)
• Robust scaling to challenging scenarios (lengthy video, hybrid sensors, diverse environments)
• Modular integration with advanced attention, memory, and loss modeling strategies

Prospective Developments

Anticipated advancements include:

• Extension to more complex sensor suites (event/fusion, semantic priors, language-grounded inputs)
• Integration with large-scale, nonstatic environments; coupling with downstream decision and planning modules
• Joint training with visual-language or visual-action models, supporting holistic world modeling for embodied AI

Conclusion

UNIT represents a major advancement in unifying fragmented geometry perception pipelines. Through a group autoregressive transformer, anchor-free modeling, and scale-adaptive objectives, it delivers SOTA results in multi-modal, long-horizon, and metric-scale geometric reasoning. Its systematic architectural and algorithmic design suggests a robust foundation for next-generation 3D perception in both academic research and practical deployment (2605.21131).