WildPose: A Unified Framework for Robust Pose Estimation in the Wild

Abstract: Estimating camera pose in dynamic environments is a critical challenge, as most visual SLAM and SfM methods assume static scenes. While recent dynamic-aware methods exist, they are often not unified: semantic-based approaches are brittle, per-sequence optimization methods fail on short sequences, and other learned models may degrade on static-only scenes. We present WildPose, a unified monocular pose estimation framework that is robust in dynamic environments while maintaining state-of-the-art performance on static and low-ego-motion datasets. Our key insight is to connect two powerful paradigms in modern 3D vision: the rich perceptual frontend of feedforward models and the end-to-end optimization of differentiable bundle adjustment (BA). We achieve this with a 3D-aware update operator built on a frozen, pre-trained MASt3R feature backbone, together with a high-capacity motion mask detector that uses multi-level 3D-aware features from the same backbone. Extensive experiments show WildPose consistently outperforms prior methods across dynamic (Wild-SLAM, Bonn), static (TUM, 7-Scenes), and low-ego-motion (Sintel) benchmarks.

• The paper introduces a unified framework that integrates pretrained 3D-aware features with differentiable bundle adjustment for robust pose estimation in both static and dynamic scenes.
• The system leverages a high-capacity edge-specific motion mask detector to accurately handle transient motions and mitigate outlier influence.
• Experimental results demonstrate state-of-the-art performance across challenging benchmarks, improving camera trajectory estimation in diverse environments.

WildPose: A Unified Framework for Robust Pose Estimation in the Wild

Introduction and Problem Scope

Robust pose estimation under dynamic scene conditions remains an unsolved challenge in monocular SLAM and SfM. Classical methods and mainstream deep architectures typically assume static environments, leading to severe performance degradation if dynamic entities are present. Further, many dynamic-scene methods lack generalization: semantic-based systems are limited by incomplete priors and domain mismatch, while optimization-focused learned detectors often fail on short or static-only trajectories.

WildPose introduces a principled unification of feed-forward, perceptual-rich encoders, and end-to-end differentiable bundle adjustment (BA), targeting generalizable pose estimation in both static and dynamic settings. The approach augments the robust optimization backbone of DROID-SLAM with 3D-aware, pretrained feature extractors and augments the BA layer with a high-capacity edge-specific motion mask detector, addressing crucial failure modes in existing pipelines.

Figure 1: Overall system architecture of WildPose, illustrating robust camera trajectory estimation using features from a frozen MASt3R encoder and a high-capacity motion mask detector.

Methodology

Differentiable Bundle Adjustment Pipeline

WildPose fundamentally extends the DROID-SLAM differentiable BA paradigm. For a monocular input sequence, a learned update operator refines intermediate pose and geometry estimates by iteratively aligning predicted optical flow (using current pose and disparity estimates) with observed image evidence, leveraging Gauss-Newton optimization and Schur complement acceleration. Crucially, the weighting matrix for the BA objective incorporates confidences and motion masks, robustifying optimization against outliers induced by non-rigid scene content.

Feed-Forward 3D-Aware Feature Integration

WildPose replaces the simple, task-trained CNN frontend of DROID-SLAM with the MASt3R transformer encoder, pretrained on large-scale multi-view and correspondence objectives that yield strong 3D geometry priors. Features from the last ViT layer are adapted through lightweight convolutions to provide both spatially localized and global feature maps, forming inputs for the recurrent ConvGRU update operator.

Edge-Based Motion Mask Detector

The system includes a high-capacity, frame-pair-dependent motion mask detector, leveraging multiscale tokens from the MASt3R transformer. Unlike recurrent-state-based detectors (e.g., MegaSaM), this module fuses multi-level transformer features through a DPT-style fusion block and outputs a residual-weighting mask per edge in the pose graph. This is critical for fine-grained identification of transient and non-canonical motion, which is frequent in real-world videos.

Figure 2: Top: Per-edge motion masks enable fine-grained detection of temporally inconsistent dynamic regions. Bottom: Per-frame masks from previous methods cannot capture transient motions.

Training Strategy

WildPose is trained in staged fashion for robustness and modularity. The update operator is initially trained on only static sequences (static TartanAir V2, TartanGround, synthetic Kubric data). Fine-tuning then incorporates dynamic and noisy-masked data (Dynamic Replica, OmniWorld-Game, dynamic Kubric sequences) with annotated or noisy dynamic masks. The motion detector is trained last, with the update operator frozen, using binary cross-entropy loss where reliable motion annotations exist.

This training approach, coupled with diverse synthetic datasets and procedurally generated camera motions (including underrepresented trajectories like pure rotation and target-locked translation), empirically ensures strong coverage over different motion regimes and intrinsic scene ambiguity.

Experimental Results

WildPose demonstrates consistently superior or competitive absolute trajectory error (ATE RMSE) on challenging real and synthetic datasets spanning static, dynamic, and low-motion sequences. Notably:

• On Wild-SLAM MoCap and Bonn RGB-D Dynamic benchmarks, WildPose attains best or second-best ATE among both dynamic-scene and static-oriented baselines.
• WildPose maintains SOTA or near-SOTA performance also on canonical static datasets (TUM RGB-D, 7-Scenes, ScanNet), outperforming methods like MegaSaM and ViPE which degrade outside dynamic scenarios.
• In short-sequence, low-motion settings (Sintel), WildPose preserves accuracy where per-sequence optimization-based approaches like WildGS-SLAM fail due to lack of sufficient mapping context.

Camera trajectory visualizations further confirm superior or matched tracking accuracy over recent baselines.

Figure 3: Example camera trajectories estimated by WildPose, color-coded by translation error (ATE), showing accurate alignment to ground truth across diverse scenarios.

Architectural Analysis

The empirical ablation study confirms:

• Dynamic-scene fine-tuning is essential but insufficient alone; explicit use of high-fidelity motion masks is necessary to close the performance gap.
• Removing the depth regularizer from global BA during final optimization further improves highly refined pose estimates, indicating the limitations of monocular depth priors when solution is near-optimal.

Detailed Architectural Components

The update operator, built on a ConvGRU core, integrates both the spatial correlation of flow features and global scene context, all derived from the MASt3R encoder.

Figure 4: Update operator architecture showing the integration of feature correlation, context encoding, and current flow for iterative state update.

To bridge the resolution and channel mismatch between MASt3R features and the operator’s requirements, the system introduces parallel flow and context encoder modules with tailored resolution and dimensionality.

Figure 5: Flow and context encoder architectures specifically adapted to extract high-utility representations from MASt3R features.

Limitations and Implications

WildPose, while robust, is sensitive to phenomena unseen in its synthetic-focused curriculum, e.g., drastic photometric changes prevalent in certain real datasets (e.g., Person sequence from Bonn RGB-D). Moreover, it assumes fixed, known camera intrinsics, limiting application to uncalibrated video. Mitigation strategies include incorporating domain adaptation and extending the pipeline with intrinsic parameter optimization.

Figure 6: Challenging samples from Bonn RGB-D showing photometric inconsistency, which presents failure cases for WildPose due to domain gap.

Practical and Theoretical Implications

Practically, WildPose sets a new benchmark for unified, robust pose estimation without reliance on brittle semantic segmentation or heavy loss in static-only regimes. The explicit decoupling of perceptual representation (frozen pretrained 3D features), dynamic outlier handling (high-capacity mask prediction), and geometric optimization (differentiable BA) establishes a modular, extensible framework. This architecture makes possible unified, real-world deployment in unconstrained AR/VR, robotics, and video depth estimation scenarios, as confirmed by SOTA downstream results on long-video depth estimation benchmarks.

Theoretically, the work demonstrates the importance of explicitly managing temporal and spatial ambiguity in dynamic SLAM, leveraging transformer-based geometric pretraining, and the limitations of end-to-end models that lack explicit robustification modules. Future work will likely extend these principles to intrinsic parameter learning, open-world video understanding, and cross-modal sensor fusion.

Conclusion

WildPose provides an authoritative advancement in unified, robust monocular pose estimation, integrating transformer-based, geometrically-pretrained features with a principled, differentiable optimization backend and a novel edge-aware motion mask detector. It achieves strong state-of-the-art accuracy on both static and dynamic datasets, with applicability to demanding downstream geometric tasks. Open challenges remain in further closing the synthetic-to-real domain gap and handling unknown camera intrinsics, which are promising targets for subsequent research.

---

Reference: "WildPose: A Unified Framework for Robust Pose Estimation in the Wild" (2605.12774)