DROID-W: In-the-Wild SLAM & Manipulation

• DROID-W is a comprehensive framework featuring a geographically diverse robot manipulation dataset and a state-of-the-art, uncertainty-aware SLAM system designed for real-world environments.
• The manipulation dataset comprises 76,000 trajectories across 564 unique scenes, offering rich statistical insights like verb entropy (~4.1 bits) and doubled workspace coverage compared to previous collections.
• The SLAM component employs differentiable uncertainty-aware bundle adjustment and multi-view feature consistency to achieve real-time performance (~10 FPS) with superior accuracy on challenging benchmarks.

DROID-W, in the context of contemporary robotics and vision literature, refers to two closely related, independently developed systems that exemplify large-scale, in-the-wild data collection and robust, real-world SLAM (Simultaneous Localization and Mapping). The first instantiation, “DROID-W: An In-the-Wild Large-Scale Robot Manipulation Dataset,” constitutes a geographically and semantically diverse manipulation dataset underpinning advances in policy learning for generalized robotic manipulation (Khazatsky et al., 2024). The second, “DROID-W: DROID-SLAM in the Wild,” denotes a differentiable, uncertainty-aware RGB SLAM system optimized for dynamic, cluttered, and highly variable real-world environments (Li et al., 19 Mar 2026). Both contributions address the critical adaptation of robotics and visual SLAM research towards realistic deployment scenarios beyond laboratory constraints.

1. In-the-Wild Robot Manipulation Dataset: DROID-W

DROID-W is a large-scale dataset comprising 76,000 successful robot manipulation demonstration trajectories (totaling 350 hours of interaction), collected over twelve months by 50 teleoperators across 13 institutions and three continents. These demonstrations capture diverse manipulation skills—86 distinct tasks (“verbs”) performed in 564 unique scenes distributed across ten scene classes (industrial/home/office, kitchen, bathroom, etc.). Scene and object variability is further amplified by systematic scene augmentations (camera movement, distractors, lighting changes), and task instructions are annotated post hoc via crowd-sourcing.

Task categories present in the dataset span pick-and-place, opening/closing, pouring, and multi-step instructions. The empirical attribute distributions—such as verbs and scene types—exhibit heavy-tailed properties, with mean demonstrations per verb ≈ 884 and mean per scene ≈ 135, but significant skew (many verbs/scenes with ~100, some >2000). The dataset also logs ~16,000 “unsuccessful” trajectories, included for research on robustness and error correction but excluded from headline statistics.

A salient metric is the workspace coverage $C_\text{scene}$, defined as the normalized count of unique 3D gripper-interaction points, which is approximately twice as high in DROID-W compared to prior datasets. The verb entropy $H(V)\approx 4.1$ bits, exceeding Bridge V2 ($\sim3.5$) and RT-1 ($\sim2.3$), indicates increased skill diversity. Viewpoint diversity is exceptional: 1417 unique camera poses versus less than 200 in typical laboratory collections.

2. Collection Methodology and Dataset Construction

Data was gathered using a standardized hardware platform: a Franka Emika Panda 7-DoF arm, Robotiq 2F-85 gripper, and three synchronized stereo cameras (two Zed 2 exterior, one Zed Mini wrist-mounted) operating at 1280×720/15 Hz. The Polymetis controller mediated joint/end-effector commands at 15 Hz, and operators issued commands via Oculus Quest 2 controllers (6-DoF pose, gripper state). Rapid scene changes were enabled by a mobile, height-adjustable desk and a single-cable power/design.

Operators proceeded scene-by-scene, calibrating external cameras with checkerboard/OpenCV, entering free-form task lists into a GUI, and performing randomized teleoperated demonstrations. The GUI enforced (by random sampling) task/scene diversity and intermittently prompted scene perturbations. Data capture included RGB/depth, robot state, and operator-assigned success/failure labels. Post hoc natural-language task annotation was crowd-sourced. This process emphasizes both quantitative and qualitative diversity, essential for generalization.

3. Policy Learning Architectures and Evaluation on DROID-W

Policy learning on DROID-W adopts diffusion-based imitation learning architectures (DiffusionPolicy framework, built on Robomimic), aggregating multimodal information: dual 128×128 RGB image views, text embeddings from frozen DistilBERT (768-dim), and proprioceptive (3D gripper position, state). The vision stack leverages ImageNet-pretrained ResNet-50 for 512-dim embeddings. The observation MLP (1024→512→512, ReLU) and U-Net diffusion head (256→512→1024) together parameterize 16-step end-effector action trajectories; the first 8 are executed open-loop each control cycle.

Training utilizes behavioral cloning and diffusion score-matching objectives:

• Behavioral cloning: $$L_{\text{BC}} = \mathbb{E}_{(s,a) \sim D} [ -\log \pi_\theta(a | s) ]$$
• Diffusion denoising: $$L_\text{diff} = \mathbb{E}_{\tau \sim D} \mathbb{E}_{t,\epsilon}[ \|\epsilon - \epsilon_\theta(a_t^\beta | s)\|^2 ]$$, $\epsilon \sim \mathcal{N}(0, I)$

Hyperparameters include batch size 128 (50/50 in-domain/DROID), Adam optimizer (LR $1 \times 10^{-4}$, linear decay), 25k steps (50k for complex multi-step tasks), and data augmentations (random crop, color jitter). Evaluation on real-robot tasks across laboratory/office/household settings quantifies in-distribution and out-of-distribution (OOD) performance:

• In-distribution success rates: In-domain 45%, +OXE 62%, +DROID 67% (±3%)
• OOD: In-domain 21%, +OXE 44%, +DROID 61% (±4%) Subsampling ablations reveal that scene diversity, rather than density within a limited set of scenes, is critical for OOD generalization (+15% absolute SR for the diverse-scene subset).

4. DROID-SLAM in the Wild: System Design and Pipeline

DROID-W (in the SLAM context) operates on monocular RGB video in dynamic, cluttered conditions (Li et al., 19 Mar 2026). Its architecture comprises:

• A FiT3D-fine-tuned DINOv2 backbone extracting per-pixel semantic features $\mathbf{F}_i \in \mathbb{R}^{H \times W \times C}$.
• Pretrained Metric3D network for a soft, low-resolution inverse-depth prior $\mathbf{D}_i \in \mathbb{R}^{H/8 \times W/8}$.
• Per-pixel uncertainty estimation: a shallow affine mapping with Softplus, $H(V)\approx 4.1$0.
• Frame-graph tracking: alternates between differentiable uncertainty-aware bundle adjustment (UBA) and uncertainty parameter updates via multi-view feature consistency.
• Mapping: global BA performed over all keyframes with uncertainty map frozen, producing static/dynamic point cloud segmentations via uncertainty thresholding.

Per-pixel uncertainty is updated to minimize bidirectional visual feature inconsistency across keyframes. Uncertainty $H(V)\approx 4.1$1 suppresses dynamic regions during optimization, operationalized via the similarity-based energy

$H(V)\approx 4.1$2

and a log-prior regularization:

$H(V)\approx 4.1$3

Minimization is performed online with stochastic gradient descent and weight decay on the affine mapping $H(V)\approx 4.1$4; uncertainty cost and BA cost are interleaved in an efficient differentiable pipeline.

5. Differentiable Uncertainty-Aware Bundle Adjustment

DROID-W’s core technical innovation is integrating differentiable UBA, simultaneously estimating camera poses $H(V)\approx 4.1$5 and per-pixel depths $H(V)\approx 4.1$6 by minimizing pixelwise reprojection error, weighted by local uncertainty:

$H(V)\approx 4.1$7

All loss terms (projection, depth, uncertainty regularization) are smoothly differentiable. BA parameters are iteratively optimized (typically 5 unrolled Gauss–Newton steps per window), with uncertainty variable updates proceeding via interleaved gradient steps. This enables efficient, real-time operation (≈10 FPS on RTX 3090), and uncertainty maps automatically suppress influence from dynamic and inconsistent observations.

6. Empirical Performance and Results

DROID-W’s SLAM system demonstrates state-of-the-art robustness and accuracy under challenging conditions:

• On dynamic indoor benchmarks (Bonn RGB-D, TUM RGB-D, DyCheck), DROID-W achieves the lowest ATE RMSE (2.30 cm, 1.36 cm, 0.034 m, respectively).
• On the outdoor “Downtown” DROID-W dataset (with LiDAR/IMU/RTK ground truth), average error is 0.23 m, outperforming Splat-SLAM (1.60 m), DROID-SLAM (1.46 m), WildGS (0.64 m).
• The system operates at ~10 FPS, ~40× faster than comparable WildGS-SLAM pipelines.

Ablation analysis confirms that removing UBA or depth prior substantially degrades performance (ATE 5.13 cm and 3.30 cm, respectively). Uncertainty estimation is critical: omitting decoupled bidirectional loss, no affine mapping, or lack of weight decay all degrade performance to 2.57–2.47 cm. Full system with all components achieves the best tracking fidelity.

7. Reproducibility, Open Resources, and Outlook

All datasets, model code, and hardware assembly guides for both DROID-W datasets and DROID-W SLAM are publicly available (Khazatsky et al., 2024, Li et al., 19 Mar 2026). The manipulation dataset is released under CC-BY 4.0, along with a hardware Bill of Materials, ROS data-collection stack, and policy codebase. The SLAM code and datasets, including challenging outdoor benchmarks, are also open-sourced. Both lines of work prioritize reproducibility and ease of adoption.

DROID-W’s manipulation data and uncertainty-aware SLAM framework collectively represent a substantial advance towards scalable, robust robotic systems capable of reliable operation in novel, unstructured environments. Remaining limitations include the continued reliance on monocular depth priors for metric scale and the sensitivity of early uncertainty initialization to pose accuracy. Proposed directions include integration of stereo/RGB-D streams and regularized joint learning of scene geometry and appearance models.

---

References:

• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (Khazatsky et al., 2024)
• DROID-SLAM in the Wild (Li et al., 19 Mar 2026)