RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video

Abstract: Accurate robot segmentation is a fundamental capability for robotic perception. It enables precise visual servoing for VLA systems, scalable robot-centric data augmentation, accurate real-to-sim transfer, and reliable safety monitoring in dynamic human-robot environments. Despite the strong capabilities of modern segmentation models, surprisingly it remains challenging to segment robots. This is due to robot embodiment diversity, appearance ambiguity, structural complexity, and rapid shape changes. Embracing these challenges, we introduce RobotSeg, a foundation model for robot segmentation in image and video. RobotSeg is built upon the versatile SAM 2 foundation model but addresses its three limitations for robot segmentation, namely the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations, by introducing a structure-enhanced memory associator, a robot prompt generator, and a label-efficient training strategy. These innovations collectively enable a structure-aware, automatic, and label-efficient solution. We further construct the video robot segmentation (VRS) dataset comprising over 2.8k videos (138k frames) with diverse robot embodiments and environments. Extensive experiments demonstrate that RobotSeg achieves state-of-the-art performance on both images and videos, establishing a strong foundation for future advances in robot perception.

• The paper introduces a novel structure-aware segmentation model that integrates temporal memory and edge detection for accurate robot segmentation.
• The model achieves 85.1 JF in whole robot segmentation, outperforming SAM 2.1 and RoboEngine by over 11 points in automatic settings.
• The VRS dataset offers 138,000+ annotated frames from 2,812 videos, enabling efficient label propagation and detailed part-level analysis.

RobotSeg: Foundation Model and Dataset for Segmentation of Robots in Images and Video

Introduction

RobotSeg introduces a specialized foundation model for segmenting robots in both images and videos, addressing persistent challenges in robotic perception that have limited the accuracy and reliability of existing segmentation systems. Contemporary state-of-the-art models such as SAM 2.1 and RoboEngine, though powerful in generic object segmentation, exhibit marked deficiencies in robot-specific scenarios due to embodiment diversity, visual ambiguity, complex articulated structures, and dynamic shape changes. RobotSeg builds directly upon SAM 2 architecture, incorporating tailored innovations that enable structure-aware, temporally consistent, and prompt-efficient segmentation, alongside a newly created Video Robot Segmentation (VRS) dataset for robust benchmarking.

VRS Dataset: Scale and Diversity

The VRS dataset is designed as the first comprehensive benchmark for video-based robot segmentation. Comprising 2,812 videos with over 138,000 frames, it is 38$\times$ larger than the prior RoboEngine dataset, spanning diverse robot types (Franka, UR5, Sawyer, MobileALOHA, and more), environments, and lighting conditions. Annotation is performed hierarchically, with detailed masks for whole robot, arm, and gripper, enabling fine-grained analysis and downstream applications such as part-level tracking and robot-centric data augmentation.

Figure 1: Examples from VRS across scenes, embodiments, and lighting, with the robot arm in red and gripper in green.

Figure 2: Distribution of robot embodiments represented in VRS, underscoring explicit multi-platform coverage.

The dataset supports semi-supervised learning: in each training video, only the first frame is fully annotated, reducing labor while retaining quality for temporal propagation experiments.

RobotSeg Architecture: Structure-Aware Temporal Segmentation

RobotSeg advances robotic segmentation through three core modules:

1. Structure-Enhanced Memory Associator (SEMA):

RobotSeg overcomes limitations in temporal and part-level consistency by encoding previous frame features and masks into memory, applying self-attention and cross-attention to fuse current frame features and temporal context. A structure branch extracts image edges via Canny filtering and enhances multi-scale representations, enabling precise mask generation especially for articulated and deformable robots.

Figure 3: SEMA integrates memory from prior frames with structural boundaries extracted via edge detection.

2. Robot Prompt Generator (RPG):

Autonomous segmentation is achieved without manual prompt input. RPG generates semantic class tokens (robot, arm, gripper) from a learnable bank and object tokens by clustering historical features within segmented regions, leveraging both prior context and temporal history for highly stable mask propagation.

Figure 4: RPG produces class tokens for semantic priors and object tokens from memory clustering to guide segmentation.

3. Label-Efficient Training Strategy (LET):

RobotSeg dispenses with the need for per-frame annotation; only the first frame’s mask supervises training. This is enabled by three hierarchical consistency losses:

• Cycle Consistency Loss: Ensures that forward and backward mask predictions at the start/end frames are aligned.
• Semantic Consistency Loss: Forces mask predictions in intermediate frames to remain semantically similar to the first frame, measured via feature cosine similarity.
• Patch Consistency Loss: Uses DINOv3 patch similarity to propagate pseudo-labels for fine spatial alignment and patch-level fidelity.

  Figure 5: LET integrates cycle, semantic, and patch consistency, supervising video learning from the first frame only.

  

  Figure 6: DINOv3-generated pseudo-labels enable reliable patch-level supervision across temporally distant frames.

Experimental Results

Quantitative evaluations on both the VRS and RoboEngine datasets demonstrate that RobotSeg achieves state-of-the-art performance under automatic and prompt-based modes (1-click, 3-clicks, bounding box, and online-interactive). On VRS, RobotSeg attains 85.1 $J_\text{F}$ for whole robot segmentation, outperforming SAM 2.1 (finetuned) by +11.5 points and RoboEngine by +11.0 points in automatic settings. Significant gains are also observed for arm and gripper segmentation, supporting fine-grained downstream use.

Figure 7: Qualitative comparison (left: gripper, right: whole robot) highlighting structural and temporal improvements with RobotSeg.

Comparative ablation shows each architectural innovation—robot-specific fine-tuning, hierarchical supervision via LET, the RPG, and multi-scale structure enhancement—delivers additive improvements. Notably, RobotSeg achieves this with a compact parameterization (41.3M), ensuring practical inference speed ($\sim$94.2 ms/frame) and scalability.

Qualitative Comparison and Category-Wise Robustness

Visual results confirm that RobotSeg yields temporally consistent and structurally accurate masks even in cluttered, ambiguous, or rapidly changing scenes. Existing models often fail in these settings, producing fragmentary or incorrect segmentation.

Figure 8: Whole-robot segmentation by RoboEngine vs. RobotSeg under automatic mode.

Figure 9: Arm segmentation comparison, where EVF-SAM misses articulation and RobotSeg retains temporal part localization.

Figure 10: Robot gripper segmentation: EVF-SAM confuses background, RobotSeg maintains specificity.

RobotSeg maintains top accuracy across all ten robot embodiments in VRS, with minimal drop in challenging cases (MobileALOHA, Hello Stretch), attesting to strong generalization.

Efficiency and Practical Deployment

RobotSeg outperforms much larger models in both efficiency (FLOPs, inference time) and segmentation quality, rendering it suitable for real-world deployment in robot tracking, policy learning, simulation transfer, and safety systems.

Future Directions

The structure-aware workflow in RobotSeg admits further extension: integration of multimodal inputs (depth, tactile), lightweight architectures for embedded deployment, and closed-loop connections with downstream policy and reasoning. Application to 3D reconstruction is promising, given improved mask completeness.

Figure 11: Complete masks from RobotSeg enable accurate 3D reconstruction with SAM 3D.

Conclusion

RobotSeg delivers a unified solution to robot segmentation in images and video, uniquely addressing the challenges posed by articulation, ambiguity, and temporal dynamics. Through structure-aware memory association, autonomous prompt generation, and label-efficient learning, RobotSeg sets a new baseline for both academic research and practical robotics, facilitating high-fidelity data augmentation, precise perception, and robust robotic learning.