RobotSeg: Foundation Model for Robot Segmentation

• RobotSeg is a foundation model for segmenting robots in images and videos, addressing challenges such as embodiment diversity, appearance ambiguity, and rapid shape changes.
• It integrates SAM 2’s architecture with specialized modules like SEMA for structure-aware temporal feature refinement, RPG for automatic robot prompts, and LET for efficient sparse-label training.
• RobotSeg achieves state-of-the-art performance on large-scale benchmarks by enhancing mask accuracy and temporal consistency compared to generic segmentation models.

to=arxiv_search.search 天天大奖彩票站 research papers on arXiv matching query with abstracts, titles ամեն ինչ Searching arXiv for RobotSeg and closely related robot segmentation work. RobotSeg is a foundation model and dataset for segmenting robots in images and videos. It is designed for robotic perception settings in which general-purpose segmenters remain unreliable because robot embodiment diversity, appearance ambiguity, structural complexity, and rapid shape changes produce broken masks, missing thin parts, and temporal instability. Built on SAM 2’s video segmentation architecture, RobotSeg introduces a Structure-Enhanced Memory Associator (SEMA), a Robot Prompt Generator (RPG), and a Label-Efficient Training Strategy (LET), and is trained together with the VRS benchmark, a large-scale video robot segmentation dataset with hierarchical labels for whole robot, robot arm, and robot gripper (Mei et al., 28 Nov 2025).

1. Definition and problem setting

RobotSeg addresses robot segmentation as a specialized visual perception problem rather than as a direct instance of generic object segmentation. The target is not merely objectness, but accurate mask extraction for articulated robots across images and temporally extended videos, including industrial arms, research platforms, mobile manipulators, and home robots. The motivating use cases are explicit: visual servoing for VLA systems, robot-centric data augmentation, real-to-sim transfer, and safety monitoring in dynamic human–robot environments (Mei et al., 28 Nov 2025).

The difficulty of the problem is tied to four robot-specific factors. First, embodiment diversity means that a single system must cope with platforms such as Franka, UR5, Kuka iiwa, Fanuc, Google Robot, MobileALOHA, xArm, WindowX, Sawyer, and Hello Stretch. Second, appearance ambiguity arises because robot colors and textures often blend with background elements in cluttered workspaces. Third, structural complexity causes general segmenters to break masks at joints or miss thin and occluded parts such as grippers and cables. Fourth, rapid shape changes induce mask flickering and temporal inconsistency in standard video segmentation pipelines (Mei et al., 28 Nov 2025).

RobotSeg is explicitly positioned against three limitations of SAM 2 in this domain. SAM 2 lacks explicit modeling of articulated robot structure, relies on manual prompts such as clicks or boxes, and is typically trained with dense per-frame video annotations. RobotSeg is designed to be structure-aware, automatic, and label-efficient under these conditions (Mei et al., 28 Nov 2025).

A potential source of confusion is terminological. In some robot learning literature, “RobotSeg-like” methods denote trajectory segmentation systems that discover or annotate temporal boundaries and phase identities, whereas RobotSeg in the present sense denotes visual segmentation of robot bodies in image and video. The distinction is material: the former operates over behavioral time series, while the latter operates over spatial masks and temporal propagation in visual streams (Rahman et al., 2 Mar 2026).

2. Architectural design

RobotSeg retains SAM 2’s backbone and mask-decoding structure but adds three robotics-specific modules: SEMA, RPG, and LET. The high-level pipeline is: image encoding, memory encoding and structure-aware feature refinement, automatic prompt-token generation, and mask decoding, with optional interactive refinement at test time (Mei et al., 28 Nov 2025).

The image encoder produces per-frame features $F_t$ for frame $I_t$. Memory from previous frames is encoded into $M_t$, which is then used by SEMA to refine current-frame features into a structure-aware representation $F_t''$. RPG converts memory and segmentation history into robot-specific prompt tokens. A SAM-like two-way transformer mask decoder then consumes the image features, robot tokens, and optional user prompts and produces candidate robot masks, from which the highest IoU-scoring mask is selected (Mei et al., 28 Nov 2025).

The three added components have distinct functions.

Component	Function	Key mechanism
SEMA	Structure-aware temporal propagation	Memory refinement plus edge and multi-scale structural modulation
RPG	Automatic robot prompting	Class tokens and object tokens derived from memory
LET	Sparse-label training	Cycle, semantic, and patch consistency from first-frame-only labels

This decomposition suggests that RobotSeg is less a single architectural novelty than a targeted reconfiguration of a promptable video segmentation foundation model into a robot-specialized system. A plausible implication is that its gains derive from the combination of robotics-specific structure modeling, prompt automation, and supervision economy rather than from any one module alone (Mei et al., 28 Nov 2025).

3. Structure-Enhanced Memory Associator and Robot Prompt Generator

SEMA extends SAM 2’s memory association with explicit structural reasoning. Its temporal context branch refines current features using self-attention and cross-attention with memory: $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$ This branch aggregates temporal information but remains appearance-driven unless supplemented by structure cues (Mei et al., 28 Nov 2025).

The structure branch begins with Canny edge extraction,

$E_t = \mathcal{C}(I_t),$

and uses it to form edge-modulated features

$F_t^{\text{edge}} = F_t \odot (1 + E_t).$

A multi-scale module then produces

$F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$

after which a memory-guided cross-attention operation yields a structure map

$$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$

Finally, SEMA amplifies structurally important locations through

$F_t'' = F_t' \odot \left(1 + \alpha S_t\right),$

where $I_t$0 is learnable. The structure map is supervised during training, encouraging alignment with true robot structure and boundaries (Mei et al., 28 Nov 2025).

RPG removes the need for manual prompts by generating internal robot-aware prompt tokens. It produces two token types. Class tokens are learned embeddings corresponding to “robot arm,” “robot gripper,” and “whole robot,” supplying semantic priors. Object tokens summarize the specific robot instance over time and are computed from previous-frame robot-region features by hierarchical clustering (Mei et al., 28 Nov 2025).

The object-token construction is explicit. Given previous-frame features and binary masks, the method uses a farthest point sampling variant and KMeans to create $I_t$1 macro regions and then $I_t$2 micro prototypes per macro region. Region tokens are formed as

$I_t$3

then concatenated into

$I_t$4

and stacked across memory frames as

$I_t$5

These object tokens provide instance-specific temporal cues to the mask decoder, while class tokens encode target semantics (Mei et al., 28 Nov 2025).

The ablations reported for these modules are consistent with their intended roles. Class tokens alone yield approximately $I_t$6 J&F in the 1-click setting, class plus object tokens raise this to approximately $I_t$7, and SEMA further increases J&F to $I_t$8 on VRS. This suggests that automatic prompting and structure-aware temporal memory are complementary rather than substitutable (Mei et al., 28 Nov 2025).

4. Label-efficient supervision and the VRS dataset

RobotSeg is trained under a sparse-annotation regime. In VRS-Train, only the first frame of each training video is labeled with a ground-truth mask for arm, gripper, and whole robot, while all later frames are unlabeled. LET is the mechanism that makes this regime viable (Mei et al., 28 Nov 2025).

LET combines three losses: $I_t$9 The cycle consistency loss is video-level. Masks are propagated from frame $M_t$0 to frame $M_t$1 and then back to frame $M_t$2, and the loss is

$M_t$3

where $M_t$4 is a linear combination of focal loss and dice loss. The semantic consistency loss regularizes intermediate frames by maximizing cosine similarity between the first-frame robot embedding and the average feature inside each predicted mask: $M_t$5 The patch consistency loss uses DINOv3 patch embeddings to propagate pseudo-labels from frame $M_t$6: $M_t$7 with $M_t$8 an IoU loss (Mei et al., 28 Nov 2025).

VRS is the first large-scale video dataset for robot segmentation. It contains $M_t$9 videos and $F_t''$0 frames. The train/test split is $F_t''$1 videos and $F_t''$2 frames for training, with only the first frame of each training video labeled, and $F_t''$3 videos and $F_t''$4 frames for test, fully annotated across all frames. Total labeled frames across train and test are $F_t''$5 (Mei et al., 28 Nov 2025).

The dataset is assembled from many robot datasets, including DROID, Berkeley Fanuc manipulation, Columbia PushT, Stanford Kuka Multimodal, RoboVQA, MobileALOHA, UCSD Kitchens, Berkeley Bridge, RoboTurk, and DobbE. Its label hierarchy covers whole robot, robot arm, and robot gripper, with polygon masks supplied by professional annotators. This hierarchical design supports part-specific evaluation and downstream use cases such as gripper tracking (Mei et al., 28 Nov 2025).

5. Training protocol and empirical performance

RobotSeg uses SAM 2.1 as its base architecture and is implemented in PyTorch. Training uses RoboEngine-Train with $F_t''$6 robot images and VRS-Train with $F_t''$7 videos and $F_t''$8 frames, runs for $F_t''$9 epochs on $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$0 NVIDIA RTX A5000 GPUs with $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$1 GB, and takes approximately $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$2 hours. The optimizer is AdamW. The image encoder learning rate is $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$3, the remaining components use $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$4, and training follows a cosine decay schedule (Mei et al., 28 Nov 2025).

The model size is $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$5 M parameters, compared with $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$6 M for SAM 2.1; the extra $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$7 M parameters come from SEMA and RPG. FLOPs are $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$8 G versus $$F_t' = \mathrm{MLP}\big(\mathrm{CrossAttn}(\mathrm{SelfAttn}(F_t),\, M_t)\big).$$9 G for SAM 2.1, and inference time is $E_t = \mathcal{C}(I_t),$0 ms per frame on RTX A5000, corresponding to more than $E_t = \mathcal{C}(I_t),$1 FPS (Mei et al., 28 Nov 2025).

Evaluation uses the DAVIS-style J&F metric, where $E_t = \mathcal{C}(I_t),$2 is region similarity and $E_t = \mathcal{C}(I_t),$3 is boundary F-measure, averaged as J&F. The reported prompt settings are Automatic (AU), 1-click (1C), 3-click (3C), Bounding box (BB), and Online-Interactive (OI) (Mei et al., 28 Nov 2025).

On VRS for whole-robot video segmentation, RobotSeg achieves J&F scores of $E_t = \mathcal{C}(I_t),$4 in AU, $E_t = \mathcal{C}(I_t),$5 in 1C, $E_t = \mathcal{C}(I_t),$6 in 3C, $E_t = \mathcal{C}(I_t),$7 in BB, and $E_t = \mathcal{C}(I_t),$8 in OI. For arm segmentation it reports $E_t = \mathcal{C}(I_t),$9 in AU, $F_t^{\text{edge}} = F_t \odot (1 + E_t).$0 in 1C, and $F_t^{\text{edge}} = F_t \odot (1 + E_t).$1 in 3C; for gripper segmentation, $F_t^{\text{edge}} = F_t \odot (1 + E_t).$2 in AU, $F_t^{\text{edge}} = F_t \odot (1 + E_t).$3 in 1C, and $F_t^{\text{edge}} = F_t \odot (1 + E_t).$4 in 3C. By comparison, SAM 2.1 original on whole robot attains $F_t^{\text{edge}} = F_t \odot (1 + E_t).$5 in 1C, $F_t^{\text{edge}} = F_t \odot (1 + E_t).$6 in 3C, $F_t^{\text{edge}} = F_t \odot (1 + E_t).$7 in BB, and $F_t^{\text{edge}} = F_t \odot (1 + E_t).$8 in OI, while SAM 2.1 finetuned on robot data attains $F_t^{\text{edge}} = F_t \odot (1 + E_t).$9 in 1C, $F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$0 in 3C, $F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$1 in BB, and $F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$2 in OI (Mei et al., 28 Nov 2025).

On RoboEngine-Test for whole-robot image segmentation, RobotSeg reports $F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$3 in AU, $F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$4 in 1C, $F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$5 in 3C, and $F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$6 in BB. The corresponding figures in the same table are $F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$7 for RoboEngine in AU, $F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$8 for EVF-SAM in AU, $F_t^{\text{ms}} = \mathcal{MS}(F_t^{\text{edge}}),$9, $$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$0, and $$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$1 for SAM 2.1 original in 1C, 3C, and BB, and $$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$2, $$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$3, and $$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$4 for finetuned SAM 2.1 in 1C, 3C, and BB (Mei et al., 28 Nov 2025).

Category-wise supplementary results show RobotSeg ranking first on $$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$5 robot types for whole robot, $$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$6 for arm, and $$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$7 for gripper. The ablations also isolate the importance of the supervision strategy: finetuned baseline at $$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$8 J&F in 1C, then $$S_t = \sigma(\mathrm{CrossAttn}(F_t^{\text{ms}},\, M_t)).$$9 with cycle loss, $F_t'' = F_t' \odot \left(1 + \alpha S_t\right),$0 with semantic consistency, $F_t'' = F_t' \odot \left(1 + \alpha S_t\right),$1 with patch consistency, and up to $F_t'' = F_t' \odot \left(1 + \alpha S_t\right),$2 after adding RPG and SEMA (Mei et al., 28 Nov 2025).

6. Applications, limitations, and relation to other segmentation literatures

RobotSeg’s reported applications span perception, control, and data generation. Precise masks of arms and grippers can serve visual servoing and VLA control by providing exact effector and body localization. Robot-centric data augmentation methods such as RoboEngine and RoVi-Aug depend on robot masks to composite robots into new backgrounds or swap embodiments. Clean silhouettes can support silhouette-based 3D reconstruction and real-to-sim pipelines. Third-person robot masks can also be used for workspace occupancy monitoring, unsafe-motion detection, and dynamic safety distance enforcement (Mei et al., 28 Nov 2025).

The authors also note limitations. RobotSeg is not best on every robot category, some unusual robots and extreme conditions remain difficult, and the added computation over SAM 2.1 may matter on very resource-limited platforms. Proposed future directions include adding depth, motion, and tactile modalities, developing lighter variants through distillation or architecture compression, and evaluating tighter integration with downstream tasks such as policy learning and navigation (Mei et al., 28 Nov 2025).

The broader term “robot segmentation” is used in at least two additional senses in adjacent literature. In few-shot robotic sequence retrieval, ROSER distinguishes retrieval from “RobotSeg-like” trajectory segmentation by emphasizing ranked retrieval of task-centric subsequences from long unlabeled logs, with segmentation emerging through sliding windows and non-maximum suppression rather than through explicit global boundary prediction (Rahman et al., 2 Mar 2026). In collaborative multi-robot systems, role-engine formulations treat segmentation as the structuring of roles and continuous-time process roles, using Gaussian-process trajectory models, skeleton-based environment representations, Group Role Assignment, and online replanning to segment behavior at both allocation and trajectory levels (Akbari et al., 2023).

This terminological spread suggests that “RobotSeg” has become a family resemblance term across robotics: in perception, it refers to spatial mask extraction of robot bodies in images and videos; in robot learning and multi-robot coordination, it can denote segmentation of trajectories, maneuvers, roles, or behavior streams. The 2025 RobotSeg model is the canonical use for visual segmentation, but the neighboring usages reveal a broader conceptual theme: robot-centric decomposition of continuous raw data into structurally meaningful units (Mei et al., 28 Nov 2025).