HANDAL-X: Robotic-Ready Cross-View Correspondence

• HANDAL-X is a benchmark for robotic-ready cross-view mask correspondence that tests pixel-level object matching across diverse viewpoints without using camera metadata.
• It provides 39,000 training and 13,000 test image pairs with precise binary masks across 17 object classes, emphasizing fine-grained manipulation for embodied AI.
• The evaluation employs mIoU and cycle-consistency loss, with baseline methods showing significant improvements over zero-shot approaches via multi-prompt and test-time adaptation.

HANDAL-X is a large-scale benchmark for robotic-ready cross-view object correspondence, targeting the challenging problem of associating object masks across large viewpoint and appearance changes. It is designed to test models’ ability to predict precise pixel-level masks for the same object instance in coordinated but substantially different image pairs, with particular emphasis on scenarios directly relevant to robotic manipulation and perception research.

1. Dataset Specification

HANDAL-X consists of paired images with precise, pixel-level binary object masks in both views. The primary task is, given a query image and corresponding object mask, to localize the same object in a target image taken from a different viewpoint, with no access to camera pose or geometric metadata. The core characteristics are as follows:

• Size and Structure: The dataset contains 39,000 training image pairs (78,000 masks) and 13,000 test pairs (26,000 masks), with each image pair annotated at the pixel level. Each mask is a single-channel binary image aligned in resolution with the corresponding RGB frame.
• Object Classes: The 17 object classes were curated for “robotic-ready” relevance, focusing on tools, utensils, and other manipulable items. HANDAL-X emphasizes fine-grained manipulation scenarios of importance to embodied AI and robotics.
• Viewpoints: Each sample includes two tightly synchronized images—denoted $I_q$ (query) and $I_t$ (target)—without any extrinsic or pose information. Cross-view variation is substantial, aimed at mimicking real multiview perception and manipulation settings.
• Annotation Protocol: Pixel-level binary masks $M$ are provided for each view and object, ensuring spatial correspondence can be evaluated with high precision.

A summary is presented in the following table:

Split	#Pairs	#Masks	#Classes
Train	39,000	78,000	17
Test	13,000	26,000	17

An alternate version documented in (Yan et al., 22 Feb 2026) expands to 44,102 training pairs and 14,074 test pairs, covering ≈30 object classes with multi-view exocentric imaging, although the core robotic-oriented evaluation remains constant (Pan et al., 25 Nov 2025, Yan et al., 22 Feb 2026).

2. Evaluation Metrics

Performance on HANDAL-X is evaluated using mean Intersection-over-Union (mIoU), defined as

$$\mathrm{mIoU} = \frac{1}{N}\sum_{i=1}^N \frac{|\hat M_i \cap M_i|}{|\hat M_i \cup M_i|}$$

where $\hat M_i$ and $M_i$ represent the predicted and ground-truth masks for the $i$-th test sample, and $N$ is the number of samples. Only mIoU is reported for official comparisons; other segmentation metrics such as precision and recall are provided for reference but not used for ranking (Pan et al., 25 Nov 2025, Yan et al., 22 Feb 2026).

3. Baseline and State-of-the-Art Results

Several prominent methods have been benchmarked on HANDAL-X, both in a strict zero-shot (trained elsewhere, tested directly) and fine-tuned regime. Representative quantitative results are summarized below:

Method	mIoU (Zero-Shot)	mIoU (Fine-Tuned)
XSegTx (COCO pretrained)	1.5	–
SEEM (COCO/RefCOCO)	2.5	–
PSALM (multi-modal ZSL)	14.2	83.4
PSALM (Ego-Exo4D FT)	39.9	–
ObjectRelator (Ego-Exo4D)	42.8	84.7
V$^2$-SAM (single expert)	66.4	–
V$^2$-SAM (multi-expert)	77.2	–
CCMP (ours)	78.8	85.0

V$I_t$0-SAM (Pan et al., 25 Nov 2025) and the cycle-consistent mask prediction model CCMP (Yan et al., 22 Feb 2026) demonstrate substantial improvements (>+34 mIoU) over previous best results in the zero-shot scenario. Fine-tuning further narrows the gap between these models, with the best CCMP configuration achieving up to 85.0 mIoU on the test set.

Test-time training (TTT) in (Yan et al., 22 Feb 2026) delivers small but reproducible gains (1.8 points zero-shot, 0.3 points fine-tuned), further underscoring the impact of strong self-supervision in this setting.

4. Methodological Innovations and Evaluation Protocols

4.1 Multi-Prompt and Multi-Expert Mechanisms

In V$I_t$1-SAM (Pan et al., 25 Nov 2025), two complementary prompt generators are deployed:

• V$I_t$2-Anchor (coordinate prompt): Uses DINOv3 features to compute a dense cosine similarity heatmap $I_t$3 between query and target patches, identifying anchor correspondences robust to viewpoint/scale variation, with stratified sampling to enforce minimal spatial separation:

$I_t$4

$I_t$5

• V$I_t$6-Visual (appearance prompt): Pools masked SAM2 encoder features and applies VPMatcher, which combines multi-head cross-attention, residual MLP alignments, and a CNN upsampling pathway modulated by FiLM, generating an embedding aligned to the target domain.

A Post-hoc Cyclic Consistency Selector (PCCS) adaptively chooses the expert (anchor-only, visual-only, fused) that achieves the highest cyclic mask agreement, boosting mIoU by approximately 10 points over single-expert setups. Sparse anchors (one per object) are empirically superior.

4.2 Cycle-Consistent Mask Prediction

(Yan et al., 22 Feb 2026) introduces a flexible, lightweight cross-view mask prediction model trained with a composite objective:

• Segmentation loss: Combination of pixelwise BCE and DICE:

$I_t$7

• Cycle-consistency loss: Output mask in the target view is recursively mapped back to source, with agreement enforced by the same BCE objective:

$I_t$8

• Total objective:

$I_t$9

A test-time adaptation protocol leverages $M$0 exclusively to finetune the last $M$1 transformer layers per test image pair with $M$2 steps at $M$3, providing label-free online adaptation.

5. Application Domains and Significance

HANDAL-X is tailored for evaluating and benchmarking cross-view object correspondence algorithms relevant to:

• Robotic manipulation, where precise multi-view association supports grasp planning, workspace understanding, and automation under challenging occlusion or lighting.
• Embodied perception, especially in settings where pose information is unavailable or unreliable.
• The development of generic, robust mask-matching paradigms transferable across datasets, as evidenced by strong zero-shot generalization performances.

The inclusion of object categories that mirror practical manipulation tasks and the use of pixel-level mask targets set a high bar for spatial accuracy, moving beyond bounding-box or coarse-grained correspondence.

6. Known Limitations and Open Questions

Despite progress, several performance ceilings and open challenges persist:

• Failure cases typically involve small or partially occluded objects and visual confusion between similar items.
• Existing methods struggle if direct geometric cues are missing and cannot always resolve correspondence for transparent or reflectively-illuminated categories.
• Class-incremental and continual learning in the context of HANDAL-X remains underexplored.

A plausible implication is that additional structural constraints or fusion of sparse spatial anchors with stronger priors may yield future improvements. The field remains active, with the HANDAL-X benchmark serving as a standard for evaluating generalization and robustness in cross-view mask correspondence (Pan et al., 25 Nov 2025, Yan et al., 22 Feb 2026).