HANDAL-X Benchmark: Cross-View Correspondence

• HANDAL-X Benchmark is a large-scale dataset focused on cross-view object correspondence with an emphasis on hand–object interactions.
• It comprises over 44,000 training and 14,000 test image pairs with precise binary mask annotations for robust quantitative evaluation.
• The benchmark enables fair comparison of transformer-based models and test-time training strategies, achieving IoU scores beyond 85%.

HANDAL-X is a large-scale benchmark introduced to evaluate algorithms for cross-view object correspondence under static multi-view image pairs. Emphasizing object-centric perception, particularly hand–object interactions, HANDAL-X enables quantitative comparison of models tasked with localizing the same object instance from one viewpoint to another. The benchmark features densely annotated pairs, rigorous protocols, and serves as a primary testbed for recent advances in conditional segmentation and cross-view matching (Yan et al., 22 Feb 2026).

1. Dataset Construction and Properties

HANDAL-X, as introduced in ObjectRelator [fu2024objectrelator], is constructed using static multi-view image pairs, not video sequences. Each pair comprises images captured from two distinct camera viewpoints arranged around the same object, generally covering a 360° view spectrum. The dataset focuses on everyday objects with a pronounced emphasis on hand–object interaction scenarios; however, a granular category taxonomy is not specified in (Yan et al., 22 Feb 2026).

The public release contains 44,102 training pairs and 14,074 test pairs. Each image pair is annotated with object-centric binary masks. For each pair, the “source” image presents a ground-truth object mask $M_s$, and the “target” image is accompanied by a ground-truth mask $M_t$, utilized solely for evaluation. No distinct validation split is provided.

2. Cross-View Object Correspondence Task Specification

The principal task evaluated on HANDAL-X is cross-view object correspondence. The input to a model consists of a source image $I_s \in \mathbb{R}^{3 \times H \times W}$ with its binary object mask $M_s \in \{0,1\}^{H \times W}$ and a target image $I_t \in \mathbb{R}^{3 \times H \times W}$. The output is a predicted binary mask $\hat{M}_t \in [0,1]^{H \times W}$ that localizes the same object within the target view.

This formulation isolates the problem of viewpoint-invariant object localization, disentangled from temporal cues or video continuity, and evaluates spatial generalization and representation capabilities under challenging appearance changes.

3. Evaluation Protocols and Metrics

HANDAL-X operates with fixed train and test splits: 44,102 training and 14,074 test pairs. The primary evaluation metric is the mean Intersection-over-Union (IoU) between the predicted mask $\hat{M}_t$ and the ground-truth mask $M_t$, averaged over the test set.

Multiple experiment configurations are reported:

• Zero-shot performance (without HANDAL-X fine-tuning)
• Fine-tuned on outside data (e.g., Ego-Exo4D only), with and without test-time training (TTT)
• Fine-tuned directly on the HANDAL-X training split, with and without TTT

Model performance is directly compared across these scenarios, enabling fair and transparent benchmarking of generalization and adaptation capabilities.

4. Comparative Results and State-of-the-Art

The following summarizes quantitative results (mean IoU %) from the HANDAL-X test set:

Method	Fine-tune data	IoU (%)
XSegTx [grauman2024ego]	–	1.5
SEEM [zou2023segment]	–	2.5
PSALM [zhang2024psalm]	–	14.2
PSALM	Ego-Exo4D	39.9
ObjectRelator [fu2024objectrelator]	Ego-Exo4D	42.8
Ours (no TTT)	Ego-Exo4D	78.8
Ours (with TTT)	Ego-Exo4D	80.6
PSALM	Ego-Exo4D + HANDAL-X	83.4
ObjectRelator	Ego-Exo4D + HANDAL-X	84.7
Ours (no TTT)	Ego-Exo4D + HANDAL-X	85.0
Ours (with TTT)	Ego-Exo4D + HANDAL-X	85.3

The results demonstrate significant disparity in cross-view correspondence ability. Zero-shot models achieve modest IoU scores (≤14.2%), while state-of-the-art approaches exploiting HANDAL-X supervision and TTT attain mean IoU beyond 85%, indicating the effectiveness of conditional prediction and adaptation strategies (Yan et al., 22 Feb 2026).

5. Model Architectures and Training Regimes

The most successful pipelines on HANDAL-X utilize architectures centered around transformer backbones, specifically DINOv3-L, with ConvNeXt applied to the source and ViT to the target image feature extraction. The source object mask is normalized as $\tilde{M}_s = M_s / (\sum_{i,j} M_s[i, j] + \tau)$ (with $\tau=1 \times 10^{-6}$), followed by pooling the object-conditioned feature $M_t$0 which is projected as a condition token (CDT).

The transformer input sequence is $M_t$1, where $M_t$2 are the target image patch tokens. Decoding heads predict the mask $M_t$3; a secondary CLS head for binary visibility is present but not leveraged for HANDAL-X evaluation.

Training employs the total loss:

$M_t$4

with $M_t$5, $M_t$6, $M_t$7.

• Mask loss (supervised on $M_t$8): $M_t$9
• Auxiliary loss $I_s \in \mathbb{R}^{3 \times H \times W}$0: identical to mask loss, applied to intermediate outputs
• Cycle-consistency loss (self-supervised, applied on $I_s \in \mathbb{R}^{3 \times H \times W}$1): $I_s \in \mathbb{R}^{3 \times H \times W}$2

Offline training is conducted for 10 epochs with AdamW and a cosine learning rate schedule ($I_s \in \mathbb{R}^{3 \times H \times W}$3), batch size 16, and image resolution $I_s \in \mathbb{R}^{3 \times H \times W}$4, requiring approximately 2 hours.

6. Test-Time Training and Adaptation

A distinctive feature of recent approaches on HANDAL-X is test-time training (TTT). During inference for each test pair, only the last $I_s \in \mathbb{R}^{3 \times H \times W}$5 transformer encoder layers are fine-tuned using the cycle-consistency loss $I_s \in \mathbb{R}^{3 \times H \times W}$6 (no access to ground-truth $I_s \in \mathbb{R}^{3 \times H \times W}$7 is required). Typical hyperparameters are a learning rate $I_s \in \mathbb{R}^{3 \times H \times W}$8, $I_s \in \mathbb{R}^{3 \times H \times W}$9 layers, $M_s \in \{0,1\}^{H \times W}$0 update steps (in accordance with the Ego-Exo4D TTT protocol). The TTT process adds approximately 1 hour of computation per test pair on 8×A800 GPUs.

TTT yields measurable refinement of mask boundaries and improved suppression of distractors, as qualitatively illustrated in the referenced figures. However, the computational overhead is significant. A plausible implication is that TTT offers a substantial accuracy boost on challenging cross-view correspondence, though its utility is contingent on available test-time compute resources (Yan et al., 22 Feb 2026).

7. Qualitative Insights and Failure Modes

Qualitative analysis (Fig. 7 and supplementary Fig. 10 in (Yan et al., 22 Feb 2026)) reveals TTT's efficacy at correcting mask delineation and mitigating false positive activations. Nonetheless, several recurrent failure cases persist:

• Incomplete coverage of the actual target object region
• False predictions over visually similar distractor objects
• Occasional complete misses of the intended object

These challenges underscore the remaining bottlenecks for viewpoint-invariant object segmentation, particularly when category-level ambiguity or strong occlusions are present.

---

HANDAL-X thus establishes a rigorous, large-scale platform for benchmarking cross-view object localization, with annotation quality and evaluation protocols tailored to facilitate progress in mask-based correspondence under substantial viewpoint changes (Yan et al., 22 Feb 2026).