---
title: 'HANDAL-X Benchmark: Cross-View Correspondence'
url: https://www.emergentmind.com/topics/handal-x-benchmark
type: topic
---

# HANDAL-X Benchmark: Cross-View Correspondence

HANDAL-X is a large-scale benchmark introduced to evaluate algorithms for cross-view object correspondence under static multi-view image pairs. Emphasizing object-centric perception, particularly hand–object interactions, HANDAL-X enables quantitative comparison of models tasked with localizing the same object instance from one viewpoint to another. The benchmark features densely annotated pairs, rigorous protocols, and serves as a primary testbed for recent advances in conditional segmentation and cross-view matching [2602.18996].

## 1. Dataset Construction and Properties

HANDAL-X, as introduced in ObjectRelator [fu2024objectrelator], is constructed using static multi-view image pairs, not video sequences. Each pair comprises images captured from two distinct camera viewpoints arranged around the same object, generally covering a 360° view spectrum. The dataset focuses on everyday objects with a pronounced emphasis on hand–object interaction scenarios; however, a granular category taxonomy is not specified in [2602.18996].

The public release contains 44,102 training pairs and 14,074 test pairs. Each image pair is annotated with object-centric binary masks. For each pair, the “source” image presents a ground-truth object mask $M_s$, and the “target” image is accompanied by a ground-truth mask $M_t$, utilized solely for evaluation. No distinct validation split is provided.

## 2. Cross-View Object Correspondence Task Specification

The principal task evaluated on HANDAL-X is cross-view object correspondence. The input to a model consists of a source image $I_s \in \mathbb{R}^{3 \times H \times W}$ with its binary object mask $M_s \in \{0,1\}^{H \times W}$ and a target image $I_t \in \mathbb{R}^{3 \times H \times W}$. The output is a predicted binary mask $\hat{M}_t \in [0,1]^{H \times W}$ that localizes the same object within the target view.

This formulation isolates the problem of viewpoint-invariant object localization, disentangled from temporal cues or video continuity, and evaluates spatial generalization and representation capabilities under challenging appearance changes.

## 3. Evaluation Protocols and Metrics

HANDAL-X operates with fixed train and test splits: 44,102 training and 14,074 test pairs. The primary evaluation metric is the mean Intersection-over-Union (IoU) between the predicted mask $\hat{M}_t$ and the ground-truth mask $M_t$, averaged over the test set.

Multiple experiment configurations are reported:
- Zero-shot performance (without HANDAL-X fine-tuning)
- Fine-tuned on outside data (e.g., Ego-Exo4D only), with and without test-time training (TTT)
- Fine-tuned directly on the HANDAL-X training split, with and without TTT

Model performance is directly compared across these scenarios, enabling fair and transparent benchmarking of generalization and adaptation capabilities.

## 4. Comparative Results and State-of-the-Art

The following summarizes quantitative results (mean IoU %) from the HANDAL-X test set:

| Method                     | Fine-tune data        | IoU (%) |
|----------------------------|----------------------|---------|
| XSegTx [grauman2024ego]    | –                    | 1.5     |
| SEEM [zou2023segment]      | –                    | 2.5     |
| PSALM [zhang2024psalm]     | –                    | 14.2    |
| PSALM                      | Ego-Exo4D            | 39.9    |
| ObjectRelator [fu2024objectrelator] | Ego-Exo4D | 42.8    |
| Ours (no TTT)              | Ego-Exo4D            | 78.8    |
| Ours (with TTT)            | Ego-Exo4D            | 80.6    |
| PSALM                      | Ego-Exo4D + HANDAL-X | 83.4    |
| ObjectRelator              | Ego-Exo4D + HANDAL-X | 84.7    |
| Ours (no TTT)              | Ego-Exo4D + HANDAL-X | 85.0    |
| Ours (with TTT)            | Ego-Exo4D + HANDAL-X | 85.3    |

The results demonstrate significant disparity in cross-view correspondence ability. Zero-shot models achieve modest IoU scores (≤14.2%), while state-of-the-art approaches exploiting HANDAL-X supervision and TTT attain mean IoU beyond 85%, indicating the effectiveness of conditional prediction and adaptation strategies [2602.18996].

## 5. Model Architectures and Training Regimes

The most successful pipelines on HANDAL-X utilize architectures centered around transformer backbones, specifically DINOv3-L, with ConvNeXt applied to the source and ViT to the target image feature extraction. The source object mask is normalized as $\tilde{M}_s = M_s / (\sum_{i,j} M_s[i, j] + \tau)$ (with $\tau=1 \times 10^{-6}$), followed by pooling the object-conditioned feature $z_s = \sum_{i,j} \tilde{M}_s[i, j] F_s[:, i, j]$ which is projected as a condition token (CDT).

The transformer input sequence is $[\mathrm{CLS}, \mathrm{CDT}, x_1, \ldots, x_n]$, where $x_i$ are the target image patch tokens. Decoding heads predict the mask $\hat{M}_t$; a secondary CLS head for binary visibility is present but not leveraged for HANDAL-X evaluation.

Training employs the total loss:
$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{mask}} + \lambda_{\text{aux}} \mathcal{L}_{\text{aux}} + \lambda_{\text{cycle}} \mathcal{L}_{\text{cycle}}
$$
with $\lambda_{\text{dice}}=5$, $\lambda_{\text{aux}}=1$, $\lambda_{\text{cycle}}=10$.

- Mask loss (supervised on $M_t$): $\mathcal{L}_{\text{mask}}(M_t,\hat{M}_t) = \mathcal{L}_{\text{bce}}(M_t,\hat{M}_t) + \lambda_{\text{dice}} \mathcal{L}_{\text{dice}}(M_t,\hat{M}_t)$
- Auxiliary loss $\mathcal{L}_{\text{aux}}$: identical to mask loss, applied to intermediate outputs
- Cycle-consistency loss (self-supervised, applied on $M_s$): $\mathcal{L}_{\text{cycle}} = \mathcal{L}_{\text{bce}}(M_s, \hat{M}_s)$

Offline training is conducted for 10 epochs with AdamW and a cosine learning rate schedule ($2\times10^{-4} \rightarrow 2\times10^{-6}$), batch size 16, and image resolution $512^2$, requiring approximately 2 hours.

## 6. Test-Time Training and Adaptation

A distinctive feature of recent approaches on HANDAL-X is test-time training (TTT). During inference for each test pair, only the last $K$ transformer encoder layers are fine-tuned using the cycle-consistency loss $\mathcal{L}_{\text{cycle}}$ (no access to ground-truth $M_t$ is required). Typical hyperparameters are a learning rate $\mathrm{lr}_{ttt}=5\times10^{-6}$, $K=11$ layers, $T=6$ update steps (in accordance with the Ego-Exo4D TTT protocol). The TTT process adds approximately 1 hour of computation per test pair on 8×A800 GPUs.

TTT yields measurable refinement of mask boundaries and improved suppression of distractors, as qualitatively illustrated in the referenced figures. However, the computational overhead is significant. A plausible implication is that TTT offers a substantial accuracy boost on challenging cross-view correspondence, though its utility is contingent on available test-time compute resources [2602.18996].

## 7. Qualitative Insights and Failure Modes

Qualitative analysis (Fig. 7 and supplementary Fig. 10 in [2602.18996]) reveals TTT's efficacy at correcting mask delineation and mitigating false positive activations. Nonetheless, several recurrent failure cases persist:
- Incomplete coverage of the actual target object region
- False predictions over visually similar distractor objects
- Occasional complete misses of the intended object

These challenges underscore the remaining bottlenecks for viewpoint-invariant object segmentation, particularly when category-level ambiguity or strong occlusions are present.

---

HANDAL-X thus establishes a rigorous, large-scale platform for benchmarking cross-view object localization, with annotation quality and evaluation protocols tailored to facilitate progress in mask-based correspondence under substantial viewpoint changes [2602.18996].

Source: https://www.emergentmind.com/topics/handal-x-benchmark