---
title: 'MM-ITF: Transformer for Pointing-Target Estimation'
url: https://www.emergentmind.com/topics/multi-modality-inter-transformer-mm-itf
type: topic
---

# MM-ITF: Transformer for Pointing-Target Estimation

Searching arXiv for MM-ITF and the cited related transformer work to ground the article in current papers.
Multi-Modality Inter-TransFormer (MM-ITF) is a transformer-based architecture for pointing-target estimation in human-robot interaction (HRI), introduced in “Pointing-Guided Target Estimation via Transformer-Based Attention” [2509.05031]. It is designed for a controlled tabletop scenario with the NICOL humanoid robot, where a person indicates a target object through natural pointing gestures. The model treats target estimation as a multimodal matching problem over monocular RGB observations: it combines 2D hand pose landmarks, 2D object locations, and an optional relationship feature that captures the angular alignment between the finger direction and each object, then uses inter-modality attention to assign a likelihood score to each candidate object and select the most likely target [2509.05031].

## 1. Problem formulation and task setting

MM-ITF addresses the problem of inferring, from an image, whether the human is pointing and, if so, which object in the scene is the target [2509.05031]. The formulation differs from explicit geometric pipelines that predict a pointing ray and intersect it with a scene plane. Instead, each detected object is scored directly, and the highest-scoring object is selected as the predicted target. The task is described as a ranking or binary relevance problem over candidate hand-object pairs: each object receives a candidate score, the score reflects whether that object satisfies the pointing relation, and the highest-scoring object is chosen.

The model was evaluated in a fixed tabletop HRI setup with the NICOL robot. The dataset consists of **30 videos**, **18 participants**, recordings from the **fisheye camera in NICOL’s left eye**, and **10 YCB objects** placed on the table. Each participant completed **9 pointing tasks**, comprising **7 single-object pointing tasks** and **2 bi-manual tasks with two objects simultaneously**. The resulting corpus contains **572 samples** total, of which **356** are pointing and **216** are resting or non-pointing. Because transformers usually need much more data, the training set was expanded by mirroring, random shifts in \(x\) and \(y\), rotations, and Gaussian noise on 2D coordinates, yielding **2,342,912 samples** [2509.05031].

The stated motivation is methodological as well as practical. Prior pointing-target estimation methods are described as relying on explicit geometric reasoning, 3D reconstruction, depth sensors, wearables, or multiple processing stages. MM-ITF was proposed as a purely 2D, RGB-only, modular, end-to-end alternative that can operate in a robot setting without extra calibration or hardware. This suggests that the contribution is not only a new predictor, but also a reframing of deictic gesture understanding as multimodal transformer inference over structured 2D signals rather than explicit scene geometry.

## 2. Input modalities and representation design

MM-ITF uses three kinds of information: hand pose, object locations, and a relationship feature [2509.05031]. The hand pose input is the set of **21 MediaPipe landmarks**, represented as
$$
\mathcal{P} = \{\mathbf{lm}^p_i\}_{i=1}^{21}, \quad \mathbf{lm}^p_i \in \mathbb{R}^2.
$$
These landmarks encode hand configuration, orientation, and the distinction between pointing and resting postures.

Object information is derived from object detection bounding boxes, with centroids used as object positions:
$$
\mathcal{O} = \{\mathbf{c}^o_i\}_{i=1}^{N_t} \cup \{\mathbf{c}_{\text{non-object}}\}.
$$
A special non-object token is added with coordinates \((-1,-1)\) to represent the possibility that no object is being pointed at. This allows the architecture to model non-pointing cases within the same candidate-scoring framework.

The relationship feature encodes angular alignment between the finger direction and each candidate object centroid. The finger direction is defined using the **index fingertip** and **index finger DIP joint**:
$$
\mathbf{v}_\text{finger} = \mathbf{lm}^p_{\text{index\_finger\_tip}} - \mathbf{lm}^p_{\text{index\_finger\_dip}}.
$$
For object centroid \(i\),
$$
\mathbf{v}_{\text{to\_centroid}, i} = \mathbf{c}^o_i - \mathbf{lm}^p_{\text{index\_finger\_tip}},
$$
and the angle is computed as
$$
\theta_i = \arccos \left( \frac{\mathbf{v}_\text{finger} \cdot \mathbf{v}_{\text{to\_centroid}, i}}{\|\mathbf{v}_\text{finger}\| \cdot \|\mathbf{v}_{\text{to\_centroid}, i}\|} \right).
$$
A non-relation token \(\theta_{\text{non-relation}}=-1\) is used for the non-pointing case. The paper emphasizes that this relationship feature is important because it explicitly encodes how well each object aligns with the pointing direction, while still leaving the transformer responsible for learning how to combine this with broader contextual information.

Before entering the transformer, 2D coordinates are normalized to \([0,1]\):
$$
\tilde{x} = \frac{x}{W}, \quad \tilde{y} = \frac{y}{H},
$$
where \(W\) and \(H\) denote image width and height. Inputs are then projected into a common transformer dimension \(d_T\), with sinusoidal positional encoding applied to spatial coordinates:
$$
\mathcal{PE}(\tilde{x}, \tilde{y}) = \text{concat}(PE(\tilde{x}), PE(\tilde{y})).
$$
The embedded streams are
$$
\mathcal{P'} = \{\mathcal{PE}(W_h \mathbf{lm}^p_i)\}_{i=1}^{21},
$$
$$
\mathcal{O'} = \{\mathcal{PE}(W_o \mathbf{c}^o_i)\}_{i=1}^{N_t+1},
$$
$$
\mathcal{R'} = \{W_r \mathbf{\theta}^r_i\}_{i=1}^{N_t+1}.
$$
These definitions make the representation explicitly modular: pose, object, and relation are separate streams that are aligned only after embedding.

## 3. Inter-modality attention and transformer architecture

MM-ITF is implemented as a modular encoder-decoder transformer with separate input streams for pose, object, and relationship information [2509.05031]. Its central architectural novelty is inter-modality attention.

In the encoder, **pose features are used as queries**, while **object features serve as keys and values**. As a result, hand pose tokens attend to object tokens, forming what the paper describes as a pose-object memory that captures the global relationship between the hand and the scene objects. Conceptually, the encoder asks: given this hand configuration, which objects in the scene are relevant? The “inter-modality” designation refers precisely to this cross-stream attention, rather than attention operating only within a single modality.

In the decoder, the relationship tokens \(\mathcal{R'}\), which correspond to candidate hand-object pairs, first undergo self-attention and then cross-attend to the pose-object memory from the encoder. Each relationship token can therefore integrate both its own angular alignment information and the global scene context learned by the encoder. The output is a sequence of decoded hand-object pair embeddings, one per candidate object.

Scoring is performed by a feedforward network with sigmoid activation. Each decoded token corresponds to one candidate object pair, and the network produces a likelihood score indicating whether that pair satisfies the pointing relation. The final prediction is the index \(j\) of the token with maximum score,
$$
\hat{j} = \arg\max_j s(o_j).
$$
The system therefore does not depend on a separate geometric post-processing stage: target estimation is obtained directly from learned ranking over candidate object tokens.

The paper frames the design as modular and extendable. That characterization follows from the explicit separation of pose, object, and relationship streams, the use of learned projections into a shared transformer space, and the fact that relationship tokens act as a candidate-wise interface between encoder context and final ranking. A plausible implication is that additional streams, such as gaze, could be integrated in a structurally similar way without changing the overall encoder-decoder logic.

## 4. Training protocol, baseline, and reported performance

The task is formulated as binary classification over candidate hand-object pairs and is optimized with **Binary Cross-Entropy (BCE)** loss [2509.05031]. At inference time, the model does not threshold scores into a hard pointing or non-pointing decision; instead, it ranks raw scores and chooses the highest-scoring object. The paper presents this as allowing the network to function as a target proposal mechanism.

The experimental protocol uses **8-fold cross-validation** over **30 scenes**. In each fold, the model is trained on **21 scenes**, validated on **3 scenes**, and tested on **6 scenes**; all scenes are used for validation once. Three systems are compared: a baseline from **Ali et al.**, MM-ITF with **2 modalities** (pose + object), and MM-ITF with **3 modalities** (pose + object + relationship). The baseline uses an MLP to determine whether the hand is pointing, then applies a geometric rule based on a line through the **wrist and index finger**, with the target defined as the object centroid nearest to that line.

| Variant | Accuracy / F1 | Top-2 accuracy |
|---|---:|---:|
| Baseline | \(0.89 \pm 0.008\) / \(0.85 \pm 0.008\) | \(0.96 \pm 0.004\) |
| MM-ITF, 2 modalities | \(0.71 \pm 0.044\) / \(0.67 \pm 0.041\) | \(0.92 \pm 0.014\) |
| MM-ITF, 3 modalities | \(0.90 \pm 0.017\) / \(0.90 \pm 0.019\) | \(0.96 \pm 0.008\) |

The full three-modality model reports **Accuracy \(0.90 \pm 0.017\)**, **Precision \(0.88 \pm 0.019\)**, **Recall \(0.92 \pm 0.019\)**, **F1 \(0.90 \pm 0.019\)**, and **Top-2 accuracy \(0.96 \pm 0.008\)**. The two-modality version reports **Accuracy \(0.71 \pm 0.044\)**, **Precision \(0.70 \pm 0.037\)**, **Recall \(0.68 \pm 0.044\)**, **F1 \(0.67 \pm 0.041\)**, and **Top-2 accuracy \(0.92 \pm 0.014\)**. The baseline reports **Accuracy \(0.89 \pm 0.008\)**, **Precision \(0.84 \pm 0.007\)**, **Recall \(0.90 \pm 0.012\)**, **F1 \(0.85 \pm 0.008\)**, and **Top-2 accuracy \(0.96 \pm 0.004\)**.

The main empirical conclusion in the paper is that the three-modality MM-ITF matches or slightly exceeds the baseline, while the two-modality version is substantially worse in exact top-1 prediction. The contrast between \(0.71\) and \(0.90\) accuracy is used to argue that the relationship token is crucial for accurate ranking. The high **Top-2 accuracy** of the two-modality model nevertheless indicates that it learns a useful spatial relation even without the explicit angular feature.

## 5. Patch confusion matrix, ablations, and limitations

A distinctive contribution of the paper is the **patch confusion matrix**, introduced because a standard confusion matrix over object indices is not very informative when object identities and locations vary across scenes [2509.05031]. Image space is discretized into evenly sized, non-overlapping patches. Each object centroid, predicted or ground truth, is assigned to a patch based on its \(x,y\) location, and the matrix records how often a ground-truth patch is predicted as a different patch. Rows correspond to ground-truth patches, columns to predicted patches, the diagonal corresponds to correct spatial predictions, and the matrix includes a non-object row and column for non-pointing cases.

The paper uses this matrix to expose spatial error structure rather than mere categorical error counts. It reports that the model often gets close spatially, especially for objects near the pointing ray, but also reveals a specific failure mode: when multiple objects are aligned with the pointing direction, the model often predicts an object **behind** the actual target from the participant’s perspective. Concrete examples include patch \((1,12)\) misclassified as \((3,12)\) in **88%** of cases, \((2,10)\) as \((3,10)\) in **38%**, and \((1,14)\) as \((3,15)\) in **50%**. The paper interprets these patterns as evidence of strong reliance on hand-object alignment rather than full relational reasoning.

The principal ablation compares the two-modality and three-modality variants. Without the relationship feature, exact top-1 performance drops sharply to **71%** accuracy; with the relationship feature, top-1 accuracy rises to **90%** and top-2 accuracy to **96%**. The paper therefore treats the angular relationship token not as an auxiliary cue but as a major source of fine-grained target discrimination.

The limitations are explicit. The setup assumes a **fixed camera, fixed table, and fixed participant position**, and evaluation is confined to a **controlled tabletop environment**. The model can misclassify when **multiple objects lie on the same pointing line**, may predict an object **behind the true target** if several objects are aligned, and sometimes relies more on **hand position** than detailed hand articulation, which can cause confusion with resting hands. The authors suggest future work on more dynamic and flexible environments, adding further modalities such as **gaze**, and enriching the global context modeled by the encoder. This suggests that the present MM-ITF should be understood as a proof of viability for RGB-only transformer-based deictic grounding, rather than a complete account of general pointing understanding in unconstrained environments.

## 6. Position within multimodal transformer research

Within the wider multimodal transformer literature, MM-ITF belongs to a family of architectures that use tokenized modality-specific inputs and explicit cross-stream interaction, but it is narrower in scope than general-purpose multimodal backbones [2509.05031]. The paper itself states that it adapts ideas from **Human-Object Interaction (HOI)** transformers and especially **inter-modality attention**, repurposing them for deictic gesture understanding in HRI.

Several related transformer directions clarify this placement. “Meta-Transformer: A Unified Framework for Multimodal Learning” proposes a modality-shared encoder with a unified tokenizer across **12 modalities**, emphasizing a frozen shared backbone and unpaired multimodal learning rather than explicit inter-modality fusion [2307.10802]. “Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation” introduces MUTR, which integrates multi-modal references and temporal reasoning for referring video object segmentation through **Multi-scale Temporal Aggregation** and **Multi-object Temporal Interaction**, highlighting temporal correspondence alongside cross-modal grounding [2305.16318]. “GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attention” replaces pairwise multimodal attention with a volumetric score based on the Gram determinant, explicitly targeting any-order modality interactions rather than candidate ranking in a fixed HRI scene [2606.06249]. In a different application domain, “X2I: Seamless Integration of Multimodal Understanding into Diffusion Transformer via Attention Distillation” uses a lightweight bridge and attention distillation to inject multimodal understanding into a Diffusion Transformer, again emphasizing multimodal interfacing rather than direct object selection [2503.06134].

Against that background, MM-ITF is best characterized as a task-specific inter-modality transformer for deictic target estimation. Its distinguishing features are the exclusive use of **monocular RGB data**, the reduction of the problem to scoring candidate hand-object pairs, and the explicit use of a relationship token that encodes angular alignment. It does not aim at universal multimodal representation learning, temporal multimodal segmentation, arbitrary-order modality geometry, or multimodal image generation. Instead, it operationalizes multimodal interaction at the level required by a specific HRI decision problem: determining which object a human intends to indicate.

Source: https://www.emergentmind.com/topics/multi-modality-inter-transformer-mm-itf