MMIPN: Multimodal CNN for Grasp Intention

• The paper introduces a deterministic, geometry-focused multimodal CNN that estimates 3D grasp targets by integrating gaze, visual, and kinematic data.
• It achieves a low MAE of 15.2 mm when all sensory modalities are used, highlighting the pivotal role of gaze in reducing perceptual ambiguity.
• Integration within the MIRAGE framework significantly improves grasp success rates and reduces grasp time, validating its utility for VR teleoperation.

Multimodal-CNN-Based Human Intention Perception Network (MMIPN) denotes a multimodal, multi-branch CNN introduced within the MIRAGE shared-control framework for VR-based multi-object teleoperation. In that setting, MMIPN processes an observing image, recent robot motion, candidate object positions, and gaze data to estimate the human’s intended grasp position in $\mathbb{R}^3$, thereby addressing perceptual ambiguities and depth errors that arise under a single-perspective VR view without haptic feedback (Sun et al., 2 Sep 2025). More broadly, the term identifies a design pattern in which intention perception is treated as multimodal inference over heterogeneous sensory streams, with CNN-based encoders serving as the front end for intention-sensitive control.

1. Conceptual scope and intention representation

In MIRAGE, human intention is defined as “the object and its position that humans intend to grasp.” MMIPN therefore does not output a symbolic class label or a sentence; it outputs a grasp target position $\boldsymbol{o}_t \in \mathbb{R}^3$, which is then used directly by the grasp planner (Sun et al., 2 Sep 2025). This design places MMIPN in a specific intention-perception regime: immediate target disambiguation for shared-control teleoperation.

Other recent systems operationalize intention differently. MoGIC represents intention explicitly as discrete natural-language tokens generated autoregressively from partial motion and context (Shi et al., 3 Oct 2025). Bayesian Intention in tabletop HRC defines intent as the target object $T \in \mathbf{O}$ and estimates $P_t(T_{o_i}\mid \mathbf{E}_t)$ from head orientation, hand orientation, and hand motion (Hernandez-Cruz et al., 2024). The action-probability framework of “Inferring Human Intentions from Predicted Action Probabilities” treats intention as a latent discrete label inferred from sequences of predicted next-action distributions (Shi et al., 2023). MINT-RVAE formulates intent prediction as a frame-level binary label $y_{k,j}\in\{0,1\}$ indicating whether a human intends to interact with a robot (Mohsen et al., 26 Sep 2025).

This comparison suggests that MMIPN is best understood as a deterministic, geometry-centered instantiation of intention perception: its output space is continuous and action-proximal, whereas neighboring approaches often use textual, categorical, or posterior-distribution formulations.

2. Role within the MIRAGE shared-control framework

MIRAGE decomposes teleoperation into two control phases. During the movement phase, direct shared control maps controller motion into end-effector motion according to

$$\bm{x}_r(t) = \bm{x}_r(t_0) + \bm{\aleph} \int_{t_0}^{t_s} k_m T^r_h \dot{\bm{x}}_h \, dt,$$

where $\bm{\aleph}\in\{0,1\}$ is the teleoperation switch, $k_m$ is a scaling factor, and $T^r_h$ maps the local controller frame to the robot base frame (Sun et al., 2 Sep 2025). In this phase, MIRAGE may activate a Virtual Admittance (VA) module based on artificial potential fields to reshape the trajectory.

MMIPN operates only in the grasping phase. When the operator presses the “A” button, multimodal data around the decision instant are collected, MMIPN predicts the 3D grasp target position $\boldsymbol{o}_t$, and a grasp trajectory is autonomously planned and executed (Sun et al., 2 Sep 2025). The integration with VA is sequential rather than joint: VA improves coarse approach during movement, whereas MMIPN resolves final object and grasp-point ambiguity at grasp time.

The motivating failure mode is explicit. In multi-object VR scenes, end-effector pose alone is insufficient to infer which object the operator intends to grasp, and depth misperception makes “closest object to gripper” heuristics unreliable. MMIPN addresses this by combining four modalities that provide complementary evidence: gaze indicates which object is being visually attended, end-effector motion reflects how the user is steering and approaching, and environmental context constrains feasible targets geometrically (Sun et al., 2 Sep 2025).

3. Architecture and multimodal fusion

MMIPN is a multi-branch CNN with late fusion. Its multimodal input over a temporal window is

$\boldsymbol{o}_t \in \mathbb{R}^3$0

where $\boldsymbol{o}_t \in \mathbb{R}^3$1 is the observing image, $\boldsymbol{o}_t \in \mathbb{R}^3$2 the recent end-effector trajectory, $\boldsymbol{o}_t \in \mathbb{R}^3$3 the candidate object positions, and $\boldsymbol{o}_t \in \mathbb{R}^3$4 the gaze sequence (Sun et al., 2 Sep 2025).

The modality shapes are specified as follows. The image modality is

$\boldsymbol{o}_t \in \mathbb{R}^3$5

with $\boldsymbol{o}_t \in \mathbb{R}^3$6. Object positions are

$\boldsymbol{o}_t \in \mathbb{R}^3$7

where $\boldsymbol{o}_t \in \mathbb{R}^3$8 is the number of candidate objects; in the experiments, $\boldsymbol{o}_t \in \mathbb{R}^3$9 cubes. The robot action history is

$T \in \mathbf{O}$0

and the gaze input is

$T \in \mathbf{O}$1

with $T \in \mathbf{O}$2 and 12-D binocular 6-DoF eye-tracking data per time step (Sun et al., 2 Sep 2025).

Each modality is processed by a dedicated CNN sub-network $T \in \mathbf{O}$3, producing feature vectors

$T \in \mathbf{O}$4

The paper describes these as “2D or 3D shaped single-modal CNN structure”: a standard 2D CNN for the image branch, temporal CNNs for robot pose and gaze sequences, and a small 1D CNN or MLP-style CNN for object positions (Sun et al., 2 Sep 2025).

Fusion is late and purely feature-based. The modality features are concatenated,

$T \in \mathbf{O}$5

projected by a fully connected layer,

$T \in \mathbf{O}$6

and mapped by a final regression head to the grasp target,

$T \in \mathbf{O}$7

where $T \in \mathbf{O}$8 (Sun et al., 2 Sep 2025).

This architecture differs from multimodal intention systems that fuse modalities through posterior combination or token-level attention. A plausible implication is that MMIPN prioritizes short-horizon geometric disambiguation over explicit uncertainty modeling or language-like compositionality.

4. Data acquisition, supervision, and optimization

MMIPN is trained on multimodal samples collected in the full MIRAGE VR teleoperation system. Three subjects, distinct from the sixteen participants in the main user study, collected 125 samples each, yielding 375 multimodal samples (Sun et al., 2 Sep 2025). For each sample, the system records the current first-person image, the gaze data for the first three frames before grasp, the end-effector positions for the same three frames, the 3D positions of all available objects in the field of view, and the ground-truth target object position $T \in \mathbf{O}$9.

Training is posed as point regression. The primary loss is Mean Absolute Error: $P_t(T_{o_i}\mid \mathbf{E}_t)$0 Evaluation also reports Mean Squared Error

$P_t(T_{o_i}\mid \mathbf{E}_t)$1

and Mean Absolute Percentage Error (MAPE) (Sun et al., 2 Sep 2025).

Optimization uses Adam with learning rate $P_t(T_{o_i}\mid \mathbf{E}_t)$2, batch size 8, and 100 epochs. The dataset is divided into training and testing subsets using a “usual ratio,” and validation is monitored on the test set during training to prevent overfitting (Sun et al., 2 Sep 2025). At runtime, MMIPN requires $P_t(T_{o_i}\mid \mathbf{E}_t)$3 ms to infer a grasp target on the reported RTX 4060 laptop hardware.

Relative to broader intention-perception pipelines, this supervision regime is unusually direct. There is no intermediate ontology of grasp goals, no posterior over discrete candidates, and no auxiliary captioning or action-prediction loss. This suggests that MMIPN is optimized for immediate control utility rather than explanatory or probabilistic interpretability.

5. Empirical findings and modality contribution

The MMIPN ablation results identify gaze as the dominant modality. With all four modalities present, MMIPN achieves $P_t(T_{o_i}\mid \mathbf{E}_t)$4 mm, $P_t(T_{o_i}\mid \mathbf{E}_t)$5 mm$P_t(T_{o_i}\mid \mathbf{E}_t)$6, and $P_t(T_{o_i}\mid \mathbf{E}_t)$7 (Sun et al., 2 Sep 2025). Removing gaze causes a catastrophic degradation to $P_t(T_{o_i}\mid \mathbf{E}_t)$8 mm, $P_t(T_{o_i}\mid \mathbf{E}_t)$9 mm$y_{k,j}\in\{0,1\}$0, and $y_{k,j}\in\{0,1\}$1. Removing robot action yields $y_{k,j}\in\{0,1\}$2 mm; removing object positions yields $y_{k,j}\in\{0,1\}$3 mm; removing the observing image yields $y_{k,j}\in\{0,1\}$4 mm (Sun et al., 2 Sep 2025).

Variant	Removed modality	MAE (mm)
MMIPN	none	15.2
MMIPN-1	gaze	442.8
MMIPN-2	robot’s action	30.6
MMIPN-3	object positions	21.2
MMIPN-4	observing image	20.8

These numbers support a clear hierarchy of informativeness: gaze is crucial, robot motion is second most important, and scene image and object positions provide additional but less dominant gains. The paper explicitly concludes that “gaze data emerged as the most crucial input modality” (Sun et al., 2 Sep 2025).

The user study further connects MMIPN’s regression accuracy to teleoperation behavior. Across sixteen participants and forty trials per condition, MMIPN significantly increased successful grasp count ($y_{k,j}\in\{0,1\}$5), grasp success rate ($y_{k,j}\in\{0,1\}$6), and Perfect Blocks ($y_{k,j}\in\{0,1\}$7), and significantly reduced Bad Blocks ($y_{k,j}\in\{0,1\}$8) (Sun et al., 2 Sep 2025). MMIPN also reduced mean grasp time ($y_{k,j}\in\{0,1\}$9) and significantly increased presence ($$\bm{x}_r(t) = \bm{x}_r(t_0) + \bm{\aleph} \int_{t_0}^{t_s} k_m T^r_h \dot{\bm{x}}_h \, dt,$$0). By contrast, VA had no significant effect on grasp-success metrics, and MMIPN had no significant effect on movement metrics, indicating a functional separation between target disambiguation and path shaping (Sun et al., 2 Sep 2025).

A few participants reported confusion when MMIPN’s predicted position diverged from their intended object. This observation is important because the deployed system has no explicit confidence estimate or rejection mechanism; the predicted point is directly handed to the planner.

6. Relation to adjacent intention-perception architectures

MMIPN belongs to a broader family of multimodal intention systems but occupies a distinct architectural niche. In MoGIC, intention understanding and multimodal motion synthesis are jointly trained through a shared latent representation produced by a Conditional Masked Transformer, with mixture-of-attention and adaptive LayerNorm modulation used to align motion, text, and vision (Shi et al., 3 Oct 2025). That formulation is explicitly “understand-and-generate”: intention is a natural-language sentence, and motion generation is an auxiliary objective that improves intention understanding. MMIPN, by contrast, uses late fusion and direct grasp-point regression, without a generative branch or discrete intention language.

Bayesian intention models define a different axis of comparison. The framework of “Inferring Human Intentions from Predicted Action Probabilities” separates a DNN front end for next-action prediction from an MCMC-based Bayesian back end that infers a posterior over intentions and provides credible intervals (Shi et al., 2023). “Bayesian Intention for Enhanced Human Robot Collaboration” uses a Bayesian Network over head orientation, hand orientation, and hand motion to estimate target-object intent in $$\bm{x}_r(t) = \bm{x}_r(t_0) + \bm{\aleph} \int_{t_0}^{t_s} k_m T^r_h \dot{\bm{x}}_h \, dt,$$1 ms, with reported gains of 36% in precision, 60% in F1 Score, and 85% in accuracy over its best baseline (Hernandez-Cruz et al., 2024). These methods foreground uncertainty, causal structure, and posterior reasoning. MMIPN instead foregrounds short-horizon multimodal regression.

Gaze-centered intention modeling also links MMIPN to other recent research. SIF3D treats gaze as human intention in multimodal 3D motion forecasting, using gaze, scene, and motion in a ternary intention-aware attention mechanism (Lou et al., 2024). GazeVLA likewise models intention through gaze, representing it as a 2D pixel coordinate that precedes manipulation actions and serves as a cross-embodiment bridge between human data and robot control (Li et al., 24 Apr 2026). MIRAGE’s ablation results are consistent with that broader pattern: the gaze stream is the decisive modality for inferring short-term grasp intention (Sun et al., 2 Sep 2025).

At the same time, not all multimodal intention networks are grasp-point regressors. MINT-RVAE predicts frame-level interaction intent from RGB-only pose and emotion and reports AUROC $$\bm{x}_r(t) = \bm{x}_r(t_0) + \bm{\aleph} \int_{t_0}^{t_s} k_m T^r_h \dot{\bm{x}}_h \, dt,$$2 (Mohsen et al., 26 Sep 2025); BMCLOP focuses on opinion-pool fusion and multimodal confidence learning across gaze, speech, and gestures (Zhao et al., 2024); TAMFormer addresses pedestrian early intention prediction using a multi-modal transformer with learned temporal masks (Osman et al., 2022). This suggests that MMIPN is best viewed as one specialized member of a larger design space defined by three orthogonal choices: output space, temporal horizon, and uncertainty treatment.

7. Limitations and prospective development

The principal limitations are explicit. The training set comprises only 375 samples from 3 subjects; the objects are simple colored cubes; scenes are controlled; and generalization to more complex arrangements or object geometries is not tested (Sun et al., 2 Sep 2025). The system is also VR-specific, so real-world deployment may introduce additional noise sources, including lighting variation, camera calibration issues, and richer background clutter. Inference adds approximately 250 ms once per grasp, and some participants reported mild discomfort that the paper associates with rendering load on the laptop hardware (Sun et al., 2 Sep 2025).

A further limitation is representational rather than statistical: MMIPN always outputs a point. There is no confidence score, posterior over candidate targets, or rejection option when the multimodal evidence is ambiguous. This suggests a plausible development path toward hybrid systems in which MMIPN-style multimodal CNN front ends are coupled to confidence-learning or Bayesian reasoning layers. BMCLOP offers one such route through log-opinion-pool fusion with learned modality confidences (Zhao et al., 2024); the Bayesian intention frameworks offer another through posterior inference and credible intervals (Shi et al., 2023, Hernandez-Cruz et al., 2024).

The future directions named for MIRAGE are consistent with that trajectory: more participants, more complex tasks, multi-type or stacked objects, dynamic environments, additional cues such as speech commands or EMG/EEG, real-world teleoperation with RGB-D sensing, online adaptation and personalization, and explicit transparency mechanisms for visualizing inferred target objects (Sun et al., 2 Sep 2025). Taken together, these directions imply that the current MMIPN should be understood less as a finalized general-purpose intention architecture than as a compact, late-fusion benchmark module for immediate VR grasp disambiguation.