MINT-RVAE for HRI Intent Prediction

• The paper introduces MINT-RVAE, a multimodal recurrent variational autoencoder that generates synthetic sequences to rebalance imbalanced HRI training data.
• MINT-RVAE fuses 2D human pose and facial emotion cues extracted from RGB video using YOLOv8-pose and DeepFace within a GRU-based encoder–decoder architecture.
• Experimental results demonstrate that synthetic augmentation via MINT-RVAE significantly boosts frame-level intent prediction, especially in out-of-sample and multi-person scenarios.

Searching arXiv for the specified paper and directly related HRI intent-prediction work. MINT-RVAE is a multimodal recurrent variational autoencoder introduced for human–robot interaction (HRI) intent prediction from RGB-only video, specifically to generate synthetic sequences that rebalance highly imbalanced training data while preserving temporal and multimodal coherence. In the associated pipeline, a monocular RGB camera is used to extract 2D human pose and facial emotion cues, and discriminative sequence backbones then predict at frame level whether a human intends to interact with a robot arm. The method is defined by two linked contributions: frame-accurate onset prediction of interaction intent, and a generative augmentation mechanism that improves generalization under severe class imbalance, including out-of-sample multi-person settings (Mohsen et al., 26 Sep 2025).

1. Problem formulation and scope

The target problem is detecting when a human intends to interact with a robot, particularly in service or receptionist-type settings such as libraries, corridors, and shopping centers. The central requirement is temporal precision: the system is designed to detect the onset of interaction intent at frame level rather than only deciding whether an entire temporal window is interactive or non-interactive (Mohsen et al., 26 Sep 2025).

The paper situates this requirement against prior practice in HRI intent prediction. Existing work is described as relying largely on multimodal sensing suites, including RGB-D cameras, gaze tracking, microphones, or motion capture, and as operating primarily with sequence-level labels in which all frames within a window share the same interaction label. It also identifies two related limitations: post-hoc action segmentation that labels interactions retrospectively, and methods that output per-frame probabilities while training and evaluating from sequence-level labels replicated across frames rather than frame-accurate onset annotations. The argument is that such settings can obscure whether a system detects intent early enough for practical robotic response (Mohsen et al., 26 Sep 2025).

A further challenge is class imbalance. Real HRI deployments contain many more non-interaction intervals than interaction intervals, and even within interactive sequences, intent-positive frames are much fewer than no-intent frames. The paper cites examples such as 112 / 4245 interacted versus non-interacted trajectories in PAR-D and engagement rates around 3.6% in malls, and reports that in its own dataset only about 30–38% of frames are labeled as intent in each environment. This motivates a sequence-aware synthetic augmentation mechanism rather than naive undersampling or SMOTE-style oversampling, which are described as not sequence-aware and liable to break temporal and multimodal coherence (Mohsen et al., 26 Sep 2025).

2. RGB-only multi-cue pipeline

The broader system uses only a monocular RGB camera mounted on an Elephant Robotics MyCobot 320 arm. The stated significance is reduced hardware complexity and cost, contrasted in the paper with a depth sensor costing approximately $350 versus a $10 webcam. Within this RGB-only setting, the pipeline extracts two classes of cues from each frame: 2D body pose via YOLOv8-pose, and face detection plus emotion probabilities via YOLOv8 and DeepFace (Mohsen et al., 26 Sep 2025).

The dataset is represented as

$$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$

where $f_{k,j}$ denotes the frame feature vector and $y_{k,j}\in\{0,1\}$ is the binary frame-level label. Each feature vector is the concatenation of pose and emotion features: $$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$ This formalization underlies both the discriminative intent predictors and the generative MINT-RVAE model (Mohsen et al., 26 Sep 2025).

For pose, YOLOv8-pose provides $M$ keypoints with 2D coordinates $(x_m,y_m)$, detector confidences $c_m$, and a person bounding box $b_{k,j}=(x_{\min},y_{\min},w,h)$. To obtain scale- and translation-invariant coordinates, the keypoints are normalized relative to the bounding box and arranged as

$$f^{\text{pose}}_{k,j}\in\mathbb{R}^{3M},\qquad f^{\text{pose}}_{k,j}=[\,\bar{x}_1,\bar{y}_1,c_1,\dots,\bar{x}_M,\bar{y}_M,c_M\,].$$

The normalized coordinates are then z-normalized using the training-set mean and standard deviation, while the confidence values are left unchanged (Mohsen et al., 26 Sep 2025).

For emotion, the face crop is passed to DeepFace, which returns a 7-dimensional emotion distribution over standard emotions: $$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$0 During MINT-RVAE training, the frame intent label is appended to these pose and emotion features, yielding a joint multimodal representation that includes both behavioral cues and their associated intent labels (Mohsen et al., 26 Sep 2025).

3. Model architecture

MINT-RVAE, expanded as Multimodal INTention Recurrent Variational AutoEncoder, is described as a multimodal recurrent variational autoencoder designed specifically to generate synthetic HRI sequences. The model learns a joint latent representation over pose sequences, emotion sequences, and intent labels, using a GRU-based encoder–decoder backbone so that generated samples retain temporal coherence and multimodal consistency (Mohsen et al., 26 Sep 2025).

The model operates on fixed sequence windows of length $$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$1 frames: $$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$2 with

$$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$3

The dimensionality is $$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$4, decomposed into 51 dimensions for pose, 7 for emotion, and 1 for the binary label (Mohsen et al., 26 Sep 2025).

The encoder $$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$5 begins with a per-frame MLP that maps each $$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$6 through layers $$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$7, each consisting of linear transform, batch normalization, ReLU activation, and dropout. The resulting sequence is processed by a GRU encoder to capture temporal dependencies. Two linear layers then produce the parameters of the Gaussian variational posterior,

$$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$8

and latent samples are drawn with the reparameterization trick,

$$D=\{(f_{k,j},y_{k,j})\mid j=1,\dots,S;\; k=1,\dots,N_j\},$$9

The latent space is 32-dimensional (Mohsen et al., 26 Sep 2025).

The decoder $f_{k,j}$0 is autoregressive. A GRU decoder receives the latent vector and a decoded or teacher-forced input at each time step, and its hidden state is passed through an output MLP back to the multimodal observation space: $f_{k,j}$1 The reconstructed pose coordinates are produced by a linear output layer, the emotion distribution by a softmax over 7 classes, and the interaction label by a sigmoid output $f_{k,j}$2. The training target is one-step-ahead prediction, so the decoder learns to predict the next frame given the past frames and latent code (Mohsen et al., 26 Sep 2025).

4. Synthetic sequence generation and data rebalancing

The primary role of MINT-RVAE is to generate synthetic minority-class sequences to rebalance the downstream intent-prediction dataset. The paper frames this as a response to the scarcity of positive intent frames and sequences, which otherwise biases classifiers toward the no-intent class and harms recall and generalization (Mohsen et al., 26 Sep 2025).

During training, each real sequence window $f_{k,j}$3 is split into an input sequence $f_{k,j}$4 and a target sequence $f_{k,j}$5. The encoder processes the first part and the decoder predicts one step ahead, using either teacher forcing or autoregressive feedback according to scheduled sampling: $f_{k,j}$6 where $f_{k,j}$7 starts at 1 and linearly decays to 0 during training. This design stabilizes optimization early while gradually aligning training with the autoregressive regime used at inference (Mohsen et al., 26 Sep 2025).

After training, synthetic generation proceeds by sampling a latent vector from the prior,

$f_{k,j}$8

and running the decoder in fully autoregressive mode, seeded with an initial frame or a learned start token. The resulting synthetic trajectories are generated directly in feature space rather than in pixel space and include pose, emotion, and label sequences. This feature-space generation is computationally efficient and directly compatible with the downstream sequence classifiers (Mohsen et al., 26 Sep 2025).

The paper evaluates realism with a discriminative test in which a small RNN attempts to separate real from synthetic sequences. The reported discriminative score is defined as

$f_{k,j}$9

where $y_{k,j}\in\{0,1\}$0 is test accuracy in the discrimination task, so lower $y_{k,j}\in\{0,1\}$1 indicates higher realism. MINT-RVAE yields $y_{k,j}\in\{0,1\}$2 and $y_{k,j}\in\{0,1\}$3, and the paper reports substantial overlap between real and synthetic embeddings in t-SNE visualizations. These generated sequences are then used to augment minority-class training data for GRU, LSTM, and Transformer intent detectors (Mohsen et al., 26 Sep 2025).

A plausible implication is that the model’s inclusion of labels inside the generated sequence, rather than treating labels as external metadata, helps preserve alignment between temporal dynamics and intent transitions. The paper does not state this as a formal ablation result, but it is consistent with the stated objective of learning joint temporal dynamics of pose, emotion, and labels (Mohsen et al., 26 Sep 2025).

5. Objective function and optimization strategy

MINT-RVAE is trained with a composite objective comprising pose reconstruction, emotion reconstruction, label reconstruction, and KL regularization: $y_{k,j}\in\{0,1\}$4 The associated hyperparameters $y_{k,j}\in\{0,1\}$5 balance the contribution of these terms (Mohsen et al., 26 Sep 2025).

Pose reconstruction uses a confidence-weighted Huber loss together with an auxiliary MSE term for confidence prediction: $y_{k,j}\in\{0,1\}$6 with $y_{k,j}\in\{0,1\}$7 and Huber threshold $y_{k,j}\in\{0,1\}$8. The paper states that this emphasizes reliable joints, down-weights occluded or noisy joints, and improves robustness to outliers such as motion blur or partial occlusion (Mohsen et al., 26 Sep 2025).

Emotion reconstruction is based on KL divergence between true and predicted categorical distributions,

$y_{k,j}\in\{0,1\}$9

while label reconstruction uses binary cross-entropy,

$$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$0

The variational regularizer uses a Gaussian prior $$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$1 and per-dimension KL terms

$$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$2

with free-bits clipping

$$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$3

This is explicitly intended to prevent posterior collapse by ensuring that the KL term does not become too small (Mohsen et al., 26 Sep 2025).

The KL weight is linearly warmed up over epochs according to

$$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$4

with $$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$5 and $$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$6. The training details reported are Adam with learning rate $$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$7, $$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$8 weight decay $$f_{k,j}=\big[f^{\text{pose}}_{k,j};\, f^{\text{emo}}_{k,j}\big].$$9, batch size 64, and 700 epochs. The empirically chosen loss weights are $M$0, $M$1, and $M$2, which the paper interprets as prioritizing pose reconstruction first, then emotion, then labels (Mohsen et al., 26 Sep 2025).

6. Frame-level intent onset prediction and dataset design

Although MINT-RVAE is generative, final intent prediction is performed by separate discriminative backbones: GRU, LSTM, and Transformer. These models are trained on pose and emotion features, together with labels, and produce per-frame probabilities of interaction intent (Mohsen et al., 26 Sep 2025).

The labeling scheme is frame-accurate. Participants carry a wireless presenter button and press it at the first moment they intend to interact with the robot; this timestamp is synchronized with the video to define the onset frame. Labels are then assigned so that $M$3 before onset and $M$4 at and after onset for some interval. This distinguishes the dataset from systems that rely on replicated sequence labels or retrospective segmentation and supports direct onset prediction (Mohsen et al., 26 Sep 2025).

At inference, backbones output a per-frame probability $M$5, and frame-level classification is thresholded by $M$6. For sequence-level evaluation, a 15-frame window is considered positive in the ground truth if at least $M$7 frames in the window are labeled intent; on the model side, a window is predicted positive if the intent probability exceeds $M$8 in at least 7 consecutive frames. The paper also reports median intent-probability trajectories aligned around onset time $M$9, showing that probabilities rise sharply around onset when MINT-RVAE augmentation is used (Mohsen et al., 26 Sep 2025).

The released dataset comprises 10 participants across three indoor environments:

Environment	Sequences	Frames / intent rate
Library (Env 1)	54	7,620 frames, 30.2% intent
Corridor (Env 2)	23	3,900 frames, 32.8% intent
Two-person scene (Env 3)	11	1,095 frames, 37.5% intent

The data are described as de-identified because only pose and emotion features are stored rather than raw videos. The dataset is openly released at https://tinyurl.com/2ymx6ahr. The paper states that this is likely the first public dataset with frame-level onset annotations of human intent to interact with a robot arm. Because this is phrased as “likely,” it is best understood as a qualified claim rather than a definitive historical assertion (Mohsen et al., 26 Sep 2025).

7. Empirical results, comparison, and limitations

The discriminative intent backbones include a single-layer GRU with linear head, a single-layer LSTM with linear head, and a lightweight Transformer encoder with positional embeddings, one encoder block, four heads, and a classification head. These are evaluated in pose-only, emotion-only, multimodal without augmentation, and multimodal with MINT-RVAE augmentation variants (Mohsen et al., 26 Sep 2025).

The evaluation protocol uses 5-fold stratified cross-validation over Env 1 and Env 2, plus a held-out out-of-sample evaluation on Env 3. Metrics are AUROC, macro-$(x_m,y_m)$0, and balanced accuracy, chosen because they are informative under class imbalance. In cross-validation on Env 1 and Env 2, multimodal models with MINT-RVAE consistently outperform pose-only and multimodal-without-augmentation baselines. For the Transformer with pose and emotion plus MINT-RVAE, the reported performance is frame AUROC $(x_m,y_m)$1 and sequence AUROC $(x_m,y_m)$2. The paper compares this with prior reported AUROC values of approximately 0.90 for Abbate et al. (2024) and 0.912 for Arreghini et al. (2024), while emphasizing that the present system uses RGB only and supports frame-level onset prediction (Mohsen et al., 26 Sep 2025).

The out-of-sample Env 3 results are more pronounced. For the Transformer, the model without augmentation achieves frame AUROC $(x_m,y_m)$3 and sequence AUROC $(x_m,y_m)$4, whereas the MINT-RVAE-augmented version reaches frame AUROC $(x_m,y_m)$5 and sequence AUROC $(x_m,y_m)$6. The paper interprets this as evidence that synthetic sequence generation improves generalization to more complex multi-person scenes not seen during training. It also reports strong gains for recurrent baselines, including LSTM + VAE sequence AUROC $(x_m,y_m)$7 versus $(x_m,y_m)$8 without augmentation (Mohsen et al., 26 Sep 2025).

Several qualitative conclusions are drawn from the ablations. Emotion-only models perform poorly, indicating that full-body dynamics are the primary cue for early intent detection, while emotion is useful when fused with pose. ROC and precision–recall analyses are described as showing that MINT-RVAE shifts operating curves upward and enables higher recall at similar precision, especially for the Transformer. This suggests that the synthetic augmentation strategy improves minority-class sensitivity without materially increasing false alarms (Mohsen et al., 26 Sep 2025).

The limitations are explicit. The approach depends on upstream perception quality from YOLOv8-pose and DeepFace, so errors in pose or emotion estimation propagate into both intent prediction and MINT-RVAE training. The dataset is limited to indoor public spaces, 10 participants, and a particular robot arm, so broader generalization across robots, environments, and interaction styles remains open. Emotion is also characterized as a secondary cue because emotion-only performance is weak. The paper identifies several future directions: integrating other nonverbal cues such as gaze and proxemics while remaining RGB-only or low-cost, extending MINT-RVAE to additional modalities and environments, exploring alternative generative models such as GANs or flows, and applying the pipeline beyond arm interaction to full-body collaborative tasks (Mohsen et al., 26 Sep 2025).