Rethinking Video Human-Object Interaction: Set Prediction over Time for Unified Detection and Anticipation

Abstract: Video-based human-object interaction (HOI) understanding requires both detecting ongoing interactions and anticipating their future evolution. However, existing methods usually treat anticipation as a downstream forecasting task built on externally constructed human-object pairs, limiting joint reasoning between detection and prediction. In addition, sparse keyframe annotations in current benchmarks can temporally misalign nominal future labels from actual future dynamics, reducing the reliability of anticipation evaluation. To address these issues, we introduce DETAnt-HOI, a temporally corrected benchmark derived from VidHOI and Action Genome for more faithful multi-horizon evaluation, and HOI-DA, a pair-centric framework that jointly performs subject-object localization, present HOI detection, and future anticipation by modeling future interactions as residual transitions from current pair states. Experiments show consistent improvements in both detection and anticipation, with larger gains at longer horizons. Our results highlight that anticipation is most effective when learned jointly with detection as a structural constraint on pair-level video representation learning. Benchmark and code will be publicly available.

• The paper introduces HOI-DA, a unified pair-centric framework for simultaneous video human-object interaction detection and future anticipation.
• It formulates anticipation as residual transitions over detected states using temporal summarization and task-specific decoders.
• Experimental results show significant mAP gains on benchmarks, validating the model’s robust forecasting and persistent pair tracking.

Unified Detection and Anticipation of Video Human-Object Interaction via Pair-Centric Set Prediction

Introduction

"Rethinking Video Human-Object Interaction: Set Prediction over Time for Unified Detection and Anticipation" (2604.10397) addresses the fundamental challenge of jointly detecting ongoing human-object interactions (HOIs) in videos and anticipating their temporal evolution. Prior art predominantly treats detection and anticipation as decoupled tasks, relying on two-stage pipelines that first construct human-object pairs exogenously and then forecast interactions as downstream classification problems. This paper proposes HOI-DA, a unified pair-centric architecture that couples detection and multi-horizon anticipation structurally within the same representation, arguing that anticipation should be formulated as a residual transition over the grounded present interaction state.

Related Work

Video HOI presents additional complexity compared to image-level counterparts due to the need for temporally persistent subject-object association. Conventional methods either enumerate pair candidates per frame and perform post-hoc interaction reasoning, or aggregate tubelet-level features for context, often failing to enforce pair identity as a first-class object. Previous anticipation efforts, especially in egocentric video, hinge on predicting imminent actions but mostly lack fine-grained pair-centric grounding. Large foundation models and vision-language priors (e.g., CLIP, RLIPv2, diffusion models) have been leveraged for open-vocabulary HOI in images, but have not been architecturally constrained for anticipation under causal and temporally coherent evaluation in videos. The proposed HOI-DA framework distinguishes itself by maintaining pair identity and modeling future dynamics as structured departures from present pair semantics.

Methodology

Pair-Centric Representation and Unified Decoder

HOI-DA instantiates persistent human-object pair slots across the observed video clip, eschewing explicit external tracking. The model builds a shared spatio-temporal visual memory via a backbone and Transformer encoder. Pair hypotheses are indexed in slot space and updated by cross-attention, yielding a trajectory of pair states over the clip.

Present interaction detection is grounded with dedicated task embeddings and slot-wise decoding at the observation boundary, producing spatially localized and semantically labeled HOIs. Future anticipation is formulated not as independent forecasts but as horizon-specific residual transitions over the detected interaction state. Horizon anchors attend over the temporal trajectory to construct history-aware anticipation queries. The anticipation decoder predicts future verb labels at multiple explicit time offsets, structurally coupling forecasting and detection.

Figure 1: Architecture overview delineating shared spatio-temporal memory, unified decoding for detection and anticipation, language-guided semantic regularization, and dual residual orthogonality constraints.

Figure 2: The Temporal Summary Module aggregates observed pair trajectories with learnable horizon anchors to enable history-conditioned anticipation.

Language-Guided Semantic Regularization

To improve recognition under the long-tailed verb distribution and support robust anticipation, a pretrained RoBERTa-based text encoder provides object and verb prototypes. These semantic embeddings serve as auxiliary classification heads and inject relational structure into both present and future verb prediction, supporting compositional generalization without future label leakage.

Dual Orthogonality Regularization

Joint detection/anticipation can cause horizon collapse or degenerate copying of present state. Two geometric losses are introduced: task orthogonality enforces that anticipation residuals encode what will change, not what is, by requiring orthogonality to the detected embedding; horizon orthogonality ensures diversity in horizon-specific residuals, avoiding temporal mode collapse. These constraints yield explicit factorization between current grounding and future evolution in representation space.

Training and Loss Formulation

Supervision is unified via bipartite matching, aligning pair slots across detection and anticipation branches. A focal-style loss penalizes multi-label verb prediction, and the anticipation loss is scheduled to emphasize short-term forecasting. The final objective combines detection, anticipation, and both orthogonality regularizers, with auxiliary losses applied to intermediate decoder stages.

DETAnt-HOI: Temporally Corrected Benchmark

Evaluation protocols in VidHOI and Action Genome are confounded by sparse keyframe annotations, leading to temporal gaps between observed and anticipated interaction labels. DETAnt-HOI remedies this by supplementing non-interactive frames (from VidOR in VidHOI) and reconstructing clips to enforce temporal continuity, valid pair tracking, and consistent horizon coverage. The split maintains original train/test partitions but corrects clip construction and alignment, ensuring that measured anticipation genuinely reflects future dynamics.

Figure 3: Analysis of temporal discontinuities in VidHOI, highlighting prevalence and duration of non-interactive keyframe gaps disrupting anticipation evaluation.

Experimental Results

HOI-DA markedly outperforms prior state-of-the-art baselines (e.g., Gaze-Tran, STTran) on both VidHOI and Action Genome components of DETAnt-HOI, in present-time detection and all anticipation horizons. The margin grows with horizon length: on VidHOI, HOI-DA achieves 16.27 mAP on detection, outperforming Gaze-Tran by +5.87, and reaches 18.73 mAP at the longest anticipation horizon ($h=7$), a gain of +8.59. Recall@k metrics confirm stronger ranking of plausible future HOIs. Ablation studies demonstrate that structured coupling via temporal summarization, language regularization, and orthogonality constraints are essential; naive fusion or single-decoder designs degrade both detection and anticipation.

Qualitative Results and Analysis

Qualitative visualizations exhibit temporally persistent pair tracking: HOI-DA maintains human-object pair hypotheses robustly under occlusion and camera motion. In joint detection and anticipation tasks, primary interactions evolve coherently across horizons while transient relations (e.g., watch) appear/disappear in alignment with video context. Attention heatmaps show task-specific specialization: present detection focuses on contact zones, anticipation expands to gaze and environmental cues.

Figure 4: Unified present-time detection and multi-horizon future anticipation, maintaining consistent pair identity and temporally coherent HOI predictions.

Figure 5: Robust pair tracking under occlusion and camera motion; persistent slot queries maintain pair identity, validated by supplementary annotated frames.

Figure 6: Examples across diverse scenarios; HOI-DA accurately models temporal transitions and interaction evolution.

Figure 7: Cross-attention heatmaps at localization, detection, and anticipation decoder stages, illustrating progression from instance features to broader context.

Implications and Future Directions

The structural coupling of detection and anticipation suggests that pair-level representations must encode temporally transferable semantics. Practically, this advances anticipatory systems for collaborative robotics, proactive surveillance, and intent forecasting in human-centered scenes. Theoretically, HOI-DA motivates further study of residual modeling, causal vision-language grounding, and explicit architectural constraints in temporally coherent reasoning. Future work may extend to cross-modal anticipation (text or audio context), reinforcement learning for active anticipation, and scaling to open-set verbs via prompt-based LLMs.

Conclusion

This work establishes anticipation as a structural constraint in video HOI by jointly optimizing detection and prediction within a unified pair-centric representation space, supported by temporally corrected benchmarks and explicit regularization. Empirical and qualitative results demonstrate persistent pair identity, robust forecasting, and horizon stability, fundamentally reframing video HOI anticipation from post-hoc classification to temporally consistent structured reasoning.