OnlineHOI: Causal Streaming HOI Modeling

• OnlineHOI is a causal formulation of streaming human–object interaction that relies only on current and past frames, ensuring strict real-time constraints.
• It defines two main tasks—HOI Generation and HOI Perception—evaluated on benchmarks like Core4D, OAKINK2, and HOI4D using metrics such as FID, RA, and F1@k.
• The framework employs a Mamba-based architecture with explicit memory mechanisms that outperform Transformer-based models in online settings and offer potential for robotics and open-vocabulary extensions.

OnlineHOI denotes a causal formulation of human–object interaction modeling in which, at time step $t$, only the current observation and history up to $t$ are available. In its precise 2025 usage, the term refers both to two tasks—Online HOI Generation and Online HOI Perception—and to a Mamba-based framework with explicit memory for streaming HOI motion synthesis and action understanding (Ji et al., 12 Sep 2025). In adjacent literature, the label has also been used more loosely for real-time HOI anticipation in robotics and for promptable open-vocabulary HOI systems whose label space can be extended at test time; these usages are related but not identical to the causal benchmark introduced by OnlineHOI (Mascaro et al., 2023, Sun et al., 29 Mar 2026, Yang et al., 2024).

1. Terminological scope and core formulation

OnlineHOI is centered on the distinction between offline and online HOI. In the offline setting, the model is given the entire interaction sequence before making predictions, so future frames are available during inference. In the online setting, by contrast, future information is strictly forbidden: at frame $t$, only frames $\{1,\dots,t\}$ may be used. The OnlineHOI paper argues that this distinction is not cosmetic; offline HOI models, especially Transformer-based ones, degrade substantially when forced into causal operation because their full-sequence self-attention is not designed for streaming inference (Ji et al., 12 Sep 2025).

Within that formulation, OnlineHOI separates two complementary tasks. Online HOI Generation asks the model to synthesize the current frame of a reactor’s motion from historical actor and object information only. Online HOI Perception asks the model to assign the current interaction label from streaming sensory input only. The former is evaluated on 3D motion datasets such as Core4D and OAKINK2, while the latter is evaluated on streaming 4D point cloud sequences from HOI4D (Ji et al., 12 Sep 2025).

A broader usage of the term exists in neighboring subfields. In collaborative robotics, HOI4ABOT treats online HOI as causal video-based detection and anticipation from a sliding window of past frames, with real-time robot deployment and multi-horizon forecasting (Mascaro et al., 2023). In open-vocabulary image HOI, SL-HOI and MP-HOI are described as compatible with “online / promptable / dynamic-query” systems because new interaction names or prompts can be added at test time without retraining, even though they do not define a streaming protocol (Sun et al., 29 Mar 2026, Yang et al., 2024). This suggests that “OnlineHOI” currently spans at least two axes: temporal causality and dynamic label-space extensibility. The explicit benchmarked meaning, however, is the causal streaming one introduced by OnlineHOI itself (Ji et al., 12 Sep 2025).

2. Task definitions, datasets, and evaluation protocols

The OnlineHOI formulation is task-specific and dataset-specific rather than purely conceptual. Its two benchmark tasks are summarized below.

Task	Input up to time $t$	Output at time $t$	Datasets / metrics
Online HOI Generation	Actor motion and object state history	Reactor motion	Core4D, OAKINK2; FID, DIV, RA, user study
Online HOI Perception	4D point cloud history $\Psi=\{(P_t,V_t)\}$	Current action label	HOI4D; Acc, Edit, F1@10/25/50

For online generation, the system predicts the reactor’s motion causally. In Core4D, the conditioning signals are the actor’s full-body motion and object geometry, and the target is the reactor’s full-body motion. In OAKINK2, the conditioning signals are the actor’s hand motion and object geometry, and the target is the reactor’s hand motion (Ji et al., 12 Sep 2025). The evaluation metrics are standard generative quality measures adapted to motion: FID for distributional fidelity, DIV for diversity, RA for semantic recognizability via an action recognizer trained on ground-truth motions, and human preference in user studies (Ji et al., 12 Sep 2025).

For online perception, the input is a streaming point-cloud sequence

$\Psi = \{(P_t, V_t) \mid t = 1,\dots,L\},$

where $P_t$ denotes the point cloud and $V_t$ the normals at frame $t$0. The output is a framewise hand-motion interaction category from the 19 HOI4D classes, under a strict causal constraint (Ji et al., 12 Sep 2025). The metrics are Acc, Edit, and segmental F1@k, where $t$1, so evaluation emphasizes not only framewise correctness but also temporally coherent segmentation (Ji et al., 12 Sep 2025).

This protocol differs materially from several adjacent formulations. HOI4ABOT also uses only past frames, but it focuses on video-based HOI detection and anticipation in VidHOI, with horizons $t$2 seconds and robot control integration (Mascaro et al., 2023). By contrast, OnlineHOI defines a unified causal setting spanning both generation and perception, and enforces that causality in both modeling and evaluation (Ji et al., 12 Sep 2025).

3. Mamba-based architecture and memory mechanism

The defining architectural choice in OnlineHOI is the replacement of global-attention temporal modeling with Mamba, a selective state-space model. In its basic discretized form, the hidden state update is

$t$3

so temporal processing is performed by a forward scan rather than by explicit attention over all previous tokens (Ji et al., 12 Sep 2025). This makes the backbone intrinsically causal and linear-time in sequence length, which the paper identifies as the key reason it outperforms Transformers in the online regime (Ji et al., 12 Sep 2025).

OnlineHOI-G

For generation, OnlineHOI-G is a diffusion-based model with a U-Net-like encoder–decoder in which both encoder and decoder are built from unidirectional spatial Mamba blocks. The forward diffusion process is written as

$t$4

and the model directly predicts motion from Gaussian noise under complex human–object conditions rather than using a purely non-causal denoising stack (Ji et al., 12 Sep 2025). A small self-attention block is inserted at the encoder end to integrate conditional information, but temporal propagation remains causal because the underlying sequence model is Mamba (Ji et al., 12 Sep 2025).

OnlineHOI-P

For perception, OnlineHOI-P combines a 4D point-cloud backbone with a Mamba temporal head. Its 4D convolution aggregates local spatiotemporal neighborhoods before the Mamba module scans the sequence forward. The paper gives the local aggregation at $t$5 as

$t$6

followed by temporal scanning with the Mamba state update (Ji et al., 12 Sep 2025). The result is a fully causal 4D perception stack.

Memory Augment Model

Mamba’s recurrent state is complemented by an explicit Memory Augment Model with three components: short-term memory $t$7, long-term memory $t$8, and enhanced memory $t$9 (Ji et al., 12 Sep 2025). The short-term memory is a fixed-size FIFO buffer over the most recent $t$0 frames. The long-term memory is constructed by repeatedly merging the most similar adjacent frames in $t$1, where similarity is computed by dot product,

$t$2

until the memory shrinks to capacity $t$3 (Ji et al., 12 Sep 2025). The enhanced memory is then

$t$4

with concatenation followed by max-pooling and fusion into the decoder state (Ji et al., 12 Sep 2025).

The paper’s ablations show that neither short-term nor long-term memory alone is optimal; the combined $t$5 performs best, and the best fusion strategy is concatenation with max-pooling rather than simple addition or max fusion (Ji et al., 12 Sep 2025).

4. Empirical performance and ablation findings

OnlineHOI reports state-of-the-art results on the Core4D and OAKINK2 online generation tasks and on the HOI4D online perception task (Ji et al., 12 Sep 2025). The central empirical claim is not merely that the proposed model is strong, but that the ordering of backbones flips between offline and online regimes.

Setting	Transformer-style baseline	Mamba-style baseline	Main observation
Offline perception (HOI4D)	Acc 71.2 / Edit 73.1 / F1@50 58.2	66.0 / 65.4 / 50.8	Transformer better offline
Online perception (HOI4D)	66.7 / 62.0 / 46.3	71.0 / 78.5 / 61.5	Mamba better online
Offline generation (Core4D)	FID 2.89 / RA 80.36	3.05 / 79.01	Transformer slightly better offline
Online generation (Core4D S2)	3.73 / 76.56	2.72 / 78.26	Mamba better online

On Core4D, OnlineHOI-G achieves FID $t$6, RA $t$7, and user-study preference $t$8 on S1, and FID $t$9, RA $\{1,\dots,t\}$0, and user preference $\{1,\dots,t\}$1 on S2, outperforming MDM and OMOMO under the online constraint (Ji et al., 12 Sep 2025). On OAKINK2, it reaches FID $\{1,\dots,t\}$2, DIV $\{1,\dots,t\}$3, and user preference $\{1,\dots,t\}$4, again improving over MDM and OMOMO (Ji et al., 12 Sep 2025). The qualitative comparisons emphasize more stable contact, less object penetration, and better collaborative semantics in both full-body and bimanual settings (Ji et al., 12 Sep 2025).

On HOI4D, OnlineHOI-P achieves $\{1,\dots,t\}$5 clips/s, Acc $\{1,\dots,t\}$6, Edit $\{1,\dots,t\}$7, F1@10 $\{1,\dots,t\}$8, F1@25 $\{1,\dots,t\}$9, and F1@50 $t$0, substantially exceeding PointNet++ and online-adapted P4Transformer baselines (Ji et al., 12 Sep 2025). The gap is especially pronounced in Edit and F1@50, indicating that the gain is not limited to local framewise classification but extends to temporally coherent segmentation (Ji et al., 12 Sep 2025).

Memory contributes measurable gains in both branches. In generation on Core4D S2, adding memory improves FID from $t$1 to $t$2 and RA from $t$3 to $t$4. In perception on HOI4D, it improves Acc from $t$5 to $t$6 and F1@50 from $t$7 to $t$8 (Ji et al., 12 Sep 2025). These deltas are modest compared with the Mamba-versus-Transformer regime shift, but they are consistent and validate the explicit memory design.

5. Relation to adjacent HOI paradigms

OnlineHOI occupies one pole of a wider landscape in which “online” can mean different things. The clearest neighboring usage is causal anticipation for robotics. HOI4ABOT uses only past observations over a video window, predicts current and future HOIs at $t$9, and integrates the predictions into a robot control stack via Dynamic Movement Primitives and Behavior Trees (Mascaro et al., 2023). In that sense, HOI4ABOT is an online HOI system, but it addresses anticipation and robot interaction rather than the Mamba-based generation/perception pair formalized by OnlineHOI (Mascaro et al., 2023, Ji et al., 12 Sep 2025).

A second neighboring usage is dynamic open-vocabulary HOI. SL-HOI is explicitly described as compatible with an online, query-driven HOI detection service because its classifier is text-driven, all DINOv3 parameters are frozen, and new interaction names can be embedded and scored at test time without retraining (Sun et al., 29 Mar 2026). MP-HOI generalizes this further with multi-modal prompts and a 20K-HOI training universe, so new textual prompts or visual exemplars can be added without redesigning the classifier (Yang et al., 2024). This suggests that some authors use “online” to denote runtime expansion of the label space, not temporal causality.

A third related direction is interactive HOI generation from weak controls. HunyuanVideo-HOMA is not online in the streaming sense, but it supports interactive offline specification through a human reference image, an object reference image, sparse pose control, object center trajectories, optional text, and optional audio (Huang et al., 10 Jun 2025). That line is closer to controllable generation than to causal inference, but it overlaps with OnlineHOI insofar as both reject full-sequence offline conditioning as the only meaningful operational regime (Huang et al., 10 Jun 2025, Ji et al., 12 Sep 2025).

A fourth neighboring direction is online update inside otherwise offline HOI systems. DIFFUSIONHOI introduces an online update strategy for relation embeddings during supervised HOI training, but not a streaming inference protocol (Li et al., 2024). The distinction is important: OnlineHOI’s “online” refers to causal test-time constraints, whereas DIFFUSIONHOI’s refers to representation refinement during training (Li et al., 2024).

A common misconception, therefore, is to treat all of these as the same problem. They are not. OnlineHOI in the strict sense is a causal streaming formulation. Open-vocabulary promptability, continual vocabulary expansion, and real-time robot anticipation are adjacent but non-identical problem settings.

6. Limitations and prospective directions

The limitations explicitly identified for OnlineHOI are mostly architectural and modal. The framework is demonstrated on 3D motion and 4D point cloud benchmarks, so extension to pure RGB or RGB-D online HOI would require new front ends (Ji et al., 12 Sep 2025). Although OnlineHOI-P reaches $t$0 clips/s on HOI4D, the 4D convolution plus Mamba plus memory stack is still heavier than single-frame baselines, so tight-latency deployment on embedded systems remains nontrivial (Ji et al., 12 Sep 2025). The paper also notes that object generalization is probed through seen-versus-unseen geometries in Core4D S2, but not through fully unconstrained open-vocabulary semantics (Ji et al., 12 Sep 2025).

The framework also remains largely low-level. It models motion and action labels, but not higher-level task semantics, language instructions, or explicit goal representations (Ji et al., 12 Sep 2025). This makes language-grounded extensions a natural next step. A plausible direction, suggested by adjacent work, is to combine OnlineHOI’s causal sequence modeling with promptable open-vocabulary heads such as those used in SL-HOI or MP-HOI, thereby unifying streaming causality with dynamic label-space expansion (Sun et al., 29 Mar 2026, Yang et al., 2024). Another plausible direction is integration with reasoning-centric open-world systems such as HOID-R1, where instruction-following and structured HOI outputs are emphasized, though that line is currently image-based and not optimized for streaming (Zhang et al., 15 Aug 2025).

Robotics provides a second obvious extension path. HOI4ABOT already demonstrates that online causal HOI anticipation can improve Human Idle Time, Robot Idle Time, Concurrent Activity, and Functional Delay in a real collaborative robot (Mascaro et al., 2023). OnlineHOI’s causal generation branch suggests a complementary use: instead of only classifying what will happen, a system could synthesize plausible future human–object motion trajectories online and feed them into downstream planners (Ji et al., 12 Sep 2025). This suggests a convergence between online perception, anticipation, and causal generation.

More broadly, the literature indicates that future OnlineHOI systems may need to integrate four ingredients that are currently dispersed across separate papers: causal temporal modeling, explicit memory, open-vocabulary prompting, and physically grounded generation or reconstruction (Ji et al., 12 Sep 2025, Zhao et al., 14 May 2026). OnlineHOI establishes the causal benchmark and a strong Mamba-based baseline; the remaining challenge is to combine that benchmark with richer semantics and broader world models without abandoning the strict online constraint.