---
title: 'OnlineHOI: Causal Streaming HOI Modeling'
url: https://www.emergentmind.com/topics/onlinehoi
type: topic
---

# OnlineHOI: Causal Streaming HOI Modeling

OnlineHOI denotes a causal formulation of human–object interaction modeling in which, at time step \(t\), only the current observation and history up to \(t\) are available. In its precise 2025 usage, the term refers both to two tasks—**Online HOI Generation** and **Online HOI Perception**—and to a Mamba-based framework with explicit memory for streaming HOI motion synthesis and action understanding [2509.12250]. In adjacent literature, the label has also been used more loosely for real-time HOI anticipation in robotics and for promptable open-vocabulary HOI systems whose label space can be extended at test time; these usages are related but not identical to the causal benchmark introduced by OnlineHOI [2309.16524] [2603.27500] [2406.07221].

## 1. Terminological scope and core formulation

OnlineHOI is centered on the distinction between **offline** and **online** HOI. In the offline setting, the model is given the entire interaction sequence before making predictions, so future frames are available during inference. In the online setting, by contrast, future information is strictly forbidden: at frame \(t\), only frames \(\{1,\dots,t\}\) may be used. The OnlineHOI paper argues that this distinction is not cosmetic; offline HOI models, especially Transformer-based ones, degrade substantially when forced into causal operation because their full-sequence self-attention is not designed for streaming inference [2509.12250].

Within that formulation, OnlineHOI separates two complementary tasks. **Online HOI Generation** asks the model to synthesize the current frame of a reactor’s motion from historical actor and object information only. **Online HOI Perception** asks the model to assign the current interaction label from streaming sensory input only. The former is evaluated on 3D motion datasets such as Core4D and OAKINK2, while the latter is evaluated on streaming 4D point cloud sequences from HOI4D [2509.12250].

A broader usage of the term exists in neighboring subfields. In collaborative robotics, HOI4ABOT treats online HOI as causal video-based detection and anticipation from a sliding window of past frames, with real-time robot deployment and multi-horizon forecasting [2309.16524]. In open-vocabulary image HOI, SL-HOI and MP-HOI are described as compatible with “online / promptable / dynamic-query” systems because new interaction names or prompts can be added at test time without retraining, even though they do not define a streaming protocol [2603.27500] [2406.07221]. This suggests that “OnlineHOI” currently spans at least two axes: **temporal causality** and **dynamic label-space extensibility**. The explicit benchmarked meaning, however, is the causal streaming one introduced by OnlineHOI itself [2509.12250].

## 2. Task definitions, datasets, and evaluation protocols

The OnlineHOI formulation is task-specific and dataset-specific rather than purely conceptual. Its two benchmark tasks are summarized below.

| Task | Input up to time \(t\) | Output at time \(t\) | Datasets / metrics |
|---|---|---|---|
| Online HOI Generation | Actor motion and object state history | Reactor motion | Core4D, OAKINK2; FID, DIV, RA, user study |
| Online HOI Perception | 4D point cloud history \(\Psi=\{(P_t,V_t)\}\) | Current action label | HOI4D; Acc, Edit, F1@10/25/50 |

For **online generation**, the system predicts the reactor’s motion causally. In Core4D, the conditioning signals are the actor’s full-body motion and object geometry, and the target is the reactor’s full-body motion. In OAKINK2, the conditioning signals are the actor’s hand motion and object geometry, and the target is the reactor’s hand motion [2509.12250]. The evaluation metrics are standard generative quality measures adapted to motion: **FID** for distributional fidelity, **DIV** for diversity, **RA** for semantic recognizability via an action recognizer trained on ground-truth motions, and human preference in user studies [2509.12250].

For **online perception**, the input is a streaming point-cloud sequence
\[
\Psi = \{(P_t, V_t) \mid t = 1,\dots,L\},
\]
where \(P_t\) denotes the point cloud and \(V_t\) the normals at frame \(t\). The output is a framewise hand-motion interaction category from the 19 HOI4D classes, under a strict causal constraint [2509.12250]. The metrics are **Acc**, **Edit**, and segmental **F1@k**, where \(k \in \{10,25,50\}\), so evaluation emphasizes not only framewise correctness but also temporally coherent segmentation [2509.12250].

This protocol differs materially from several adjacent formulations. HOI4ABOT also uses only past frames, but it focuses on video-based HOI detection and anticipation in VidHOI, with horizons \(\tau \in \{0,1,3,5\}\) seconds and robot control integration [2309.16524]. By contrast, OnlineHOI defines a unified causal setting spanning both generation and perception, and enforces that causality in both modeling and evaluation [2509.12250].

## 3. Mamba-based architecture and memory mechanism

The defining architectural choice in OnlineHOI is the replacement of global-attention temporal modeling with **Mamba**, a selective state-space model. In its basic discretized form, the hidden state update is
\[
h_{t+1} = \overline{\mathbf A} h_t + \overline{\mathbf B} x_t,\qquad
y_t = \mathbf C h_t,
\]
so temporal processing is performed by a forward scan rather than by explicit attention over all previous tokens [2509.12250]. This makes the backbone intrinsically causal and linear-time in sequence length, which the paper identifies as the key reason it outperforms Transformers in the online regime [2509.12250].

### OnlineHOI-G

For generation, OnlineHOI-G is a diffusion-based model with a U-Net-like encoder–decoder in which both encoder and decoder are built from unidirectional spatial Mamba blocks. The forward diffusion process is written as
\[
q(x_t \mid x_{t-1}) = \mathcal N(\sqrt{\alpha_t} x_{t-1},\, (1-\alpha_t)\mathbf I),
\]
and the model directly predicts motion from Gaussian noise under complex human–object conditions rather than using a purely non-causal denoising stack [2509.12250]. A small self-attention block is inserted at the encoder end to integrate conditional information, but temporal propagation remains causal because the underlying sequence model is Mamba [2509.12250].

### OnlineHOI-P

For perception, OnlineHOI-P combines a 4D point-cloud backbone with a Mamba temporal head. Its 4D convolution aggregates local spatiotemporal neighborhoods before the Mamba module scans the sequence forward. The paper gives the local aggregation at \((x,y,z,t)\) as
\[
\boldsymbol f'^{(x,y,z)}_t =
\sum_{\delta_t=-r_t}^{r_t}
\sum_{\|(\delta_x,\delta_y,\delta_z)\|\le r_s}
\bigl(\mathbf W_d \cdot (\delta_x,\delta_y,\delta_z,\delta_t)^\top\bigr)
\odot
\bigl(\mathbf W_f \cdot \boldsymbol f^{(x+\delta_x, y+\delta_y, z+\delta_z)}_{t+\delta_t}\bigr),
\]
followed by temporal scanning with the Mamba state update [2509.12250]. The result is a fully causal 4D perception stack.

### Memory Augment Model

Mamba’s recurrent state is complemented by an explicit **Memory Augment Model** with three components: short-term memory \(\mathbf M_S\), long-term memory \(\mathbf M_L\), and enhanced memory \(\mathbf M_E\) [2509.12250]. The short-term memory is a fixed-size FIFO buffer over the most recent \(S\) frames. The long-term memory is constructed by repeatedly merging the most similar adjacent frames in \(\mathbf M_S\), where similarity is computed by dot product,
\[
\mathrm{Sim}[t] = \mathrm{dot}(\mathrm{frame}_t,\mathrm{frame}_{t+1}),
\]
until the memory shrinks to capacity \(L\) [2509.12250]. The enhanced memory is then
\[
\mathbf M_E = [\mathbf M_S \mid \mathbf M_L],
\]
with concatenation followed by max-pooling and fusion into the decoder state [2509.12250].

The paper’s ablations show that neither short-term nor long-term memory alone is optimal; the combined \(\mathbf M_E\) performs best, and the best fusion strategy is concatenation with max-pooling rather than simple addition or max fusion [2509.12250].

## 4. Empirical performance and ablation findings

OnlineHOI reports state-of-the-art results on the Core4D and OAKINK2 online generation tasks and on the HOI4D online perception task [2509.12250]. The central empirical claim is not merely that the proposed model is strong, but that the **ordering of backbones flips between offline and online regimes**.

| Setting | Transformer-style baseline | Mamba-style baseline | Main observation |
|---|---:|---:|---|
| Offline perception (HOI4D) | Acc 71.2 / Edit 73.1 / F1@50 58.2 | 66.0 / 65.4 / 50.8 | Transformer better offline |
| Online perception (HOI4D) | 66.7 / 62.0 / 46.3 | 71.0 / 78.5 / 61.5 | Mamba better online |
| Offline generation (Core4D) | FID 2.89 / RA 80.36 | 3.05 / 79.01 | Transformer slightly better offline |
| Online generation (Core4D S2) | 3.73 / 76.56 | 2.72 / 78.26 | Mamba better online |

On **Core4D**, OnlineHOI-G achieves FID \(2.61\), RA \(79.52\), and user-study preference \(63.54\%\) on S1, and FID \(2.54\), RA \(79.16\), and user preference \(51.78\%\) on S2, outperforming MDM and OMOMO under the online constraint [2509.12250]. On **OAKINK2**, it reaches FID \(0.35\), DIV \(15.52\), and user preference \(54.12\%\), again improving over MDM and OMOMO [2509.12250]. The qualitative comparisons emphasize more stable contact, less object penetration, and better collaborative semantics in both full-body and bimanual settings [2509.12250].

On **HOI4D**, OnlineHOI-P achieves \(31.8\) clips/s, Acc \(71.2\), Edit \(79.3\), F1@10 \(78.6\), F1@25 \(74.1\), and F1@50 \(62.2\), substantially exceeding PointNet++ and online-adapted P4Transformer baselines [2509.12250]. The gap is especially pronounced in Edit and F1@50, indicating that the gain is not limited to local framewise classification but extends to temporally coherent segmentation [2509.12250].

Memory contributes measurable gains in both branches. In generation on Core4D S2, adding memory improves FID from \(2.72\) to \(2.54\) and RA from \(78.26\) to \(79.16\). In perception on HOI4D, it improves Acc from \(70.5\) to \(71.2\) and F1@50 from \(61.5\) to \(62.2\) [2509.12250]. These deltas are modest compared with the Mamba-versus-Transformer regime shift, but they are consistent and validate the explicit memory design.

## 5. Relation to adjacent HOI paradigms

OnlineHOI occupies one pole of a wider landscape in which “online” can mean different things. The clearest neighboring usage is **causal anticipation for robotics**. HOI4ABOT uses only past observations over a video window, predicts current and future HOIs at \(\tau \in \{0,1,3,5\}\), and integrates the predictions into a robot control stack via Dynamic Movement Primitives and Behavior Trees [2309.16524]. In that sense, HOI4ABOT is an online HOI system, but it addresses anticipation and robot interaction rather than the Mamba-based generation/perception pair formalized by OnlineHOI [2309.16524] [2509.12250].

A second neighboring usage is **dynamic open-vocabulary HOI**. SL-HOI is explicitly described as compatible with an online, query-driven HOI detection service because its classifier is text-driven, all DINOv3 parameters are frozen, and new interaction names can be embedded and scored at test time without retraining [2603.27500]. MP-HOI generalizes this further with multi-modal prompts and a 20K-HOI training universe, so new textual prompts or visual exemplars can be added without redesigning the classifier [2406.07221]. This suggests that some authors use “online” to denote **runtime expansion of the label space**, not temporal causality.

A third related direction is **interactive HOI generation from weak controls**. HunyuanVideo-HOMA is not online in the streaming sense, but it supports interactive offline specification through a human reference image, an object reference image, sparse pose control, object center trajectories, optional text, and optional audio [2506.08797]. That line is closer to controllable generation than to causal inference, but it overlaps with OnlineHOI insofar as both reject full-sequence offline conditioning as the only meaningful operational regime [2506.08797] [2509.12250].

A fourth neighboring direction is **online update inside otherwise offline HOI systems**. DIFFUSIONHOI introduces an online update strategy for relation embeddings during supervised HOI training, but not a streaming inference protocol [2410.20155]. The distinction is important: OnlineHOI’s “online” refers to causal test-time constraints, whereas DIFFUSIONHOI’s refers to representation refinement during training [2410.20155].

A common misconception, therefore, is to treat all of these as the same problem. They are not. OnlineHOI in the strict sense is a **causal streaming formulation**. Open-vocabulary promptability, continual vocabulary expansion, and real-time robot anticipation are adjacent but non-identical problem settings.

## 6. Limitations and prospective directions

The limitations explicitly identified for OnlineHOI are mostly architectural and modal. The framework is demonstrated on **3D motion** and **4D point cloud** benchmarks, so extension to pure RGB or RGB-D online HOI would require new front ends [2509.12250]. Although OnlineHOI-P reaches \(31.8\) clips/s on HOI4D, the 4D convolution plus Mamba plus memory stack is still heavier than single-frame baselines, so tight-latency deployment on embedded systems remains nontrivial [2509.12250]. The paper also notes that object generalization is probed through seen-versus-unseen geometries in Core4D S2, but not through fully unconstrained open-vocabulary semantics [2509.12250].

The framework also remains largely **low-level**. It models motion and action labels, but not higher-level task semantics, language instructions, or explicit goal representations [2509.12250]. This makes language-grounded extensions a natural next step. A plausible direction, suggested by adjacent work, is to combine OnlineHOI’s causal sequence modeling with promptable open-vocabulary heads such as those used in SL-HOI or MP-HOI, thereby unifying streaming causality with dynamic label-space expansion [2603.27500] [2406.07221]. Another plausible direction is integration with reasoning-centric open-world systems such as HOID-R1, where instruction-following and structured HOI outputs are emphasized, though that line is currently image-based and not optimized for streaming [2508.11350].

Robotics provides a second obvious extension path. HOI4ABOT already demonstrates that online causal HOI anticipation can improve Human Idle Time, Robot Idle Time, Concurrent Activity, and Functional Delay in a real collaborative robot [2309.16524]. OnlineHOI’s causal generation branch suggests a complementary use: instead of only classifying what will happen, a system could synthesize plausible future human–object motion trajectories online and feed them into downstream planners [2509.12250]. This suggests a convergence between **online perception**, **anticipation**, and **causal generation**.

More broadly, the literature indicates that future OnlineHOI systems may need to integrate four ingredients that are currently dispersed across separate papers: causal temporal modeling, explicit memory, open-vocabulary prompting, and physically grounded generation or reconstruction [2509.12250] [2605.14462]. OnlineHOI establishes the causal benchmark and a strong Mamba-based baseline; the remaining challenge is to combine that benchmark with richer semantics and broader world models without abandoning the strict online constraint.

Source: https://www.emergentmind.com/topics/onlinehoi