- The paper introduces OnlineHMR, a fully causal framework that combines key–value-cached sliding-window attention with incremental SLAM to recover globally positioned human meshes from monocular video without future frames.
- The method matches offline systems on standard benchmarks, achieving 43.7 mm PA-MPJPE and 6.4 mm acceleration error on 3DPW, plus 93.5 mm world-aligned MPJPE and 2.2% relative translation error on EMDB-2.
- The system improves practical streaming performance through soft human-region masking, trajectory smoothing, metric-scale recovery, and a spectrogram-based jitter metric, reaching 3.3 FPS with 0.30 seconds of average per-frame delay.
OnlineHMR addresses a persistent gap in world-grounded human mesh recovery (HMR): while recent methods recover global human trajectory and motion from monocular video, nearly all of them operate offline, requiring future frames or globally optimized camera poses before producing an estimate. The paper proposes a fully online framework that jointly satisfies four criteria—system-level causality, faithfulness, temporal consistency, and efficiency—and demonstrates performance comparable to chunk-based offline baselines on standard benchmarks (2603.17355).
The authors argue that online processing is essential for interactive applications such as AR/VR and telepresence, where predictions must be available immediately for perception–action loops. They formalize causality at the system level: every module, including both camera-coordinate HMR and global trajectory estimation, must depend only on current and past inputs. This framing exposes a weakness in prior work: WHAM, despite being described as online, relies on DPVO or DROID-SLAM backends whose sliding-window local optimization revises previous camera poses using later observations; the paper therefore classifies WHAM as offline in practice. Similarly, TCMR, MPS-Net, and GLoT consume fixed 16-frame windows centered on the target frame, violating causality and introducing redundant computation across overlapping windows.
The framework builds on TRAM but restructures it for streaming inference. The world-coordinate mesh is obtained by applying the estimated camera pose—quaternion rotation qic, translation tic, and metric scale factor s—to the camera-coordinate SMPL mesh.
Online camera-coordinate HMR
The HMR branch uses a sliding-window training strategy with window length N=3 and step size 1. Within each window, only the last frame's query performs self-attention on its own patch features and cross-attention over preceding frames, so single-frame outputs can be concatenated into a temporally continuous sequence. Training supervises frame-level losses (2D/3D keypoints, SMPL parameters, vertices) plus inter-window velocity and acceleration regularization on joint positions relative to the pelvis, weighted by per-joint ground-truth confidence.
At inference, a key–value cache stores features of past frames, so each new frame attends to cached keys and values without recomputation or access to future frames, yielding constant-time per-frame processing. This train-with-windows / infer-with-cache design mirrors streaming techniques such as SAM 2's memory banks and StreamVGGT's causal attention, adapted here to the HMR setting.
A notable ablation result: velocity regularization reduces Accel error on 3DPW from 8.9 to 6.4 mm and Jitter from 32.3 to 19.5, with comparable gains on EMDB-1 (Accel 15.7 → 9.0). A supplementary ablation shows accuracy peaks at window size 4, but larger windows inflate Accel due to a delay effect from fusing more past frames—an inherent tension between temporal context and responsiveness under causality.
Human-centric incremental SLAM
For the camera branch, the authors replace TRAM's globally optimized SLAM extrinsics with incremental poses from MASt3R-SLAM. Because human-centric video violates SLAM's static-scene assumption, they segment the human region with SAM 2 and mask it before feature extraction. Crucially, hard masking leaves boundary artifacts that still produce spurious matches, so they apply dilation followed by Gaussian blur to form a soft confidence mask. This yields substantial improvements: ATE drops from 2.52 to 1.07 with DROID-SLAM and from 1.22 to 0.83 with MASt3R-SLAM relative to vanilla inputs, and the soft mask improves world-coordinate WA-MPJPE from 119.6 to 93.5 mm and RTE from 4.1% to 2.2% compared to no masking.
To regularize trajectory noise online, the method applies exponential moving average (EMA) correction to incremental camera translations—with velocity-dependent clamping to avoid suppressing genuine fast motion—and hemisphere-consistent quaternion interpolation for rotations. Corrected poses are fed back into the SLAM backend. Since MASt3R-SLAM outputs are up-to-scale (Sim(3) rather than SE(3)), a metric scale factor is recovered from MoGe-V2 depth maps, deliberately excluding human pixels because dolly-zoom effects make human-region depth unreliable as a scale cue.
Frequency-domain jitter metric
Beyond Accel and Jitter, the paper introduces a spectrogram-based evaluation inspired by Mel-Cepstral Distortion: STFT of joint position sequences (Hann window, nfft=128, hop 32), followed by normalized RMSE and correlation against ground-truth spectrograms. This captures visually perceived temporal instability across frequency content that scalar acceleration metrics miss. On EMDB-1, OnlineHMR achieves RMSE 24.98 versus TRAM's 25.82 despite being online, though GVHMR attains better pattern correlation (0.12 vs. 1.41 normalized Corr) with larger amplitude deviation—a mixed picture the authors report transparently.
Experimental results
On camera-coordinate benchmarks, OnlineHMR leads all Video-Online methods: PA-MPJPE of 43.7 mm on 3DPW and 46.0 mm on EMDB-1, with Accel of 6.4 and 9.0 respectively, showing no significant degradation relative to offline systems like TRAM (35.6/45.7 PA-MPJPE) or GVHMR. On EMDB-2 world-coordinate evaluation, it achieves WA-MPJPE100 of 93.5 mm and RTE of 2.2%, outperforming Human3R (112.2 mm) and TRACE (529.0 mm), though W-MPJPE (310.4 mm) trails Human3R (267.9 mm). The authors attribute this gap to metric-scale conversion error: strong WA-MPJPE with weaker W-MPJPE indicates the trajectory shape is correct but incremental scale drifts within segments.
Efficiency results underscore the practical value of the design. Average per-frame delay is 0.30 s for OnlineHMR versus 26.18 s for WHAM, 115.95 s for TRAM, and 2435 s for SLAHMR—the latter two growing linearly with video length since total delay scales as F(F−1)/(2⋅FPS). Throughput is 3.3 FPS, below Human3R's 4.8 FPS, but with better accuracy; the authors caution that FPS comparisons are confounded by batching opportunities available only to offline methods.
Limitations and open questions
The framework depends on continuous viewpoints: abrupt camera cuts and multi-camera inputs break the incremental SLAM and EMA assumptions, and failure cases include repetitive textures and dynamic shadows. The EMA correction also embeds a smoothness prior that could distort genuinely abrupt human motion (e.g., transport scenarios), which the authors acknowledge when rejecting learned motion priors yet relying on camera-trajectory smoothing instead. Open questions left by the paper include robustness under viewpoint discontinuities, extension to multi-camera settings, and reducing the scale-estimation error responsible for the W-MPJPE gap.
Conclusion
OnlineHMR demonstrates that world-grounded human mesh recovery can be performed causally and incrementally without sacrificing reconstruction fidelity relative to offline chunk-based pipelines. Its combination of KV-cached sliding-window attention, soft-masked incremental SLAM, EMA-based physical correction, and a spectrogram-level jitter metric provides both a working system and an evaluation methodology for online motion naturalness, establishing a concrete baseline for streaming human perception.
