ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos

Abstract: Humans can effortlessly anticipate how objects might move or change through interaction--imagining a cup being lifted, a knife slicing, or a lid being closed. We aim to endow computational systems with a similar ability to predict plausible future object motions directly from passive visual observation. We introduce ObjectForesight, a 3D object-centric dynamics model that predicts future 6-DoF poses and trajectories of rigid objects from short egocentric video sequences. Unlike conventional world or dynamics models that operate in pixel or latent space, ObjectForesight represents the world explicitly in 3D at the object level, enabling geometrically grounded and temporally coherent predictions that capture object affordances and trajectories. To train such a model at scale, we leverage recent advances in segmentation, mesh reconstruction, and 3D pose estimation to curate a dataset of 2 million plus short clips with pseudo-ground-truth 3D object trajectories. Through extensive experiments, we show that ObjectForesight achieves significant gains in accuracy, geometric consistency, and generalization to unseen objects and scenes, establishing a scalable framework for learning physically grounded, object-centric dynamics models directly from observation. objectforesight.github.io

• The paper introduces a scalable framework that predicts future 3D object trajectories using a geometry-aware diffusion transformer integrated with object-centric 3D reasoning.
• It leverages a dataset of over 2 million curated object manipulation clips, demonstrating superior temporal consistency and precision compared to baseline methods.
• Quantitative results show significant improvements in ADE, FDE, and rotational errors, indicating strong potential for applications in embodied AI and robot planning.

Predicting Future 3D Object Trajectories from Human Videos: The ObjectForesight Framework

Problem Formulation and Motivation

The modeling of 3D object dynamics from passive human videos is a central challenge in scalable embodied AI. The "ObjectForesight" framework formalizes the task of forecasting future 6-DoF (SE(3)) trajectories for rigid objects solely from egocentric video streams of human manipulation, eschewing the need for direct human pose or action annotations. This approach moves beyond pixel- or latent-space world modeling, leveraging explicit object-centric 3D reasoning to capture physical affordances, object interaction semantics, and scene geometry.

Scalable Object-Centric 3D Trajectory Supervision

The lack of large-scale, metrically consistent 3D object interaction datasets has historically restricted progress in physically grounded dynamics prediction. The ObjectForesight pipeline addresses this by constructing, from raw egocentric video, over 2 million short object manipulation clips with pseudo-ground-truth 6-DoF trajectories. The pipeline couples automatic hand/object discovery and segmentation, visibility and manipulation gating with VLMs, robust instance tracking, temporal consensus mask generation, and multi-stage 3D mesh and pose alignment.

Figure 1: Data curation pipeline from egocentric video to quality-controlled, anchor-frame–canonicalized 6-DoF object trajectories.

The curated dataset significantly improves scale and semantic diversity compared to prior approaches and enables object-centric learning directly from diverse daily activity sequences.

Model Architecture: Geometry-Aware Diffusion Trajectory Prediction

ObjectForesight’s architecture integrates a geometry-aware point-based encoder with a Diffusion Transformer (DiT) for multimodal, temporally consistent trajectory sampling.

• The PointTransformerV3 encoder receives a point cloud extracted from the anchor frame’s depth, augmented with pose history encoded via anchor-query attention.
• An object-centric pooling module yields a latent scene embedding conditional on recent motion and spatial context.
• The DiT models the conditional distribution of future 6-DoF poses via a denoising process, with AdaLN-Zero layers injecting scene embedding and motion context at each block.
• Training uses v-parameterization under an SNR-weighted loss, with direct supervision on SE(3) trajectory errors, including regularization for translation, rotation, velocity, and acceleration.

  Figure 2: Model architecture combining motion-conditioned geometry encoding and diffusion-based future trajectory prediction for object-level 3D motion.

Qualitative and Comparative Evaluation

Qualitative analysis demonstrates that the framework generates physically plausible, semantically meaningful object motion predictions over diverse manipulation types—lifting, translation, rotation—conditioned only on short context windows and scene geometry.

Figure 3: Qualitative results showing prediction of plausible 6-DoF trajectories for manipulated objects, visualized with overlaid pose axes and transformed meshes.

Comparison to frame-based video generation models reveals that explicit modeling in SE(3) yields superior temporal consistency and stability in the predicted object trajectories. Specifically, pose trajectories extracted from video generation methods (Luma AI Ray3) exhibit less temporal coherence than the direct predictions from ObjectForesight.

Figure 4: Comparison of predicted motion: ObjectForesight yields sharply more stable and consistent 3D trajectories than pose estimations extracted from generated video.

Quantitative Results and Ablative Insights

The ObjectForesight DiT model achieves strong numerical performance on all primary metrics (ADE, FDE for translation; ARE, FRE for rotation; slope metrics for temporal drift):

• On EPIC-Kitchens, DiT outperforms the autoregressive baseline by over $3\times$ in ADE (0.019 vs. 0.067) and remains superior per-step in all translation metrics.
• ARE and FRE show consistent improvements, indicating both local step accuracy and global rotational stability.
• Against Luma AI Ray3, the DiT model achieves markedly lower errors (ADE 0.029 vs. 0.084; ARE $7.29^{\circ}$ vs. $12.86^{\circ}$).
• Evaluation on HOT3D-Clips demonstrates generalization beyond the training corpus, with DiT outperforming the autoregressive variant in all tested dimensions.

Ablation studies highlight that using $C=3$ context frames achieves the best error-rate tradeoff; increasing context introduces diminishing returns and potential overfitting or noise sensitivity. Training horizon analysis shows that moderate horizon lengths ($H=8$) optimize both short- and intermediate-term prediction accuracy, while excessively long horizons enable global temporal consistency at some cost to immediate-frame precision.

Implications, Limitations, and Future Directions

The explicit SE(3)-based, object-centric framework has immediate application prospects for open-world manipulation learning, robot planning, and transfer to manipulation policies or generative trajectory priors in real scenes. The combination of scalable 3D curation and robust multimodal trajectory generation positions ObjectForesight as a reference pipeline for learning physical priors from passive video.

Key limitations include restriction to rigid objects and the horizon of physically plausible predictions (typically short-term, 8–32 frames). The rigidity assumption precludes modeling of flexible, articulated, or highly deformable entities, suggesting extension toward more expressive representations (kinematic chains, learned deformation spaces, LIIFs). Furthermore, current models do not integrate explicit force or contact modeling, which could improve predictive sharpness for contact-rich or tool-mediated interactions.

In the broader context of video-based model learning and robot visual commonsense, this framework underlines the importance of explicit object-/geometry-level reasoning to improve stability, generalization, and interpretability relative to pixel-based generative models or purely latent forecasting schemes.

Conclusion

ObjectForesight establishes a scalable, physically grounded framework for forecasting future 3D object trajectories from passive human manipulation videos (2601.05237). By fusing large-scale, metrically consistent trajectory curation with geometry-aware transformer-based diffusion modeling, it delivers both strong quantitative performance and improved qualitative realism in object motion forecasting. Extensions toward broader object classes, articulated dynamics, and long-horizon interaction modeling represent promising future research avenues for this line of work.