Aether: Geometric-Aware Unified World Modeling (2503.18945v2)

Abstract: The integration of geometric reconstruction and generative modeling remains a critical challenge in developing AI systems capable of human-like spatial reasoning. This paper proposes Aether, a unified framework that enables geometry-aware reasoning in world models by jointly optimizing three core capabilities: (1) 4D dynamic reconstruction, (2) action-conditioned video prediction, and (3) goal-conditioned visual planning. Through task-interleaved feature learning, Aether achieves synergistic knowledge sharing across reconstruction, prediction, and planning objectives. Building upon video generation models, our framework demonstrates unprecedented synthetic-to-real generalization despite never observing real-world data during training. Furthermore, our approach achieves zero-shot generalization in both action following and reconstruction tasks, thanks to its intrinsic geometric modeling. Remarkably, even without real-world data, its reconstruction performance is comparable with or even better than that of domain-specific models. Additionally, Aether employs camera trajectories as geometry-informed action spaces, enabling effective action-conditioned prediction and visual planning. We hope our work inspires the community to explore new frontiers in physically-reasonable world modeling and its applications.

Aether: Geometric-Aware Unified World Modeling

The paper "Aether: Geometric-Aware Unified World Modeling" presents a novel framework named Aether, a multi-task AI system aimed at enhancing spatial reasoning through the integration of geometric reconstruction and generative modeling. The significant features of Aether include its ability to perform 4D dynamic reconstruction, action-conditioned video prediction, and goal-conditioned visual planning, all while being trained entirely on synthetic data.

Overview of the Approach

Aether is grounded on a unified framework allowing synergistic knowledge sharing across three principal tasks: dynamic 4D reconstruction, action-conditioned prediction, and goal-conditioned visual planning. The model leverages pre-trained video generation models and is further refined with post-training using synthetic 4D data. This method demonstrates remarkable zero-shot generalization, achieving higher reconstruction performance than some domain-specific models despite not observing real-world data during training. Specifically, the framework's geometric modeling facilitates zero-shot generalization in action-following and reconstruction tasks. Aether uses a geometry-informed action space, translating predictions effectively into actions, thereby enabling autonomous trajectory planning.

Technical Contributions and Innovations

1. 4D Dynamic Reconstruction: Aether's reconstruction capabilities significantly outperform existing methods on benchmarks such as KITTI, where it achieves an Absent Rel (Abs Rel) error of 0.056, demonstrating superior depth estimation accuracy.
2. Action-Conditioned Video Prediction: The framework harnesses the power of synthetic data to generate video predictions conditioned by both initial observations and specified actions, utilizing camera trajectories as a global action representation.
3. Goal-Conditioned Visual Planning: By integrating observation-goal image pairs, Aether not only predicts future frames but also plans viable paths to achieve specified goals, which underscores its potential application in autonomous navigation and manipulation tasks.
4. Synthetic Data Annotation Pipeline: An essential contribution of this paper is the creation of a robust pipeline for automatic annotation of synthetic data, which accurately reconstructs 4D dynamics while addressing challenges in data scarcity.

Quantitative Results

The effectiveness of Aether is quantified through extensive experiments. Results demonstrate that Aether's performance in zero-shot video depth estimation is either on par with or surpasses existing reconstruction models. Furthermore, in camera pose estimation tests on datasets like Sintel and ScanNet, Aether shows competitive results, underscoring its utility across dynamic and static scenarios alike.

Implications and Future Directions

The integration of geometric reasoning with generative modeling in Aether marks a substantial step toward comprehensive world modeling. The implications for AI systems' spatial intelligence are profound, suggesting applications in simulation, gaming, and autonomous vehicles, where understanding and planning within dynamic environments are crucial. Future explorations could enhance real-time capabilities, further refining action representation, and integrating more diversified data sources to improve the generalization of synthetic-trained models.

While the paper does an impressive job at demonstrating the system’s capabilities through extensive experimentation and evaluation, some future work could include enhancing camera pose estimation accuracy or improving prediction reliability in dynamically complex scenes. Furthermore, exploring co-training with real-world datasets and reinstating text prompt capabilities could augment the system’s robustness and flexibility.

In summary, Aether presents a comprehensive, geometry-aware approach to world modeling that advances the current task boundaries in spatial reasoning. This framework encourages further exploration into scalable and efficient synthetic training methodologies within the AI research community.