OASIS: Observation-Action Space Alignment via SE(3) Trajectory Prediction for Robotic Manipulation

Abstract: Recent vision-language-action (VLA) models and world action models (WAMs) advance robotic manipulation by enriching intermediate representations with auxiliary spatial features or future visual-state prediction. However, these representations largely remain within the observation space and do not share the rigid-body geometry of the action space, forcing the action decoder to implicitly recover this geometry. We propose OASIS, a visuomotor policy that aligns the intermediate representation with the action space via $SE(3)$ end-effector trajectory prediction. OASIS couples a 3D-aware feature encoder that fuses vision-language and metric-depth features with an $SE(3)$ trajectory predictor that produces a camera-frame end-effector trajectory. Conditioned on the predictor's pose-supervised hidden states, the action decoder generates action chunks consistent with rigid-body motion. Across simulation and real-world experiments, OASIS outperforms VLA and WAM baselines in success rate and out-of-distribution generalization. Our project page is available at https://npuhandsome.github.io/OASIS_web.

• The paper introduces a novel SE(3)-aligned framework that directly predicts end-effector trajectories, reducing the need for implicit geometric inference.
• It employs a 3D-aware encoder, transformer-based trajectory predictor, and cross-attending action decoder to achieve up to 97.6% success rates on key robotic benchmarks.
• The approach demonstrates exceptional generalization under out-of-distribution conditions and improved data efficiency, enabling practical real-world robotic applications.

OASIS: Geometric Alignment of Visuomotor Policy Representations via $SE(3)$ Trajectory Prediction

Motivation and Problem Statement

Recent advances in robotic visuomotor policies have leveraged multimodal representations—spanning vision, language, and auxiliary spatial cues—to drive 6-DoF action generation for manipulation tasks. However, prevailing models such as Vision-Language-Action (VLA) policies and World Action Models (WAMs) create intermediates that reside in the observation or latent feature space rather than the $SE(3)$-structured space of the end-effector action manifold. This architectural disconnect necessitates that the action decoder implicitly recover rigid-body geometry, potentially limiting precision and generalization, especially under distributional shift or compounding errors in long-horizon settings.

OASIS (Observation-Action Space Alignment via $SE(3)$ Trajectory Prediction for Robotic Manipulation) addresses this limitation by introducing a design principle: explicitly aligning intermediate representations with the $SE(3)$ action space through direct prediction of future end-effector trajectories in camera coordinates. This alignment provides a geometric inductive bias that enables downstream decoders to generate precise, physically consistent actions and mitigates the implicit pose inference otherwise required.

Figure 1: Existing VLA and WAM architectures construct intermediates in observation space, requiring implicit geometric reasoning, whereas OASIS directly aligns intermediates with the action space via $SE(3)$ trajectory prediction.

Methodological Contributions

OASIS implements the proposed design principle through a three-stage end-to-end policy architecture:

1. 3D-aware Feature Encoder: This stage fuses RGB images, natural language instructions, and metric depth information into a joint representation. Leveraging a vision-language backbone (Qwen2.5-0.5B) along with Depth Anything 3 for calibrated metric depth, the encoder produces an input that is both semantically and spatially grounded.
2. $SE(3)$ Trajectory Predictor: Conditioned on the 3D-aware representation, the predictor comprises stacked transformer blocks that generate horizon-indexed pose-supervised hidden states ($H=8$). These are projected to a sequence of $SE(3)$ camera-frame end-effector target poses using axis-angle parameterization for $SO(3)$.
3. Action Decoder: Cross-attending to the trajectory predictor's hidden states and the current robot state, the action decoder outputs executable action chunks (relative Cartesian motions and gripper commands) that realize the predicted trajectory, absorbing practical residuals such as camera extrinsics, dynamic contacts, and gripper timing offsets.

   Figure 2: OASIS architecture: the encoder merges multimodal scene information, the $SE(3)$ predictor aligns representation geometry, and the decoder generates actions conditioned on pose-supervised intermediates.

Notably, OASIS operates without large-scale robotic pretraining or dense spatial annotations, instead training exclusively from standard expert demonstrations.

Experimental Results

Simulation Benchmarks

OASIS is evaluated on the LIBERO and CALVIN benchmarks. On LIBERO, OASIS achieves a 97.6% average success rate across four suites (Spatial, Object, Goal, Long), consistently outperforming both spatially enhanced VLA baselines (e.g., QDepth-VLA, Unified-VLA) and world-model-based approaches. The model shows clear superiority especially on long-horizon and spatially intricate tasks, exceeding strong VLA baselines by up to 13.2% and maintaining robustness across varying instructions and scene compositions.

On CALVIN’s ABC$SE(3)$0D setting—testing compositional generalization in a previously unseen environment—OASIS attains an average sequence length of 4.57 and an 83.3% five-task-chaining success rate, both setting new empirical standards and reducing error accumulation typical in non-geometric intermediates.

Figure 3: Visualization of $SE(3)$1 trajectory prediction versus robot execution (left). OASIS closely tracks both predicted translation and orientation waypoints, confirming geometric consistency. Ablation (right) shows monotonic gains from richer geometric supervision, with an axis-angle parameterization yielding highest success.

Ablation Studies

Ablations on LIBERO-Long and LIBERO-Spatial reveal:

• Removing the metric-depth feature reduces success by 3–6%, showing the necessity of absolute spatial scale.
• Substituting the $SE(3)$2 trajectory predictor with 2D or 3D positional supervision, or decoupling decoder conditioning, each results in significant performance drops (from 95.2% to 89.5%), confirming that only direct $SE(3)$3 alignment provides maximal benefit.
• Among $SE(3)$4 parameterizations, axis-angle encoding leads (95.2%) versus quaternions (91.6%) and Euler angles (92.2%), validating the choice for minimal parameterization in bounded workspace settings.

Real-World Results

OASIS is deployed on both Franka Research 3 and Kinova Gen3 platforms, with tasks spanning single-step goal fulfillment, precise spatial arrangements, and long-horizon chains. OASIS achieves an average real-world success rate of 89.2% (Goal: 98.6%, Spatial: 85.8%, Long: 83.3%), substantially outperforming ACT, Seer-Large, RDT, and $SE(3)$5.

OASIS demonstrates robust generalization under various out-of-distribution (OOD) perturbations in the Goal task—unseen backgrounds, altered camera perspectives, and dynamic human interference—maintaining success rates above 86%. This is attributed to the metric depth’s resistance to distractors and the predictor’s frame alignment.

Figure 4: Real-world robot platforms, task examples (left), and summary of success rates across main suites (center). OASIS dominates in both in-distribution and OOD (right) scenarios on the Goal task.

OASIS also demonstrates pronounced data efficiency, matching or exceeding the performance of large-scale pretrained models like $SE(3)$6 using fewer demonstrations in long-horizon tasks.

Figure 5: Real-world execution samples for Goal, Spatial, and Long tasks, illustrating multi-object manipulation, precise placement, and sequential reasoning.

Figure 6: Execution under OOD conditions: altered scene, viewpoint, and dynamic intervention—all handled seamlessly by OASIS.

Implications and Future Directions

OASIS advances the design of visuomotor policies by demonstrating that explicit geometric alignment of intermediates with the $SE(3)$7 action space produces measurable improvements in both precision and generalization, without the need for large datasets or privileged sensing. This observation has several implications:

• Practical Deployment: The increased robustness and data-efficiency are conducive to rapid deployment in real-world applications—such as assistive robots, industrial assembly, and laboratory automation—where annotated datasets and calibration are limited.
• Debuggability and Transparency: Pose-supervised hidden states are interpretable, potentially enabling more transparent debugging, policy introspection, and safer real-time deployment.
• Theoretical Grounding: Geometric alignment provides an inductive bias that could become foundational in designing multimodal robot policies capable of compositional generalization, hierarchical planning, and adaptation.

Future work may extend this design principle to higher-dimension action spaces (e.g., dexterous hand manipulation, mobile-base plus arm), leverage coupled trajectory spaces ($SE(3)$8), or introduce contact-conditioned intermediates to handle richer interaction dynamics.

Conclusion

By aligning intermediate representations with the $SE(3)$9 action space via trajectory prediction, OASIS resolves a critical limitation in the observation-centric architectures prevalent in current visuomotor policies. This alignment delivers substantial gains in task success rate, robustness under OOD perturbations, and sample efficiency on both simulated and physical platforms. The approach opens a new avenue for the principled design of generalist robot policies where explicit geometry rather than implicit spatial reasoning serves as the scaffold for action generation.