Orca: The World is in Your Mind

Abstract: We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines. These results show that Orca, as a general world foundation model, presents a promising approach to understanding, predicting, and acting upon the world. Finally, we discuss the current limitations, aiming to provide useful insights and inspiration for the community.

• The paper establishes that a predictive world latent learned from rich multimodal signals significantly enhances downstream performance across text, vision, and action modalities.
• It introduces a dual learning paradigm—unconscious and conscious—that captures both natural dynamics and semantic state transitions via an encoder-decoder architecture.
• Experimental results demonstrate Orca’s scalability with improvements in text generation, image prediction, and robot manipulation, outperforming comparable baselines.

Orca: World Latent Modeling with Unified Multimodal State Transitions

Introduction and Model Motivation

Orca (2606.30534) establishes a paradigm shift in foundation model design by centering the modeling objective not on next-token, next-frame, or next-action prediction, but rather on next-state prediction within a unified world latent space. The authors argue this abstraction is essential for evolving general-purpose agents capable of understanding, predicting, and acting within complex, multimodal environments. The key hypothesis is that an internalized, predictive world latent—learned from large-scale, multimodal signals—provides a robust substrate for downstream readout across language, vision, and action, overcoming limitations associated with modality- or task-specific specialization.

Modeling Framework and Learning Paradigms

Orca is formulated around an encoder-decoder architecture. The encoder ingests multimodal world signals, mapping them to an abstracted latent state space $\mathcal{S}$. The state transitions are modeled as:

$$S_{t+\Delta}\sim p_\Theta(S_{t+\Delta}\mid S_t, z_t, c_t)$$

where $z_t$ denotes implicit (unobserved) dynamics and $c_t$ explicit semantic or task-conditioned controls. Crucially, Orca’s learning bifurcates into two paradigms:

• Unconscious Learning: Dense state transitions are acquired from continuous, unlabeled video streams ($c_t=\varnothing$), analogous to predictive coding of the physical world.
• Conscious Learning: Sparse, high-level transitions are learned by conditioning on language-annotated events and VQA data ($c_t=e_{t+\Delta}$), grounding the latent in semantically meaningful state shifts and causal reasoning.

This dual pathway enables the model to encode both naturalistic physical dynamics and instruction-governed event transitions.

Training Procedure and Data Construction

Orca is trained in two distinct stages:

1. Pre-training (world latent acquisition):
   • Objectives comprise: (a) observation-only transition (unconscious), (b) event-conditioned transition (conscious, language-guided), and (c) VQA-driven response generation.
   • Data: 125K hours of heterogeneous videos (ego-centric, exo-centric, robot execution, natural dynamics), 160M event-level annotations, and 11.5M VQA samples. Each objective supervises the model in a latent space—abandoning pixel-wise reconstruction in favor of abstract state alignment.
2. Downstream Post-training (readout):
   • The backbone encoder is frozen; only lightweight modality-specific decoders (for text, vision, action) are trained. This isolates the effect of the learned world latent on downstream probe tasks.

Orca’s scalability is facilitated by sophisticated distributed infrastructure (FSDP2, activation recomputation, chunked cross-entropy, pre-fetching), achieving a 4.4$\times$ efficiency increase over prior arts.

Experimental Results

World Latent Scalability and Effectiveness

• Scaling Loss: As model and data scale up (0.8B/4B parameters, increasing video hours), pre-training losses decrease monotonically without rapid saturation, indicating robust capacity for absorbing more structural world knowledge from data.
• Readout Probing: Downstream performance (text, image, action) improves strictly as the strength of the frozen world latent increases—with consistent gains for all modalities.

Text Generation

• Orca achieves strong performance relative to both world model (e.g., V-JEPA 2.1, Emu3/3.5) and VLM (e.g., Qwen3.5, Gemma 4, DeepSeek-VL2) baselines at comparable or much larger scale. For example, on 3DSRBench, TemporalBench, and SWITCH, Orca-4B attains up to 5–12% higher scores than the nearest VLM competitors in state transition and dynamic motion dimensions.
• Capabilities extended to complex causal, counterfactual reasoning, and spatial/dynamic inference, demonstrating occupancy of non-trivial regions of the world-model capability spectrum.

Image Prediction

• On the real-world PRICE-V0.1 benchmark, Orca’s latent, read out through a minimal trainable adaptor to a frozen SD3.5 decoder, achieves the highest average human-aligned evaluation scores among OmniGen2 and FLUX baselines. Notable improvements are observed in physical plausibility, action consistency, and object-scene grounding—contrasting with the typical hallucinations or instruction drift seen in generative baselines.

Action Generation

• For visually-conditioned robot manipulation under OOD environment and object settings, Orca’s latent—when coupled to a DiT-based action expert trained on only 200 demonstrations per task—surpasses Qwen3.5 and V-JEPA 2.1 representations in both rule-based and PRM-as-a-Judge metrics, and closely approaches the performance of specialized, large-scale VLA policies like $\pi_{0.5}$.
• Trajectories generated from the Orca latent demonstrate improved recovery from execution errors (higher DRR, FNS), maintain meaningful progress on partial failures, and generalize manipulations in novel scenarios, despite pre-training with no action labels.

Ablations

• The combination of all three pre-training objectives yields the most balanced tradeoff across modalities.
• Observation-only (unconscious) transitions are essential for robust action generation; event-conditioned (conscious) transitions primarily influence image prediction performance; language VQA supervision anchors the semantic/commonsense alignment and preserves language readout ability.

Limitations and Future Directions

The current instantiation of Orca exhibits several acknowledged constraints:

• Signal Diversity: Only vision and language signals are ingested. Physical (e.g., force, audio, proprioception) and scientific modalities remain absent.
• Supervision Structure: The model is partially constrained by the semantic space of a pre-trained VLM; native modeling from raw, multi-modal world signals is required for further generality.
• Data and Model Scale: Only a fraction of the available video corpus is utilized; performance trade-offs between downstream modalities indicate bottlenecks at the 4B parameter scale.
• Evaluation Scope: PRICE-V0.1, while novel, does not capture long-horizon, high-diversity interactions. Similarly, real-robot tasks remain short-horizon due to practical constraints.

The authors outline explicit research trajectories: integrating richer modalities, native world-latent modeling, institution of a systematic state transition evaluation corpus, closed-loop self-evolutionary frameworks for model-data generation, and extension to the scientific and physical domains.

Conclusion

Orca (2606.30534) formalizes a world-modeling-first paradigm for multimodal foundation models, placing state-transition prediction at the core of representation learning. The model demonstrates that a robust world latent—acquired from unconstrained and semantically interpreted signals—substantially boosts downstream generalization in text, vision, and embodied action, outperforming modality-specialized or next-token/frame/action-centric baselines of similar or even larger scale. The architectural and experimental design convincingly motivates a research agenda that treats the world as a dynamic latent space, with implications for future models seeking generalization, sample efficiency, and reliable physical/causal reasoning across domains.