Cortex 2.0: Grounding World Models in Real-World Industrial Deployment

Abstract: Industrial robotic manipulation demands reliable long-horizon execution across embodiments, tasks, and changing object distributions. While Vision-Language-Action models have demonstrated strong generalization, they remain fundamentally reactive. By optimizing the next action given the current observation without evaluating potential futures, they are brittle to the compounding failure modes of long-horizon tasks. Cortex 2.0 shifts from reactive control to plan-and-act by generating candidate future trajectories in visual latent space, scoring them for expected success and efficiency, then committing only to the highest-scoring candidate. We evaluate Cortex 2.0 on a single-arm and dual-arm manipulation platform across four tasks of increasing complexity: pick and place, item and trash sorting, screw sorting, and shoebox unpacking. Cortex 2.0 consistently outperforms state-of-the-art Vision-Language-Action baselines, achieving the best results across all tasks. The system remains reliable in unstructured environments characterized by heavy clutter, frequent occlusions, and contact-rich manipulation, where reactive policies fail. These results demonstrate that world-model-based planning can operate reliably in complex industrial environments.

• The paper introduces Cortex 2.0, a vision-language-action system that samples future trajectories in visual-latent space, scores them for progress, risk, and completion, and executes the highest-ranked plan.
• Cortex 2.0 achieved 0.95–0.98 success across four industrial tasks with zero human interventions, outperforming reactive baselines most sharply in screw sorting, where it reached 0.98 versus 0.40 for the strongest baseline.
• The results show that lookahead planning can reduce cascading failures in long-horizon manipulation, although performance depends on proprietary deployment data, fixed planning budgets, and unvalidated transfer from real to imagined latent states.

Overview

Cortex 2.0, developed by Sereact, extends the preceding Cortex VLA model with a world-model-based planning module that shifts the system from reactive control to plan-and-act execution. At each decision step, a visual-latent world model generates $k$ candidate future trajectories; a Process-Reward Operator (PRO) scores each candidate for task progress, risk likelihood, and completion likelihood; and the flow-matching action policy commits only to the highest-scoring rollout. The system is evaluated on four real-world industrial tasks—single-arm pick-and-place, dual-arm item-and-trash sorting, screw sorting, and shoebox unpacking—where it achieves the highest success rates across all tasks with zero human interventions, while all reactive baselines require repeated interventions. The paper's central claim is that evaluating imagined futures before action commitment eliminates the compounding failure modes that make reactive VLAs unreliable in long-horizon industrial settings.

Architecture and Method

Cortex 2.0 is a four-level hierarchical design: a high-level VLM encodes the scene into a structured task context $s_t$; a world model generates candidate futures; PRO evaluates and ranks them; and flow-based action heads execute the selected plan. Observations combine wrist-camera RGB, robot state, optional force feedback, and instruction embeddings, encoded into a visual latent $z_t$ by the VLM encoder.

PRO scoring. PRO is a dense reward model trained on executed deployment trajectories with ground-truth outcomes from operational telemetry. It consumes predicted latent sequences $$\hat{\tau}_j = (z_{t+1}^{(j)}, \ldots, z_{t+H_{\text{wm}}}^{(j)})$$ and produces three signals: a progress head estimating value change $$\Delta p^{(j)} = V_\phi(z_{t+H_{\text{wm}}}^{(j)}) - V_\phi(z_t)$$, a risk head predicting failure probability along the trajectory, and a termination head predicting completion likelihood. These aggregate into a composite score $$S_j = \Delta p^{(j)} - \lambda\rho^{(j)} + \beta d^{(j)}$$. The advantage of the selected rollout relative to the candidate mean is binarized into an indicator $I_t \in \{0,1\}$ that conditions the policy; at inference $I_t$ is fixed to 1, biasing the VLA toward realizing the selected future. Notably, the risk head penalizes rollouts passing through latent states associated with high-speed contact, compression, edge impacts, or surface scraping even when the item ultimately reaches the goal—an explicit encoding of industrial damage-avoidance criteria absent from standard success-based rewards.

World model. The world model learns predictive dynamics in visual latent space via flow matching, pretrained on internet-scale video and fine-tuned on deployment recordings at 30 Hz. Flow-time sampling uses a Beta distribution biased toward high noise levels ($\beta \gg \alpha$), which the authors argue permits reliable trajectory scoring with fewer ODE integration steps, since PRO operates on motion plausibility rather than rendering fidelity. Each candidate is initialized from an independent noise realization, yielding multimodal future sampling.

Policy and cross-embodiment transfer. The VLA policy is a 2B-parameter VLM with a flow-matching action head producing chunks over horizon $H_{\text{act}}$, conditioned on the selected rollout token and advantage indicator through learned projections processed at full transformer depth. Actions are formulated as future states rather than raw control commands, and an Action Mapping Module—a lightweight adapter initialized from the last five layers of the action heads—handles embodiment-specific kinematics and workspace constraints. Because planning operates entirely in visual latent space, the same planning loop transfers across single-arm, dual-arm, and humanoid platforms without modification; this is the paper's stated mechanism for cross-embodiment generalization.

Training Procedure

Training proceeds in two phases. First, PRO is pretrained in isolation on real executed trajectories where telemetry provides ground-truth progress, risk, and termination labels; it is then frozen permanently. Second, the world model and action heads are trained jointly under $s_t$0, with PRO providing frozen advantage supervision. The authors follow the knowledge insulation scheme, blocking gradients from the world model and action heads into the pretrained VLM backbone in stage one before full end-to-end optimization in stage two. A key assumption underlying the entire design is that PRO scores transfer from real latents to imagined latents because both inhabit the same visual representational space; the paper does not empirically quantify any distribution shift between these regimes.

Dataset Composition

The training corpus combines four sources: proprietary deployment data exceeding 10 million curated episodes (>25k hours) drawn from a fleet with over 500 million cumulative manipulation interactions; approximately 40k teleoperation episodes (~400 hours) collected via egocentric recording, motion-capture suits, instrumented handheld grippers, and direct teleoperation with sub-centimeter replay precision; roughly 970k open-source episodes from Open X-Embodiment, BridgeData V2, and DROID; and ~20k synthetic episodes generated in RoboCasa/Isaac Sim. Deployment telemetry directly supervises PRO, while visual observations train the world model. The paper emphasizes a closed-loop improvement cycle: deployments generate data, better data improves planning quality, improved execution generates cleaner signal. This constitutes a substantial proprietary-data moat, but it also means the reported results are not reproducible without access to Sereact's fleet.

Experimental Results

All models were trained under an equivalent budget of 200 GPU-hours. Baselines are $s_t$1 (absolute joint space), Diffusion Policy (relative end-effector), and RDT-2 (relative end-effector); all execute at 30 Hz with chunk size 12. Evaluation tracks unrecoverable states requiring human intervention across three categories: safety-critical collisions, persistent control deadlocks, and unrecoverable scene states. Cortex 2.0 was evaluated at a fixed low-latency setting of $s_t$2.

Task	Cortex 2.0	$s_t$3	Diffusion Policy	RDT-2	Interventions (Cortex)
Pick-and-place	0.98	0.70	0.56	0.40	0
Item/trash sorting (per-op)	0.95	0.61	0.47	0.18	0
Screw sorting (per-op)	0.98	0.40	0.20	0.00	0
Shoebox unpacking	0.96	0.60	0.12	0.00	0

Several results warrant emphasis. In screw sorting—the largest gap—Cortex 2.0 achieves 0.98 per-operation success versus 0.40 for the strongest baseline, completing in 180 s with zero interventions, while RDT-2 records zero successful placements. In item-and-trash sorting, every baseline requires human intervention in every rollout (53–95 interventions total), whereas Cortex 2.0 completes autonomously in 700 s. In shoebox unpacking, trained on only 2,900 demonstrations, Cortex 2.0 reaches 0.96 holistic success in 58 s versus 103 s for $s_t$4. The zero-intervention result across all benchmarks is presented as the metric most reflective of industrial operational cost, since it captures autonomy breakdowns rather than per-step accuracy alone.

Planning-budget trade-off. Success rate rises monotonically with $s_t$5: from 0.962 at $s_t$6 to 0.996 at $s_t$7, but per-step latency grows from ~310 ms to ~9200 ms. All headline evaluations use $s_t$8, meaning the near-perfect figures at large $s_t$9 are reported separately from the deployed configuration. Rollout quality can be independently tuned via ODE denoising steps, giving a second compute–quality axis.

The authors attribute the qualitative robustness difference to lookahead filtering: reactive baselines exhibit the characteristic escalation pattern in which a missed grasp triggers retries that compound into deadlock, whereas PRO rejects rollouts traversing unstable-contact latent states before commitment. They further report that continued operation on in-domain data raises deployment success beyond 99%, though this figure is asserted rather than benchmarked against baselines.

Limitations and Open Questions

The paper concedes several constraints. Current evaluations cover a subset of embodiments and task families, and both the planning horizon $z_t$0 and budget $z_t$1 are fixed per task rather than adapted online; uncertainty-aware dynamic budget allocation remains future work. The claimed cross-embodiment generality is demonstrated on single- and dual-arm Universal Robot platforms, with humanoid capability asserted architecturally rather than validated experimentally. The transfer of PRO scores from real to imagined latents rests on the shared-representation assumption without direct measurement of scoring reliability under model error, and the world model itself risks compounding prediction error over longer horizons than those evaluated. The in-context learning discussion—conditioning a policy on video demonstrations $z_t$2—is framed as a research direction, with no experimental validation on unseen task families. Finally, baseline comparisons inherit the fairness caveats of heterogeneous action spaces ($z_t$3 in joint space, others Cartesian) and differing pretraining regimes, though compute budgets were equalized.

Conclusion

Cortex 2.0 demonstrates that integrating a visual-latent world model and a frozen, telemetry-trained reward operator into a VLA policy yields consistent gains over strong reactive baselines on four real industrial tasks, with the distinctive outcome of zero human interventions across all evaluations. Latent-space planning enables embodiment-agnostic scoring, and the continuous deployment-to-training feedback loop provides a scaling mechanism tied directly to fleet growth. The open questions the paper leaves are concrete: whether PRO scoring remains reliable as imagined-rollout horizons extend, whether dynamic planning budgets can replace per-task tuning, and whether video-conditioned in-context task specification holds outside the trained task families.