SurgWorld Framework: VLA & World Modeling in Surgery

• SurgWorld is a comprehensive framework that integrates vision–language–action policy learning with generative world modeling to synthesize realistic surgical video–action pairs.
• It leverages a curated SATA dataset, a latent-diffusion model fine-tuned with LoRA, and an inverse dynamics model for precise kinematic inference.
• The framework demonstrates significant gains in action prediction accuracy and robotic task success rates, indicating its potential for scalable surgical autonomy.

The SurgWorld framework integrates vision–language–action (VLA) policy learning with generative world modeling to address the data scarcity challenge in autonomous surgical robotics. It leverages vast corpora of unlabeled surgical videos to synthesize realistic video–action pairs, facilitating scalable and efficient training of surgical robot control policies. SurgWorld comprises a curated video–text dataset, an advanced latent-diffusion world model, an inverse dynamics model for kinematic inference, and VLA policy training, culminating in successful deployment on real surgical robotic platforms (He et al., 29 Dec 2025).

1. Framework Architecture and Pipeline

SurgWorld operates through an orchestrated pipeline that spans data curation to real robot deployment. The workflow consists of:

1. Dataset Creation: The Surgical Action–Text Alignment (SATA) dataset features 2,447 video clips with expert-annotated textual descriptions from YouTube and six public benchmarks (GraSP, SAR-RARP50, AutoLaparo, HeiCo). Clips encompass four suturing subsequences: needle grasping, needle puncture, suture pulling, and knotting. Each annotation details instrument identities, spatial relations, and anatomical interactions.
2. World Model Fine-Tuning: The base is Cosmos-Predict 2.5, a latent-diffusion video predictor. LoRA modules are inserted into attention and feed-forward layers and the model is fine-tuned on SATA and several real robot trajectories.
3. Synthetic Rollout Generation: The world model generates photorealistic, temporally coherent surgery videos conditioned on initial frames and text prompts.
4. Inverse Dynamics Estimation: An IDM infers fine-grained pseudo-kinematics from the synthetic videos, yielding continuous action labels for each frame.
5. VLA Policy Training: A Transformer-based policy (GR00T N1.5) is trained on both real and synthetic video–action pairs via behavioral cloning.
6. Deployment: The trained policy is evaluated on real surgical hardware (e.g., dVRK-style arms for Needle Pickup & Hand-Over tasks).

2. World Modeling for Surgical Video Synthesis

The world model in SurgWorld is a latent-diffusion network adapted for surgical contexts with LoRA-based finetuning. Model inputs comprise a real initial frame and a detailed text prompt; outputs are photorealistic, anatomical video sequences.

Latent-Diffusion Formulation

Let $x_{0:T}$ denote a video sequence and $c$ the conditioning information. Encoded latent representations $z_t = E(x_t)$ are modeled as:

$$z_t = (1-t)\,z_0 + t\,\varepsilon,\quad v_t = \varepsilon - z_0;\quad \varepsilon \sim \mathcal{N}(0,I),\; t\in[0,1]$$

A velocity predictor $u_\theta(z_t, t, c)$ is optimized via flow-matching:

$$\mathcal{L}_{\rm world} = \mathbb{E}_{x, \varepsilon, c, t} \|u_\theta(z_t, t, c) - v_t\|_2^2$$

Inference proceeds via autoregressive or diffusion-based sampling in latent space, followed by decoding to video.

Dataset Conditioning and Diversity

Video generation is conditioned on initial frames and granular SATA prompts, delivering anatomically plausible and text-aligned surgical sequences. Multiple random seeds enable diverse rollout sampling (10× per prompt), which is critical for policy exploration and generalization.

3. Inverse Dynamics and Pseudo-Kinematics Inference

Given the absence of explicit kinematic annotations in surgical videos, SurgWorld employs an inverse dynamics model (IDM) to infer pseudo-actions from generated video pairs:

• Architecture: IDM shares the backbone with GR00T N1.5; it processes two frames ($x_t$, $x_{t+T}$, with $T=16$) and outputs per-frame continuous action vectors $\hat{a}_{t:t+T-1} \in \mathbb{R}^{20 \times T}$.
• Action Representation:

$$a_t = [p_L, r_L, g_L, p_R, r_R, g_R] \in \mathbb{R}^{20}$$

where $p$ is Cartesian position, $r$ rotation (6D), $g$ gripper state for both manipulators.

• Loss Function:

$$\mathcal{L}_{\rm inv} = \mathbb{E}_{(x_t, x_{t+T}, a_{t:t+T-1})} \sum_{\tau=0}^{T-1} \|\hat{a}_{t+\tau} - a_{t+\tau}\|_2^2$$

IDM is pretrained on out-of-domain episodes and finetuned on task-specific real demonstrations.

4. VLA Policy Learning from Synthetic and Real Data

Policy learning uses a Transformer (GR00T N1.5) that outputs future actions $a_{t:t+T-1}$ given current video frames $x_t$, textual prompts $c$, and optionally robot state. Behavioral cloning loss on action prediction is:

$$\mathcal{L}_{\rm bc} = \mathbb{E}_{(x_t, c, a_{t:t+T-1})} \sum_{\tau=0}^{T-1} \|\pi_\phi(x_t, c)_{t+\tau} - a_{t+\tau}\|_2^2$$

Training involves synthetic data generated via the world model and IDM (560 videos for 10× rollouts) mixed with real robot demonstrations. Fine-tuning steps are divided between synthetic and real data.

5. Experimental Results and Benchmarking

Quantitative and qualitative metrics demonstrate the efficacy of SurgWorld:

World Model Evaluation

Evaluation Metric	Baseline	Action Category Prompt	SurgWorld (Fine Prompt)
Fréchet Video Distance	175.4	143.0	106.5
VBench DD	-	-	62.4
VBench IQ	-	-	49.3
VBench OC	-	-	21.5
Success Rate (SR, %)	0.0 (Zero-Shot)	51.8 (Finetuned-Orig)	73.2 (SurgWorld)

SurgWorld achieves superior text–video alignment, tool consistency, and anatomical correctness in expert ratings.

Policy Performance

Training Setup	Cartesian MSE	Rotation MSE	Jaw MSE
Real Only	0.0123	-	-
+56 Synthetic	0.0071	↓	↓
+560 Synthetic (10×)	0.0042	↓	↓

Consistent reduction in action prediction error is observed across all components. Gains persist for varying numbers of real demonstrations and different policy variants.

6. Scalability, Data Efficiency, and Future Prospects

SurgWorld exploits internet-scale unlabeled surgical video, enabling rapid generation of large synthetic datasets for data-efficient policy training. With as few as five real demonstrations, the framework increases success rates by over 20 percentage points in robotic tasks. This suggests considerable impact on overcoming covariate shift and enabling policy generalization.

Planned extensions include expanding the SATA dataset to cover additional procedures and primitives, improving IDM with uncertainty modeling, joint end-to-end training of world model, IDM, and policy, and integrating online fine-tuning for sim2real adaptation.

7. Integration with Surgical Simulation Ecosystems

SurgWorld provides a plug-and-play environment for downstream agent, trainee, or planning tasks:

• Model-based RL: Utilizes learned dynamics $p_{\rm dyn}(z_{t+1}|z_{1:t},a_{1:t})$ for synthetic trajectory generation and policy optimization.
• Interactive Training: Enables medical trainees to specify tool actions via codebook tokens, facilitating visual feedback and hypothesis testing.
• Preoperative Planning: Allows clinicians to input patient-specific frames and simulate prospective tissue/tool responses under hypothetical maneuvers.

Current limitations include latent actions lacking semantic labels, restricted spatial resolution in the action model, and reliance on fixed-camera endoscope settings. Semi-supervised fine-tuning and integration of physics-based simulation may address these constraints.

SurgWorld constitutes a comprehensive framework unifying world modeling, pseudo-kinematic inference, and VLA policy learning for surgical autonomy, setting the foundation for broadly generalizable, scalable, and high-fidelity surgical robot training (He et al., 29 Dec 2025, Koju et al., 3 Mar 2025).