SurgWorld: Surgical Simulation & Evaluation

Updated 1 January 2026

SurgWorld is a framework that employs learned, action-conditioned models to simulate complex surgical environments by integrating spatiotemporal dynamics, biomechanical interactions, and procedural logic.
It utilizes advanced architectures such as diffusion-based video models and action-conditioned transformers to generate realistic surgical simulations and enable data synthesis for autonomous training.
Evaluation in SurgWorld combines quantitative policy metrics and expert-annotated plausibility scores to objectively benchmark simulation performance and support sim-to-real transfer.

SurgWorld defines a paradigm for learning, simulating, and evaluating surgical procedures through data-driven, action-conditioned world models. These models capture the complex spatiotemporal dynamics, biomechanical interactions, and procedural logic required for both surgical skill acquisition by autonomous agents and rigorous performance evaluation across a variety of robotic and vision-based platforms.

1. Definition and Scope

SurgWorld denotes action-conditioned, learned world models that simulate surgical environments with the explicit aim of training, policy evaluation, and data synthesis for surgical robotics and surgical video-based AI. Unlike generic visual world models, SurgWorld incorporates expert-sense causal knowledge relevant to surgical anatomy, instrument–tissue biomechanics, and procedural strategy, extending far beyond common-sense physics. These models serve as internal simulators, digital twins, and policy training environments for both synthetic generation and closed-loop agent–environment interaction (Chen et al., 3 Nov 2025).

2. Modeling Architectures and Data Foundations

SurgWorld implementations center on diffusion-based predictive video models, action-conditioned transformers, and latent variable models with explicit or learned action semantics.

Cosmos-Surg-dVRK: A surgical finetune of Cosmos-Predict2, starting from a pretrained transformer-based, action-conditioned video diffusion engine. Input comprises current endoscopic RGB frames and sequences of 12 kinematic actions encoding Cartesian translation, rotation (quaternion), and jaw angle. The model autoregressively predicts future visual sequences by unrolling conditioned on both visual and action tokens, learning kinematic articulation and soft-tissue deformation implicitly from paired video–kinematics data (Zbinden et al., 17 Oct 2025).
SurgWorld (Text-Action Alignment focus): Extends Cosmos-Predict2.5, incorporating LoRA-based adapters for task-specific adaptation. Conditioning is achieved via fine-grained text descriptions (Surgical Action–Text Alignment, SATA dataset) of four atomic actions (needle grasp, puncture, suture pulling, knotting), supporting both synthetic video generation and action-paired data synthesis through an inverse dynamics model (IDM) (He et al., 29 Dec 2025).
Vision-Latent Action Approaches: SurgWM adopts VQ-VAE-style video tokenization and latent action modeling via spatial-temporal transformers and MaskGIT-style token prediction, achieving unsupervised action quantization and interactive video generation from unlabeled surgical videos (Koju et al., 3 Mar 2025).
Specialized Suturing Models: LTX-Video and HunyuanVideo diffusion architectures simulate fine-grained sub-stitch biomechanics, enabling conditional generation of "ideal" and "non-ideal" technique demonstrations essential for objective skill assessment and closed-loop policy learning (Turkcan et al., 16 Mar 2025).

Data curation is foundational. The SATA dataset (2,447 clips, eight surgery types, 300,000+ frames) anchors action–text-world associations. Additional datasets: Tabletop Suture Pad (3,036 episodes), Ex-vivo Cholecystectomy (16,506 episodes), and diverse robot and teleop datasets ensure heterogeneity and generalizability (Zbinden et al., 17 Oct 2025, He et al., 29 Dec 2025).

3. Evaluation Benchmarks, Metrics, and Validation

Evaluation in SurgWorld environments is rigorous and multi-modal:

Quantitative Policy Evaluation: Success rate (SR), Pearson correlation (ρ), mean absolute error (MAE), mean maximum rank violation (MMRV), mean bias error (MBE), and intraclass correlation coefficient (ICC) are employed to compare simulated and real robot outcomes on tasks like needle pickup, handover, and cholecystectomy. Automated classifiers (e.g., V-JEPA 2-based video classifier, attentive head probe) deliver frame- and clip-level outcome assessments, matching human expert precision (ICC up to 0.836, Pearson ρ ≈ 0.756) (Zbinden et al., 17 Oct 2025).
Causal-Plausibility Assessment: The Surgical Plausibility Pyramid (SPP) provides a four-tiered rubric—visual appearance, instrument operation, environment feedback, surgical intent—for expert annotation of generated video rollouts. High plausibility at visual levels (score ≈3.7/5 at 1 s) contrasts with critical failures in instrument operation and surgical intent at 8 s (scores <2.0), revealing the "plausibility gap" endemic to current generation models (Chen et al., 3 Nov 2025).
Policy Data Efficiency: Use of synthetic, action-labeled rollouts (via IDM) enables transformer-based VLA policies (e.g., GR00T N1.5) to surpass baselines trained on real demonstrations only. Success rates on dexterous hand-over tasks exceed 70% using only five real demonstrations plus synthetic data, a 20+% improvement over real-only baselines. Ablations show importance of text-anchored generation and multi-view fusion (He et al., 29 Dec 2025).
Generalization and Robustness: Integrations with model-based RL frameworks, uncertainty-aware depth encoding, and dynamic state representations (e.g., 64×64×3 DSA in GAS) demonstrate robustness across object types, grippers, disturbances, and sensory noise (average real-world SR: 69%, simulation: 87%) (Lin et al., 2024).

4. Automated and Interactive Simulation Pipelines

SurgWorld pipelines support fully automated online evaluation and interactive agent training:

Action-Conditioned Simulators: Closed-loop policy evaluation occurs with the policy transmitting action sequences to the simulator, receiving simulated next frames as observation, and repeating for hundreds to thousands of steps per trial. Real-time simulation is enabled by GPU inference (Zbinden et al., 17 Oct 2025).
Video Classification and Success Detection: High-capacity video encoders (frozen V-JEPA 2 ViT-Huge, attentive classifier probes) process rollouts into temporal context embeddings. Automated labeling—success, anomaly, default—eliminates manual review, with outcomes determined by classifier output sequence logic (e.g., "success" before "anomaly" denotes task completion) (Zbinden et al., 17 Oct 2025).
Synthetic Data Generation and Augmentation: Conditioning on real initial frames and atomic or composed action prompts, the world model generates diverse, plausible video rollouts. The IDM produces pseudo-kinematics, enabling synthesis of paired video–action data at scale (He et al., 29 Dec 2025).
Adapting to Data Modalities: While current mainstream implementations focus on monocular or single-view modalities, strategies are being developed for multi-view, proprioceptive, force, and haptic inputs (Zbinden et al., 17 Oct 2025).

5. Limitations, Challenges, and Failure Modes

Despite significant progress, several fundamental challenges persist:

Causality and Physics Consistency: Even advanced video diffusion models (e.g., Veo-3, HunyuanVideo) exhibit a marked gap between visually plausible and causally consistent generations—tool trajectories, tissue deformations, and procedural logic often drift into implausible regimes over long rollouts (Chen et al., 3 Nov 2025, Turkcan et al., 16 Mar 2025).
Data Annotations and Coverage: Synthetic data via inverse dynamics is noisy relative to ground truth. Training data coverage (especially for rare or failure modes) directly impacts hallucination frequency and physics-inconsistent predictions. Balance between success/failure examples remains a target for further ablation (Zbinden et al., 17 Oct 2025, He et al., 29 Dec 2025).
Multi-Modal Sensing: Most current models leverage only RGB visual data, omitting force, tactile, and explicit haptic modalities critical to surgical biomechanics fidelity (Turkcan et al., 16 Mar 2025).
Generalization Across Embodiments: Task- and platform-specific fine-tuning is required; cross-domain generalization is non-trivial and necessitates additional data curation (He et al., 29 Dec 2025).
Computational Constraints: Realistic, high-resolution video diffusion models (e.g., HunyuanVideo, 13B parameters) are not yet real-time for clinical deployment; model compression is an ongoing effort (Turkcan et al., 16 Mar 2025).

6. Applications and Forward Directions

SurgWorld’s impact traverses a broad spectrum of applications:

Policy Learning and Sim2Real Transfer: Autonomous agents, vision–language–action policies, and model-based RL can be trained in SurgWorld simulators, leveraging synthetic paired video–action sets for improved generalization and reduced reliance on scarce real-robot data (He et al., 29 Dec 2025, Lin et al., 2024).
Automated Skill Assessment: Analysis of policy rollouts with expert-aligned or automatically learned discriminators (classifier probes) enables objective, reproducible evaluation of technique and procedural success (Zbinden et al., 17 Oct 2025, Turkcan et al., 16 Mar 2025).
Training, Education, and Benchmarking: Real-time, infinitely variable synthetic data support personalized VR/AR surgical training environments and robust procedural benchmarking (e.g., SurgVeo, SPP) (Chen et al., 3 Nov 2025, Turkcan et al., 16 Mar 2025).
Design of Next-Generation World Models: Roadmaps emphasize incorporation of structured surgical knowledge, explicit physics or biomechanics modules, multi-modal sensory fusion, domain adaptation for patient-specific anatomy, and multi-horizon procedural planning. The integration of RL agents directly into the learned environment to close the sim-to-real gap is explicitly prioritized (Zbinden et al., 17 Oct 2025, Chen et al., 3 Nov 2025, Koju et al., 3 Mar 2025).

7. Summary Table: Key SurgWorld Environments and Metrics

Environment	Data Type/Modality	Core Metric(s)	Notable Results
Cosmos-Surg-dVRK	RGB + kinematics	SR, ρ, MAE, ICC, MMRV, MBE	ρ=0.756, ICC=0.836
SurgVeo/SPP	High-res surgical video	SPP scores (1–5), Plausibility Gap	Δ_2(8s)≈1.94, Δ_4(8s)≈3.39
SurgWorld (Cosmos-P2.5)	RGB, Action–Text pairs	SR, FVD, MSE (policy)	SR=73.2% (synthetic+real, 5 demos)
GAS	Depth+mask (64×64×3)	SR, robustness tests	Real world SR 69%
Suturing Models	RGB, sub-stitch action	L2 Rec. Loss, Qualitative	HunyuanVideo L2=0.12, fine action
SurgWM	RGB (unsup. actions)	PSNR, SSIM, FVD	PSNR↑, SSIM↑ with GT actions

This synthesis demonstrates that SurgWorld, as both a conceptual and technical framework, provides the scaffolding for scalable, data-efficient, and causality-aware surgical agent development and assessment. Its trajectory is defined by the continual fusion of large-scale vision–language–action models, robust simulation, benchmarking by expert standards, and systematic targeting of the causal gaps that differentiate superficial mimicry from domain-expert competence (Zbinden et al., 17 Oct 2025, He et al., 29 Dec 2025, Chen et al., 3 Nov 2025, Turkcan et al., 16 Mar 2025, Koju et al., 3 Mar 2025, Lin et al., 2024).