Surgical Plausibility Pyramid (SPP)

• Surgical Plausibility Pyramid (SPP) is a hierarchical framework that decomposes plausibility into four tiers—visual, operational, environmental, and intent—to evaluate surgical video outputs.
• It combines expert assessments with a quantitative scoring system to differentiate between surface realism and deep procedural reasoning in model outputs.
• Quantitative findings reveal a significant gap between high visual scores and poor performance in higher-level surgical reasoning, guiding future model improvements.

The Surgical Plausibility Pyramid (SPP) is a hierarchical framework for evaluating the outputs of generative models in surgical video domains. By decomposing "plausibility" into four ascending tiers—from basic visual realism to complex procedural reasoning—the SPP provides a clinically grounded, reproducible scoring metric suite. Its primary purpose is to diagnose the performance of video foundation models (e.g., Veo-3) with respect to both surface-level fidelity and deep, causal surgical understanding, exposing critical deficits in current world-modeling approaches for high-stakes, specialized environments (Chen et al., 3 Nov 2025).

1. Motivation and Conceptual Foundation

Surgical procedure video simulation requires more than photorealistic imagery; it demands adherence to the causal biomechanics, anatomical logic, and procedural strategy intrinsic to expert practice. Generic video generation systems may achieve visually convincing outputs, yet they frequently violate surgical logic, such as incorrect tool usage or implausible tissue reaction. The SPP addresses these challenges by formalizing four qualitatively distinct levels of plausibility, enabling systematic, expert-curated discrimination between surface realism and true world-model comprehension. The intent is to "open the black box" of model outputs, supporting nuanced benchmarking and iterative improvement in the development of surgical foundation models (Chen et al., 3 Nov 2025).

2. Structure and Definitions of SPP’s Four-Tier Hierarchy

The SPP is a four-level pyramid, each tier corresponding to a deeper and more abstract aspect of plausibility. Evaluation is performed on a 5-point integer scale per dimension, detailed as follows:

Tier	Level Name	Focus and Key Criteria
1	Visual Perceptual Plausibility	Image clarity, color/lighting, tissue texture, absence of artifacts. 5: indistinguishable from real; 1: severe distortion/disappearance.
2	Instrument Operation Plausibility	Correct tool type, realistic trajectories, grasping/cutting mechanics. 5: perfect expert-level manipulation; 1: impossible tools/actions.
3	Environment Feedback Plausibility	Tissue response to actions: deformation, bleeding, abidance by biomechanical principles. 5: correct volumetric bleed, anatomical deformation; 1: absent/physically impossible response.
4	Surgical Intent Plausibility	Procedural goal alignment, evidence of clinical reasoning. 5: coherent, stage-appropriate actions; 1: incoherent or protocol-violating actions.

Each dimension is supported by a detailed rubric (see Appendix H of (Chen et al., 3 Nov 2025)), ensuring reproducibility and comparability across expert raters and experimental settings.

3. Formal Scoring Methodology

Let $i$ index a video sample, $j\in\{1,2,3,4\}$ the SPP dimensions, $t\in\{1,3,8\}$ seconds the rating timepoints, and $r\in\{1,2\}$ the two independent expert raters per track. The scoring protocol is defined mathematically as follows:

• Individual rating: $s_{i,j,t}^{(r)}\in\{1,2,3,4,5\}$.
• Per-sample/dimension/time mean: $$\overline s_{i,j,t} = \frac{1}{2}\sum_{r=1}^2 s_{i,j,t}^{(r)}$$.
• Aggregate (track-wide) dimension score: $$\mu_{j,t} = \frac{1}{N}\sum_{i=1}^N \overline s_{i,j,t}$$, $\sigma_{j,t}$ is the corresponding standard deviation.
• Sample-level SPP aggregate: $$S_{i}(t) = \frac{1}{4}\sum_{j=1}^4 \overline s_{i,j,t}$$.
• Grand-mean/SD over all $N$ samples and timepoints: $\mu_{\rm overall}$ and $\sigma_{\rm overall}$.

This methodology enables both fine-grained and aggregate analysis, supporting rigorous comparisons across models, tasks, and prompt conditions.

4. Expert Assessment Protocol

Assessment of generative model outputs is performed by a panel of four board-certified surgeons. Two laparoscopic surgery experts rate the laparoscopic video track (N=18), and two neurosurgeons rate the neurosurgical track (N=32). For each video, two independent surgeons provide ratings across all four SPP dimensions at three key timepoints (1, 3, and 8 seconds), referencing the true 8-second continuation alongside the generated output. Inter-rater reliability is quantified via standard deviation (typical $\sigma\approx 0.04$–$0.47$ for Visual Perceptual Plausibility, higher for deeper dimensions), with generally low disagreement at the visual level and greater variance for high-level surgical reasoning. Statistical testing in the study indicates no significant improvement from baseline to stage-aware prompting ($p>0.05$), confirming the robustness of repeated-measures comparisons (Chen et al., 3 Nov 2025).

5. Quantitative Findings: The Plausibility Gap

The SPP’s utility is demonstrated in the evaluation of Veo-3. Results highlight a marked "plausibility gap"—a divergence between high scores for visual realism and poor performance in action, consequence, and intent. Key results for the baseline prompt condition:

Laparoscopic Track (Baseline Prompt):

SPP Dimension	1s	8s
Visual Perceptual Plausibility	3.72 ± 0.24	3.56 ± 0.31
Instrument Operation Plausibility	3.36 ± 0.20	1.78 ± 0.00
Environment Feedback Plausibility	3.06 ± 0.08	1.64 ± 0.12
Surgical Intent Plausibility	3.11 ± 0.16	1.61 ± 0.16

Neurosurgery Track (Baseline Prompt):

SPP Dimension	1s	8s
Visual Perceptual Plausibility	3.88 ± 0.09	3.41 ± 0.22
Instrument Operation Plausibility	2.77 ± 0.02	1.75 ± 0.04
Environment Feedback Plausibility	2.84 ± 0.09	1.78 ± 0.18
Surgical Intent Plausibility	2.03 ± 0.09	1.13 ± 0.04

Findings indicate that while appearance plausibility remains moderately high (>3.4), the higher tiers (instrument operation, environment feedback, intent) degrade sharply, often approaching “severe violation” (score ≈ 1.5) by 8 seconds. The "plausibility gap" denotes the ≈1.5–2.5 point deficit between the base (appearance) tier and the upper (causal/procedural) tiers, persisting across surgical specialties. Stage-aware prompting yields negligible improvements (<0.1 points average), further underscoring that superficial context augmentation does not bridge the knowledge gap (Chen et al., 3 Nov 2025).

6. Extensions, Generalization, and Prospects

The SPP offers a modular, hierarchical template applicable to any domain in which mere visual fidelity fails to capture true functional or causal correctness. Possible future directions include:

• Expansion of sub-tiers (e.g., decomposing Instrument Operation into subcategories such as "grasping" vs. "cutting").
• Integration of objective, automated metrics—such as tool-tracking accuracy or biomechanical deformation tests—alongside expert qualitative ratings.
• Adaptive weighting of SPP tiers tailored to specific downstream functions (e.g., elevating Intent Plausibility in educational simulators).
• Generalization to adjacent interventional fields (e.g., interventional cardiology, endoscopy) via domain-specific redefinition of scoring criteria.
• Incorporation of established inter-rater reliability statistics (e.g., Cohen’s κ, ICC) and hypothesis testing frameworks (e.g., repeated-measures ANOVA) to standardize cross-model comparison.

A plausible implication is that the SPP could serve both as a diagnostic tool for research communities benchmarking world-modeling progress and as a structural guide for the design of next-generation models capable of reasoning across the appearance–action–consequence–intent hierarchy, with substantial relevance for safety-critical domains (Chen et al., 3 Nov 2025).