Phase-Goal Satisfiability in Surgery

• Phase-goal satisfiability is an evaluation paradigm that uses expert-encoded procedural rules to determine surgical plan validity.
• It distinguishes itself from sequence similarity metrics by emphasizing required steps, order constraints, and prohibitions specific to surgical phases.
• The method underpins advanced assessments of Video-LLM models, revealing model limitations and guiding improvements via structured knowledge injection.

Phase-goal satisfiability is an evaluation paradigm for surgical planning tasks in which the validity of a plan is defined by its compliance with expert-encoded procedural rules specific to clinical phases, rather than by raw sequence similarity to reference plans. Developed formally in the context of safety-critical surgical strategy assessment, it provides a precise and binary decision metric that reflects clinical correctness as judged by adherence to necessary procedural constraints, as opposed to surface-level similarity. This approach underpins the "SurgGoal: Rethinking Surgical Planning Evaluation via Goal-Satisfiability" framework, with direct applications to the assessment of vision-LLMs (VLMs), especially video foundation models, for surgical phase and step planning tasks (Li et al., 15 Jan 2026).

1. Formal Definition and Rule Structure

Let $P = (s_1, s_2, \dots, s_k)$ denote a candidate plan, a finite ordered sequence of annotated surgical steps $s_i$ from a global vocabulary $S$. Consider a target surgical phase $G \in \{P_1, \dots, P_{11}\}$—each with a set of associated rules $R_G$ as defined by experts on the basis of procedural correctness.

The binary satisfiability function is formalized as: $$SAT(P, G) = \begin{cases} 1, & \text{if } P \text{ satisfies every rule in } R_G \ 0, & \text{otherwise.} \end{cases}$$

Decomposing $R_G$:

• $Req(G) \subseteq S$: required steps, all of which must appear in $P$
• $Allow(G) \subseteq S$: ancillary steps permitted in $s_i$0 without invalidating it
• $s_i$1: ordered pairs imposing partial order (e.g., $s_i$2 must precede $s_i$3)
• $s_i$4: prohibitive constraints forbidding some orderings or step co-occurrences

The plan $s_i$5 is phase-goal satisfiable if and only if:

• $s_i$6
• $s_i$7
• For all $s_i$8, $s_i$9
• No violation of $S$0

$S$1 denotes the (first) index of $S$2 in $S$3 (or $S$4 if $S$5).

2. Construction of Expert Rules and Procedural Encodings

Rule construction is phase-specific and procedure-specific, involving extensive expert arbitration. Each phase $S$6 receives a quadruple $S$7.

For example, for phase $S$8 ("Anastomosis Test"): $S$9

The benchmark in (Li et al., 15 Jan 2026) covers 50 unique dependencies and prohibitions spanning 11 surgical phases and 45 step types.

3. Mathematical Formulation of Error Classes

To systematically describe invalid plans, rule violations are subtyped as:

• Order Errors (OE): $G \in \{P_1, \dots, P_{11}\}$0 due exclusively to $G \in \{P_1, \dots, P_{11}\}$1
• Content Errors (CE): $G \in \{P_1, \dots, P_{11}\}$2 due exclusively to $G \in \{P_1, \dots, P_{11}\}$3 or extraneous steps $G \in \{P_1, \dots, P_{11}\}$4 present in $G \in \{P_1, \dots, P_{11}\}$5
• Both Errors (BE): Simultaneous order and content error violations

Define indicator functions: $G \in \{P_1, \dots, P_{11}\}$6 with OE, CE, BE assigned per the conjunctions of these indicators.

4. Meta-Evaluation Benchmark and Protocol

The multicentric benchmark (MultiBypass140) comprises 140 surgical videos annotated for 11 phases and 45 steps. Sequence pools include:

• Correct Sequences (N = 191): Clinically faithful but varied step-orderings and inclusion of optional ancillaries, all rule-compliant.
• Incorrect Sequences (N = 199): Perturbed from correct examples to induce OE, CE, or BE labels.

The meta-evaluation protocol:

• Input: $G \in \{P_1, \dots, P_{11}\}$7 pairs
• Output: Valid/invalid classification by $G \in \{P_1, \dots, P_{11}\}$8 and comparison to reference-based metrics
• Reporting: Overall and stratified accuracy by validity/error type

5. Comparison with Sequence Similarity Metrics

Table: Accuracy of Sequence Similarity vs. Rule-Based Metrics

Subset	NED	JIS	ROA	Rule-Based
Valid	18.8%	40.3%	93.2%	100.0%
OE	87.3%	46.5%	11.3%	100.0%
CE	86.8%	85.3%	17.6%	100.0%
BE	96.7%	85.0%	20.0%	100.0%
Invalid	89.9%	71.4%	17.1%	100.0%

Empirical results indicate:

• Normalized Edit Distance (NED) and Jaccard Index on Sequences (JIS) exhibit high accuracy for detecting invalid plans but fail to recognize valid procedural variants, resulting in high false negatives.
• Relative Order Accuracy (ROA) is permissive, yielding high true-positive rates but missing critical order errors.
• The phase-goal satisfiability rule-based metric is, by construction, perfectly aligned with clinical validity.

6. Application to Video-LLM Planning and Knowledge Injection

Surgical planning LLMs (VideoLLaMA3, LLaVA-NeXT, Qwen2.5-VL, HuluMed, Lingshu) were evaluated under three task regimes:

• Task 1: Fully unconstrained, model operates on raw data and incomplete step history.
• Task 2: Control for perception ambiguity by giving the ground-truth current step.
• Task 3: Incrementally inject knowledge—structural (phase-step hierarchy), semantic (natural language descriptions), or both.

Planning validity is scored via $G \in \{P_1, \dots, P_{11}\}$9.

Key findings:

• Without external knowledge, both step recognition and phase completion remain poor (step recognition < 40%, phase completion < 5% for 7B models).
• Structural knowledge prompts yield the largest improvement (up to $R_G$0 current phase accuracy, and up to $R_G$1 next-phase planning for smaller models, reaching $R_G$2 for Qwen-32B with all available knowledge).
• Semantic-only prompts are insufficient; mixed structural and semantic cues are only synergistic at large model scales.
• Models tend to hallucinate plausible but invalid procedures when inadequately constrained.

7. Limitations and Prospective Directions

The construction of a rule-based checker is intimately tied to manual, procedure-specific rule engineering. This inhibits generalization to new procedures, clinical sites, or institutions. A future extension could involve LLM-assisted or semi-automated extraction of phase-step constraints.

The binary nature of phase-goal satisfiability ($R_G$3) excludes measures of plan efficiency, compactness, or optimality among multiple valid strategies. A plausible implication is that cost-based or graded metrics may be needed for finer-grained plan assessment.

Finally, applicability is currently shown on a single, meticulously annotated dataset. Generalization to other domains requires new strategies for phase-step hierarchy acquisition or weak supervision.

---

Phase-goal satisfiability redefines the standard of evaluation in surgical planning tasks from sequence alignment to rule-based constraint satisfaction, aligning computational evaluation with clinical requirements and exposing shortcomings in sequence-centric metrics. Its integration with guided LLM development permits systematic progress in the construction of planning agents that respect surgical logic and safety mandates (Li et al., 15 Jan 2026).