---
title: Phase-Goal Satisfiability in Surgery
url: https://www.emergentmind.com/topics/phase-goal-satisfiability
type: topic
---

# Phase-Goal Satisfiability in Surgery

Phase-goal satisfiability is an evaluation paradigm for surgical planning tasks in which the validity of a plan is defined by its compliance with expert-encoded procedural rules specific to clinical phases, rather than by raw sequence similarity to reference plans. Developed formally in the context of safety-critical surgical strategy assessment, it provides a precise and binary decision metric that reflects clinical correctness as judged by adherence to necessary procedural constraints, as opposed to surface-level similarity. This approach underpins the "SurgGoal: Rethinking Surgical Planning Evaluation via Goal-Satisfiability" framework, with direct applications to the assessment of vision-language models (VLMs), especially video foundation models, for surgical phase and step planning tasks [2601.10455].

## 1. Formal Definition and Rule Structure

Let $P = (s_1, s_2, \dots, s_k)$ denote a candidate plan, a finite ordered sequence of annotated surgical steps $s_i$ from a global vocabulary $S$. Consider a target surgical phase $G \in \{P_1, \dots, P_{11}\}$—each with a set of associated rules $R_G$ as defined by experts on the basis of procedural correctness.

The binary satisfiability function is formalized as:
\[
SAT(P, G) =
\begin{cases}
1, & \text{if } P \text{ satisfies every rule in } R_G \\
0, & \text{otherwise.}
\end{cases}
\]

Decomposing $R_G$:
- $Req(G) \subseteq S$: required steps, all of which must appear in $P$
- $Allow(G) \subseteq S$: ancillary steps permitted in $P$ without invalidating it
- $Dep(G) = \{(a, b)\}$: ordered pairs imposing partial order (e.g., $a$ must precede $b$)
- $Proh(G) = \{(c, d)\}$: prohibitive constraints forbidding some orderings or step co-occurrences

The plan $P$ is phase-goal satisfiable if and only if:
- $Req(G) \subseteq \{s_1, \dots, s_k\}$
- $\{s_1, \dots, s_k\} \subseteq Req(G) \cup Allow(G)$
- For all $(a, b) \in Dep(G)$, $\mathrm{pos}_P(a) < \mathrm{pos}_P(b)$
- No violation of $Proh(G)$

$\mathrm{pos}_P(x)$ denotes the (first) index of $x$ in $P$ (or $\infty$ if $x \notin P$).

## 2. Construction of Expert Rules and Procedural Encodings

Rule construction is phase-specific and procedure-specific, involving extensive expert arbitration. Each phase $G$ receives a quadruple $(Req(G), Allow(G), Dep(G), Proh(G))$.

For example, for phase $P_5$ ("Anastomosis Test"):
\[
\begin{aligned}
Req(P_5) &= \{S_{22}, S_{23}, S_{24}, S_{25}\} \\
Allow(P_5) &\supset \{S_3\ (\text{retractor placement}), S_{39}\ (\text{hemostasis}), S_{40}\ (\text{irrigation})\} \\
Dep(P_5) &= \{(S_{23}, S_{24}), (S_{24}, S_{25})\} \\
Proh(P_5) &= \{(S_{39}\ \text{or } S_{40},\ \text{before } S_{23}), (\text{any core } S_{2*},\ \text{after last } S_{25})\}
\end{aligned}
\]

The benchmark in [2601.10455] covers 50 unique dependencies and prohibitions spanning 11 surgical phases and 45 step types.

## 3. Mathematical Formulation of Error Classes

To systematically describe invalid plans, rule violations are subtyped as:
- **Order Errors (OE):** $SAT(P, G) = 0$ due exclusively to $\exists (a, b) \in Dep(G): \mathrm{pos}_P(a) > \mathrm{pos}_P(b)$
- **Content Errors (CE):** $SAT(P, G) = 0$ due exclusively to $Req(G) \not\subset P$ or extraneous steps $s \notin Req(G) \cup Allow(G)$ present in $P$
- **Both Errors (BE):** Simultaneous order and content error violations

Define indicator functions:
\[
\begin{aligned}
Err_{order}(P,G) &= 1 \iff \exists (a,b)\in 
Dep(G): \mathrm{pos}_P(a) > \mathrm{pos}_P(b) \\
Err_{content}(P,G) &= 1 \iff [Req(G) \not\subset P \vee (P \setminus (Req(G)\cup Allow(G)) \neq \emptyset)]
\end{aligned}
\]
with OE, CE, BE assigned per the conjunctions of these indicators.

## 4. Meta-Evaluation Benchmark and Protocol

The multicentric benchmark (MultiBypass140) comprises 140 surgical videos annotated for 11 phases and 45 steps. Sequence pools include:
- **Correct Sequences (N = 191):** Clinically faithful but varied step-orderings and inclusion of optional ancillaries, all rule-compliant.
- **Incorrect Sequences (N = 199):** Perturbed from correct examples to induce OE, CE, or BE labels.

The meta-evaluation protocol:
- Input: $(P, G)$ pairs
- Output: Valid/invalid classification by $SAT(P, G)$ and comparison to reference-based metrics
- Reporting: Overall and stratified accuracy by validity/error type

## 5. Comparison with Sequence Similarity Metrics

Table: Accuracy of Sequence Similarity vs. Rule-Based Metrics

| Subset  | NED    | JIS    | ROA    | Rule-Based |
|---------|--------|--------|--------|------------|
| Valid   | 18.8%  | 40.3%  | 93.2%  | 100.0%     |
| OE      | 87.3%  | 46.5%  | 11.3%  | 100.0%     |
| CE      | 86.8%  | 85.3%  | 17.6%  | 100.0%     |
| BE      | 96.7%  | 85.0%  | 20.0%  | 100.0%     |
| Invalid | 89.9%  | 71.4%  | 17.1%  | 100.0%     |

Empirical results indicate:
- Normalized Edit Distance (NED) and Jaccard Index on Sequences (JIS) exhibit high accuracy for detecting invalid plans but fail to recognize valid procedural variants, resulting in high false negatives.
- Relative Order Accuracy (ROA) is permissive, yielding high true-positive rates but missing critical order errors.
- The phase-goal satisfiability rule-based metric is, by construction, perfectly aligned with clinical validity.

## 6. Application to Video-LLM Planning and Knowledge Injection

Surgical planning LLMs (VideoLLaMA3, LLaVA-NeXT, Qwen2.5-VL, HuluMed, Lingshu) were evaluated under three task regimes:
- **Task 1:** Fully unconstrained, model operates on raw data and incomplete step history.
- **Task 2:** Control for perception ambiguity by giving the ground-truth current step.
- **Task 3:** Incrementally inject knowledge—structural (phase-step hierarchy), semantic (natural language descriptions), or both.

Planning validity is scored via $SAT(P_{pred} \Vert \mathrm{history}, G)$.

Key findings:
- Without external knowledge, both step recognition and phase completion remain poor (step recognition < 40%, phase completion < 5% for 7B models).
- Structural knowledge prompts yield the largest improvement (up to $\sim 65\%$ current phase accuracy, and up to $\sim 64\%$ next-phase planning for smaller models, reaching $89.2\%$ for Qwen-32B with all available knowledge).
- Semantic-only prompts are insufficient; mixed structural and semantic cues are only synergistic at large model scales.
- Models tend to hallucinate plausible but invalid procedures when inadequately constrained.

## 7. Limitations and Prospective Directions

The construction of a rule-based checker is intimately tied to manual, procedure-specific rule engineering. This inhibits generalization to new procedures, clinical sites, or institutions. A future extension could involve LLM-assisted or semi-automated extraction of phase-step constraints.

The binary nature of phase-goal satisfiability ($SAT=0/1$) excludes measures of plan efficiency, compactness, or optimality among multiple valid strategies. A plausible implication is that cost-based or graded metrics may be needed for finer-grained plan assessment.

Finally, applicability is currently shown on a single, meticulously annotated dataset. Generalization to other domains requires new strategies for phase-step hierarchy acquisition or weak supervision.

---

Phase-goal satisfiability redefines the standard of evaluation in surgical planning tasks from sequence alignment to rule-based constraint satisfaction, aligning computational evaluation with clinical requirements and exposing shortcomings in sequence-centric metrics. Its integration with guided LLM development permits systematic progress in the construction of planning agents that respect surgical logic and safety mandates [2601.10455].

Source: https://www.emergentmind.com/topics/phase-goal-satisfiability