Rule-Based Goal-Satisfiability Metrics

• Rule-based goal-satisfiability metrics are evaluation measures that verify outputs by checking adherence to expert-defined rules and procedural constraints.
• They are used in high-stakes fields like surgical planning, where only plans meeting all required, allowed, and dependency rules are considered correct.
• Comparative studies indicate that rule-based metrics outperform similarity-based methods by detecting order and content errors with 100% accuracy.

A rule-based goal-satisfiability metric is an evaluation formalism in which the quality or correctness of a structured output—such as a plan, program, or generated sequence—is determined by verifying whether it satisfies a set of explicitly defined logical or procedural rules. Unlike surface-level similarity or soft-matching metrics, this approach assesses outputs based on their adherence to critical constraints and dependencies specified by expert domain knowledge. Rule-based goal-satisfiability metrics are increasingly recognized as the gold standard for high-precision evaluation in complex, high-stakes tasks such as surgical planning, where correctness is defined not by resemblance but by conformity to procedural doctrine and fulfilment of task-critical goals (Li et al., 15 Jan 2026).

1. Formal Definition and Core Principles

The formal basis of a rule-based goal-satisfiability metric is the definition of task correctness as logical satisfaction of a goal state under a set of expert-defined constraints. Consider a setting where each instance comprises:

• A task phase $P$ (e.g., a surgical phase such as "anastomosis test")
• $R(P)$: required steps
• $A(P)$: additionally allowed (ancillary) steps
• $D(P)$: set of precedence (dependency) relations (pairs of steps, $(s_a, s_b)$ with $s_a$ preceding $s_b$)
• $H(P)$: set of prohibitive order relations

Given a candidate sequence $S = \langle s_1, ..., s_n \rangle$, $S$ is deemed to satisfy the goal of phase $R(P)$0 ($R(P)$1) if:

1. Required-step coverage: $R(P)$2
2. No forbidden content: $R(P)$3
3. Procedural dependencies: $R(P)$4 if $R(P)$5 then $R(P)$6
4. Prohibitive orderings: $R(P)$7 the forbidden ordering (e.g., $R(P)$8) does not occur, if the relation applies

A checker $R(P)$9 returns 1 if $A(P)$0, 0 otherwise. This binary labeling induces an exact notion of plan satisfiability (Li et al., 15 Jan 2026).

2. Design and Implementation in Meta-Evaluation Benchmarks

Rule-based goal-satisfiability metrics form the evaluative backbone of multicentric meta-evaluation suites where correctness must be measured across diverse sources, annotators, or institutional protocols. Relevant benchmark design involves:

• Expert-Defined Rule Sets: Comprehensively enumerated required/allowed steps, dependencies, and prohibitive orderings for each phase/task, directly encoded by domain experts.
• Construction of Benchmark Data: Sampling (phase, sequence) pairs and expert classification into correct and distinct error classes, such as order errors (OE), content errors (CE), or both (BE).
• Multicenter Consistency: Ensuring that rule sets and sequence annotations support inter-institutional evaluation by encoding allowable procedural variations and acknowledging site-specific ancillary actions without compromising critical dependencies.

For example, in the MultiBypass140 dataset across five international surgical centers, each (phase, sequence) pair has been manually classified by surgeons, and the associated rules codified for reproducible, cross-center testing (Li et al., 15 Jan 2026).

3. Comparative Evaluation against Sequence-Similarity Metrics

Rule-based goal-satisfiability fundamentally differs from sequence similarity or order-based metrics such as Normalized Edit Distance (NED), Jaccard Index on Sequences (JIS), or Relative Order Accuracy (ROA):

• NED/JIS penalize all deviations from a fixed reference, including valid procedural variants, and can miss procedural dependency violations if step content is shared.
• ROA is sensitive to gross order disruptions but cannot identify subtle illegal orderings or omissions critical to downstream safety or correctness.
• Only rule-based checkers are explicitly aligned with domain-validity: for a legal but non-reference sequence, rule-based metrics assign full credit, while similarity-based metrics do not.

Key empirical findings demonstrate that, across five clinical centers and multiple error types, only rule-based metrics achieve 100% discrimination on both correct and error cases, while similarity-based metrics exhibit high false negative and false positive rates, failing to meet the precision required for deployable systems in expert-driven environments (Li et al., 15 Jan 2026).

Metric	Valid Acc	Order-Err Acc	Content-Err Acc
NED	18.8%	87.3%	86.8%
JIS	40.3%	46.5%	85.3%
ROA	93.2%	11.3%	17.6%
Rule Checker	100.0%	100.0%	100.0%

4. Evaluation Metrics: Precision, Recall, F₁ and Alignment with Ground Truth

Rule-based goal-satisfiability enables clear definition of meta-evaluation metrics for comparing candidate evaluation protocols:

Given a candidate evaluation metric $A(P)$1 (not necessarily rule-based), precision, recall, and F₁ with respect to the rule-based ground truth ($A(P)$2) can be defined as:

• $A(P)$3: $A(P)$4 and $A(P)$5 (true positives)
• $A(P)$6: $A(P)$7 and $A(P)$8 (false positives)
• $A(P)$9: $D(P)$0 and $D(P)$1 (false negatives)
• $D(P)$2
• $D(P)$3
• $D(P)$4

By grounding evaluation in rule-based validity, benchmarks can objectively compare the diagnostic utility of other metrics or LLM-based scoring schemes (Li et al., 15 Jan 2026).

5. Impact on System Evaluation and Model Development

Deployment of rule-based goal-satisfiability metrics exposes systematic pathologies and capabilities of both classical and neural-based planning systems:

• In surgical planning, perception bottlenecks (step misrecognition) and unconstrained text generation by Video-LLMs are directly revealed as rule violations.
• Structural knowledge injection (e.g., explicit phase–step hierarchies) improves model performance under rule-based metrics, while semantic/narrative prompts alone are unreliable, with marked scaling dependence.
• Cross-center benchmarking validity is maintained: observed outcome rankings (rule-based ≫ sequence similarity ≫ LLM judges) and benefit from structural constraints are robust to institutional data distribution.

For example, 32B LLMs with injected structural knowledge attain next-phase completion accuracies up to ≈89%, whereas surface-matching metrics understate these gains or misrepresent errors (Li et al., 15 Jan 2026).

6. Multicentric Benchmarks and Generalization

Rule-based goal-satisfiability metrics are essential for multicentric meta-evaluation benchmarks, especially where:

• Domain-specific rules can be explicitly enumerated (e.g., in medical, legal, or protocol-driven settings).
• Inter-institutional comparison, reproducibility, and extension to new sites or variations are required.
• Diagnostic depth is needed to differentiate error types (e.g., order, content, and hybrid mistakes) with maximal precision.

The introduction of richly annotated, rule-grounded test beds provides a path forward for robust evaluation of generative and decision-making models. Only through exact alignment with procedural constraints can benchmarks support diagnostic analyses, cross-center comparability, and actionable insight for model improvement (Li et al., 15 Jan 2026).

7. Broader Significance and Recommendations

The adoption of rule-based goal-satisfiability as a meta-evaluation reference represents a shift from superficial, reference-based judgment to principled, domain-aligned correctness. The main recommendations emerging from recent multicentric benchmarks are:

• Use rule-based checkers—anchored by comprehensive expert knowledge—as the definitive meta-evaluation reference for structured, safety-critical outputs.
• Employ these metrics to quantify both within-center and cross-site reliability and to highlight model failure modes not visible under similarity-only metrics.
• Combine structural (e.g., taxonomy/timeline) and semantic knowledge during model training and inference to maximize compliance with procedural rules, especially in large-scale models and multicenter deployments.
• Extend the rule-based satisfiability approach to other domains where correctness is defined by explicit domains of allowed, required, and forbidden actions, beyond surgical planning to robotics, legal reasoning, and multi-step scientific protocols (Li et al., 15 Jan 2026).

Rule-based goal-satisfiability metrics thus embody a paradigm for evaluation that is precise, interpretable, and directly aligned with the domain’s ground truth—an indispensable foundation for the next generation of trustable AI systems in high-stakes applications.