Strict Guideline Accuracy (SGA)

• SGA is a formal metric that quantifies the exact adherence of model predictions to prescribed guidelines, penalizing any deviation without awarding partial credit.
• It is applied across domains like dialogue systems, clinical diagnostics, and computer vision, where outputs must meet strict structural, schema, and content requirements.
• SGA employs rule-based verifiers and reinforcement learning to ensure high precision and auditability, thereby guiding model fine-tuning in high-stakes applications.

Strict Guideline Accuracy (SGA) is a formal metric for evaluating whether model outputs—typically text or annotation sequences—fully conform to the entirety of prescribed, context-specific guidelines. SGA quantifies the fraction of predictions that are not only factually correct, but also strictly satisfy all structural, content, and procedural requirements mandated by operational, clinical, or policy rules. Unlike standard accuracy, SGA is unforgiving: partial credit is not awarded, and any deviation from the stipulated guideline—be it a missing required field, incorrect classification, or structural anomaly—renders a prediction non-compliant. SGA has gained particular traction in domains demanding verifiability and auditability, such as clinical decision support, policy compliance in LLMs, and semantically-guided computer vision.

1. Formal Definitions and Mathematical Expression

SGA is defined and operationalized in several distinct but convergent formulations. In dialogue policy compliance, CompliBench formalizes SGA per conversation as follows (Yang et al., 14 Apr 2026):

$$\mathrm{SGA} = \frac{1}{\sum_{i=1}^N (1 - v_i)} \sum_{i=1}^N (1 - v_i) \mathbf{1}[\hat g_i = g_i \land \hat v_i = 0]$$

where:

• $N$ is the number of dialogue turns,
• $v_i \in \{0,1\}$ is the ground-truth violation flag for turn $i$, with $0$ indicating a compliant turn,
• $\hat v_i$ is the judge’s predicted violation flag,
• $g_i$ is the true applicable guideline key,
• $\hat g_i$ is the predicted guideline key.

A compliant turn is only counted as SGA-correct if $\hat v_i = 0$ (non-violation) and $\hat g_i = g_i$ (correct scope).

In the AD-Reasoning Alzheimer’s diagnostic setting, SGA is the proportion of samples for which all of the following are true (Chen et al., 25 Mar 2026):

• Output obeys exact schema (fields: Reasoning, Diagnosis, Confidence),
• Diagnostic label matches ground truth and has no contradictions,
• All required biomarkers are mentioned and correctly characterized,
• All four cognitive domains are named and minimally described,
• The reasoning narrative logically entails the diagnosis.

SGA thus generalizes as:

$N$0

2. Application Domains and Task Structure

Dialogue Compliance Adjudication

In CompliBench, SGA measures whether an LLM-based judge can, for each turn in a multi-turn task-oriented dialogue, (a) identify the correct applicable guideline and (b) recognize non-violations, explicitly penalizing both misattribution of guideline and over-flagging of false violations. SGA is calculated only on compliant (non-violating) turns, reflecting the requirement for high-precision guideline adherence in operational audit trails (Yang et al., 14 Apr 2026).

Multimodal Clinical Reasoning

AD-Reasoning operationalizes SGA as strict conformance to the NIA-AA criteria for Alzheimer’s diagnosis, enforced through a composite rule-based verifier that checks structure, biomarker evidence, cognitive-feature mentions, and logical consistency. SGA is assessed by post-hoc verification on a large-scale multimodal test set (AD-MultiSense), and used as both a training reinforcement signal and an evaluation metric (Chen et al., 25 Mar 2026).

Computer Vision: Segmentation Under Labeling Rules

In vision tasks involving guideline-driven segmentation, the exact SGA term is not always used, but “guideline-consistent” evaluation aligns closely. Here, segmentation outputs are iteratively refined until no guideline violations (such as missed or spurious object masks) remain, as quantified by an “issue count” that aggregates hard and soft errors. Standard metrics (gIoU, mask precision/recall) are reported only on test sets whose ground-truth annotations are themselves strictly guideline-consistent (Vats et al., 4 Sep 2025).

3. Scoring Procedures and Enforcement Mechanisms

Rule-Based Verification

SGA evaluation typically combines deterministic, rule-based verifiers and pretrained classifiers:

• Schema Checking: Output is parsed and only counted as SGA-correct if all prescribed fields are present, correctly typed, and formatted.
• Content Consistency: For clinical or policy outputs, references to biomarkers, workflow triggers, or regulated behaviors must exactly match ground-truth expectations.
• Entailment and Reasoning: Natural Language Inference (NLI) models are invoked to verify that free-text explanations logically entail the final decision (e.g., diagnosis).
• Reinforcement Learning with Verifiable Reward: Policies are fine-tuned with composite rewards representing each strict consistency requirement, with SGA serving as a maximized objective in group-relative policy optimization (GRPO) (Chen et al., 25 Mar 2026).

Data and Evaluation Protocols

• Split-by-turn Analysis: SGA is computed only over the subset of predictions representing compliant (non-faulty) behavior.
• Exhaustive Matching: Any deviation—omission, extraneous field, scope misattribution, or logic contradiction—negates SGA credit for that prediction.
• Aggregation: Results are typically reported as a percentage over domain-specific test splits or held-out dialog sets.

4. Empirical Results and Benchmarks

Dialogue Systems

Experimental SGA values in CompliBench show that off-the-shelf proprietary LLM-based judges (such as GPT-5, Gemini-3) achieve SGA in the 90–93% range across Healthcare, Insurance, and Airline domains. Open-source backbones like Qwen3-8B achieve sub-50% SGA without fine-tuning but reach 87–89% after specialized SFT. No classical reward model baseline exceeds 60% SGA (Yang et al., 14 Apr 2026).

Model	Healthcare SGA	Insurance SGA	Airline SGA
GPT-5	92.9%	91.9%	91.7%
Gemini-3-pro	92.4%	91.7%	87.7%
Qwen3-8B (SFT)	87.1%	89.1%	88.6%

Clinical Decision Models

SGA in AD-Reasoning is reflected in 2–3 percentage point improvements in accuracy on NIA-AA-conformant classifications after introducing strict rule-based verifiable rewards during reinforcement fine-tuning, compared to baselines trained without explicit guideline enforcement (Chen et al., 25 Mar 2026).

Segmentation

Guideline-consistent segmentation methods achieve gIoU values up to 80.6% on hand-curated, strictly guideline-adherent test sets, outperforming zero-shot and non-iterative baselines by wide margins (Vats et al., 4 Sep 2025). The SGA analogue here is the zero-issue count on manual evaluations.

5. Error Analysis and Domain Effects

SGA performance is sensitive to the complexity and ambiguity of application guidelines:

• Workflow-heavy domains: Insurance and Airline domains reveal many SGA errors due to scope misattribution and near-duplicate workflow steps.
• Conditional/semantic domains: Healthcare settings elicit SGA failures from reasoning-chain mistakes and incomplete conditional logic.
• Segmentation: Dense scenes or competing instance definitions drive persistent low-impact violations, even with iterative RL-based refinement (Vats et al., 4 Sep 2025).

Analysis of error types guides targeted fine-tuning and rule clarification strategies, as over-strict verifiers may undercount admissible linguistic or structural variants.

6. Limitations and Extensions

• Verifier brittleness: Rule-based SGA checkers may lack robustness to paraphrase and partially correct variants. In clinical settings, this can lead to underestimation of clinically usable outputs (Chen et al., 25 Mar 2026).
• Generalizability: Fine-tuned LLM judges for SGA generalize well between structurally similar domains (e.g., Insurance ↔ Airline), but may require retraining for more diverse guideline types or output schemas (Yang et al., 14 Apr 2026).
• Guideline evolution: In computer vision, changing or extending guideline sets historically required model retraining; multi-agent or RL approaches promise training-free compliance under evolving textual rules (Vats et al., 4 Sep 2025).

Proposed extensions include learnable reward components, more nuanced (softer) schema constraints, and enhanced handling of free-text clinical evidence and operational triggers. SGA as a metric is likely to persist or expand as model auditing and traceability requirements grow in regulated or high-stakes domains.

7. Distinctions from Related Metrics

While SGA is closely related to overall accuracy, it is substantially more stringent. Standard accuracy metrics permit partial matches or isolated correctness, while SGA credits only predictions that satisfy all explicit and implicit guideline requirements in aggregate. Baseline and ablation analyses consistently show a non-trivial delta—often several percentage points—between SGA and looser accuracy or recall measures, highlighting SGA’s value as a high-precision, high-sensitivity compliance metric in settings where partial conformity is insufficient (Chen et al., 25 Mar 2026, Yang et al., 14 Apr 2026).