Harmful Self-Preference Ratio (HSPP)

• Harmful Self-Preference Propensity Ratio (HSPP) is a metric that measures the frequency of self-preferential outputs that may lead to biased or harmful consequences.
• The ratio is calculated by comparing instances of self-referential behavior against total system outputs, serving as a diagnostic tool for bias identification.
• Elevated HSPP values highlight potential risks in system design, guiding targeted improvements in bias mitigation and performance refinement.

Strict Guideline Accuracy (SGA) is a family of evaluation protocols and metrics that quantify the proportion of outputs from automated systems or models that fully conform to a formally specified set of operational, clinical, or labeling guidelines. SGA enforces a high bar for compliance—requiring perfect format, evidence, logical consistency, and guideline-specific properties—making it a key metric for judging system trustworthiness in domains where regulatory or expert requirements are paramount, such as enterprise dialogue, clinical decision support, and structured annotation tasks.

1. Formal Definitions and Mathematical Construction

"Strict Guideline Accuracy" is formally defined as the fraction of outputs or decisions that satisfy all prescribed requirements of a guideline, encompassing both factual and structural constraints. The precise instantiation depends on the domain and the nature of compliance being measured.

In guideline-governed dialogue evaluation (CompliBench), the SGA metric is:

$$\mathrm{SGA} = \frac{1}{\sum_{i=1}^N (1 - v_i)} \sum_{i=1}^N (1 - v_i)\,\mathbf{1}[\hat{g}_i = g_i \wedge \hat{v}_i = 0]$$

where $N$ is the number of dialogue turns, $v_i$ is the ground-truth violation label (0 for compliant, 1 for violation), $\hat{v}_i$ is the judge's predicted violation flag, $g_i$ is the ground-truth governing guideline, and $\hat{g}_i$ is the predicted guideline label. Only compliant turns ($v_i=0$) are included, and both correct guideline attribution and violation status must match for a turn to be SGA-correct (Yang et al., 14 Apr 2026).

In multimodal medical reasoning (AD-Reasoning), SGA is operationalized as a composite rule-based verification:

• The output must conform to a strict schema (Reasoning, Diagnosis, Confidence fields present, with Confidence in {High, Medium, Low}).
• The diagnostic category, biomarkers, and clinical evidence must be fully accurate according to guideline rules.
• The reasoning chain must entail the diagnosis per an entailment model.

The sample is SGA-correct only when all subcriteria are perfectly satisfied, formally:

$$\mathrm{SGA} = \frac{\#\{\text{samples SGA-correct}\}}{\#\{\text{samples total}\}}$$

where SGA-correct refers to outputs passing all above tests (Chen et al., 25 Mar 2026).

2. Methodological Frameworks for SGA Enforcement

Enforcing strict guideline compliance requires architectural and algorithmic mechanisms tuned for high-fidelity rule following:

• Rule-Based Verifiers: Deterministic verifiers parse system outputs into component fields, extract entities or features, and apply finite-state or thresholded rules derived from external guidelines. For example, AD-Reasoning uses a multi-stage rule-based verifier that checks for schema compliance, correct diagnostic category and biomarker mentions, and structured evidence coverage. Logical entailment between reasoning and diagnosis is modeled using a pretrained NLI model (Chen et al., 25 Mar 2026).
• Iterative Critique-Refine Loops: In semantic segmentation under labeling guidelines, a two-agent Worker–Supervisor architecture iteratively critiques and refines outputs. The Worker segments instances, the Supervisor identifies deviations from guidelines (missing or spurious objects; boundary refinements), and RL-based controllers determine when sufficient compliance is reached (Vats et al., 4 Sep 2025).
• Reinforcement Learning with Verifiable Rewards: Policy optimization is conducted on verifiable reward signals that are directly tied to guideline adherence—e.g., in AD-Reasoning, rewards are given only when outputs use the correct schema, evidence is cited, and the reasoning is consistent with guidelines (Chen et al., 25 Mar 2026).
• Retrieval-Augmented Prompting: Scene- or context-relevant guidelines are retrieved using embedding search, and only these are provided for dynamic enforcement, preventing model confusion due to irrelevant or overwhelming rule sets (Vats et al., 4 Sep 2025).

3. SGA in Dialogue Compliance: CompliBench Case Study

CompliBench operationalizes SGA as a turn-level metric for LLM judges in multi-turn, guideline-rich enterprise dialogue domains (Yang et al., 14 Apr 2026). The pipeline injects controlled violations into synthetic user–agent dialogues, annotates the governing guideline for each turn, and evaluates whether a judge can both (a) avoid false positives (misclassifying compliant turns as violations) and (b) attribute the correct non-violation to the proper guideline. SGA is reported separately from violation-detection accuracy (on errorful turns) and conversation-completion accuracy (all turns must be correct).

Results illustrate key phenomena:

Model	Healthcare	Insurance	Airline
GPT-5	92.9%	91.9%	91.7%
Gemini-3-pro	92.4%	91.7%	87.7%
Qwen3-8B (SFT)	87.1%	89.1%	88.6%
Open-weight (untuned)	49.3%–65%	48.6%–65%	44.4%–65%

Fine-tuning on synthetic data for a single domain (e.g., Airline) generalizes robustly, with the SFT Qwen3-8B model achieving high SGA in unseen domains. SGA is sensitive to domain complexity: workflow overlap leads to scope misattribution errors, while implicit triggers in healthcare expose reasoning-chain limitations (Yang et al., 14 Apr 2026).

4. SGA in Multimodal Clinical Reasoning: AD-Reasoning Example

In AD-Reasoning, SGA aligns with full compliance to NIA-AA diagnostic guidelines. The composite reward structure, enforced by rule-based parsing, demands outputs provide:

• Correct diagnosis category without contradiction.
• All required biomarker evidence, with correct mention and valence.
• Coverage of four cognitive domains in reasoning.
• Exact adherence to a three-field output schema.
• Logical entailment between reasoning and diagnosis.

Empirically, SGA correlates with an improvement in overall diagnostic accuracy: rule-based reward tuning raises accuracy on the CN vs CI classification from 90.46% (no RFT) to 93.33% (full RFT), reflecting an estimated 2–3 percentage point increase in strict schema-conforming outputs. Limitations include rule-based verifier brittleness to phrasing and the need for improved yet interpretable reward mechanisms (Chen et al., 25 Mar 2026).

5. SGA in Guideline-Consistent Segmentation

Although the term SGA is not explicitly used in "Guideline-Consistent Segmentation via Multi-Agent Refinement," the architecture embodies SGA principles by rigorously minimizing the number of hard and soft violations against complex textual labeling guidelines. Segmentation outputs are iteratively critiqued against a curated set of inclusion/exclusion rules, and the termination of refinement is governed by a Q-learning policy that acts on a concise issue count ($I_t$), which collapses missed inclusions ($I_\mathrm{miss}$), false inclusions ($N$0), and low-impact refinements ($N$1) as $N$2. Masks achieving $N$3 are strictly guideline-consistent (Vats et al., 4 Sep 2025).

Evaluation on Waymo and ReasonSeg reveals that full-loop architectures surpass open-vocabulary segmentation methods, particularly under "full-guideline" (paragraph-length rule) settings—e.g., guideline-consistent segmentation achieves gIoU = 80.57%, cIoU = 86.70%, and mPrecision = 91.06% on Waymo vs. gIoU <24% and cIoU <19% for standard phrase-prompted baselines.

6. Domain-Specificity, Error Modes, and Limitations

SGA is sensitive to domain, guideline granularity, and task complexity:

• Dialogue Systems: SGA errors may arise from scope mis-attribution, semantic ambiguity in guideline triggers, or acceptance of non-equivalent behaviors. Complex workflow domains expose the system’s brittleness in mapping turns to correct guidelines (Yang et al., 14 Apr 2026).
• Clinical Reasoning: Brittleness to linguistic variation and partial evidence citation can prevent perfect SGA, even when outputs are clinically reasonable but not comprehensively justified per schema (Chen et al., 25 Mar 2026).
• Segmentation: The need for explicit, manual construction of guideline-compliant masks for ground truth remains a bottleneck. RL-based refinement policies must be carefully tuned to avoid over- or under-refinement (Vats et al., 4 Sep 2025).

A plausible implication is that SGA offers high specificity for detecting guideline adherence but is constrained by the determinism and coverage of the chosen verifier or rule set.

7. Connections, Alternatives, and Future Directions

SGA occupies a niche in machine learning evaluation that is stricter than ordinary accuracy, F1, or mIoU: it is explicitly designed to reflect operational or regulatory verifiability rather than just predictive correctness. It is most relevant when partial compliance or “close enough” performance is insufficient for deployment or oversight.

Potential directions include:

• Extending SGA to more flexible, learned schema—e.g., by using sequence classifiers or learned reward models tolerant to synonyms and phrasing variation (Chen et al., 25 Mar 2026).
• Transporting SGA logic to new domains such as ATN biomarker frameworks, aviation procedure compliance, or structured legal document analysis.
• Developing richer, scalable benchmarks with controllable flaw injection, as in CompliBench, to probe SGA at scale under adversarial or evolving guidelines (Yang et al., 14 Apr 2026).

Current research demonstrates that agents and judges optimized and evaluated for SGA achieve higher real-world reliability in settings with complex, evolving, or safety-critical guidelines, but that brittle rule-based enforcement and overstrictness remain active areas for refinement.