Beyond Accuracy: Measuring Logical Compliance of Predictive Models

Abstract: Machine learning models are predominantly evaluated through predictive performance metrics such as ranking quality, prediction error, or classification accuracy. While these metrics effectively quantify how closely predictions match the ground truth, they do not assess whether model outputs respect predefined logical or domain-specific constraints. In high-stakes applications, including healthcare, finance, and autonomous systems, logical consistency can be as critical as predictive accuracy, yet no standard metric captures this dimension. We introduce the Rule Violation Score (RVS), a complementary evaluation metric that quantifies the extent to which a predictive model respects a given set of logical rules, independently of predictive accuracy. RVS treats hard rules (strict constraints) and soft rules (statistical regularities) differently, can be evaluated on any dataset and on any predictive model expressed over a relational vocabulary, and can be computed using SQL queries that are automatically generated for Horn rules. Beyond evaluating models, RVS can also evaluate the logical consistency of training datasets and help identify poorly defined rules. We evaluate RVS on three benchmarks covering knowledge graph link prediction and relational regression, including rule-based, embedding-based, and neuro-symbolic predictive models. Our results demonstrate that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance, revealing differences in model behavior that standard metrics fail to capture.

• The paper introduces Rule Violation Score (RVS), a ground-truth-independent metric that measures whether model predictions comply with hard and soft logical rules across knowledge graphs, relational databases, and regression tasks.
• Experiments show that predictive accuracy and logical compliance can diverge sharply: on Family, CompGCN achieved higher MRR than UniKER (0.923 vs. 0.867) but produced 425 versus 121 hard-rule violations.
• The paper demonstrates that dataset contradiction rates can audit rule quality before model evaluation, while RVS reveals compliance differences such as ExpressGNN’s zero hard-rule violations and 0.971 soft-rule score on FB15k-237.

Overview

The paper introduces the Rule Violation Score (RVS), an evaluation metric that quantifies the extent to which a predictive model's outputs respect a set of logical rules, independently of predictive accuracy (2606.20208). The authors' central claim is that conventional metrics—MRR, Hits@$k$, MAE, $R^2$, accuracy—measure only agreement with ground-truth labels and are silent on whether predictions satisfy domain-specific constraints. In high-stakes domains such as healthcare, finance, and autonomous systems, this omission can matter: two models with comparable predictive scores may differ sharply in logical consistency, and standard evaluation pipelines cannot detect the difference.

Motivation and design requirements

The motivating example is a small kinship dataset where two models both fail a link-prediction query wife(Alice, ?). One model predicts Alice's brother-in-law (incorrect but logically consistent), while the other predicts Alice's sister, violating the hard constraint $$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$. Both predictions score identically under any accuracy-based metric, yet they are qualitatively different failures.

From this observation the authors derive four desiderata for a logical compliance metric:

• (P1) Ground-truth independence: the metric must not require reference labels, since its purpose is to separate "merely wrong" from "wrong and logically inconsistent" predictions.
• (P2) Hard/soft rule distinction: strict constraints (e.g., one cannot be both sibling and spouse) must be reported as logical failures, whereas statistical regularities (e.g., co-parenthood implies marriage) may legitimately be violated.
• (P3) Dataset-aware comparability: soft-rule compliance should be assessed relative to how often the rule is already contradicted in the observed data.
• (P4) Model- and data-agnostic computability: the metric must be computable automatically for arbitrary datasets, models, and rule sets.

Formal definition

An evaluation setting is a triple $\mathcal{E} = (D, \widehat{D}, R)$ of observed facts, predictions, and rules, all expressed over a shared relational vocabulary. Rules are evaluable formulas $\psi_r \leftarrow \phi_r$ with head and body conditions over atoms, Boolean connectives, comparisons, aggregates, and user-defined decidable predicates; positive Horn rules are the tractable special case. A key assumption is body evaluability: every grounding's body must be decidable in finite time over finite fact sets.

For each rule $r$, the dataset contradiction rate is

$d_r = \frac{|V_r^D|}{|E_r^D|}$

the fraction of body-satisfying groundings in $D$ whose head is refuted. Rules with $d_r = 0$ are classified as hard by default (with an override for user-declared hard rules); all others are soft. Predictions are then evaluated one at a time by augmenting the dataset, $D_A = D \cup \{A\}$, so contradictions can be attributed to individual predictions. The prediction contradiction rate $R^2$0 aggregates the fraction of evaluable, prediction-participating groundings whose head is refuted.

The rule-level scores are:

• Hard rules: $R^2$1, a violation count with optimum $R^2$2.
• Soft rules: $R^2$3, a ratio where values below $R^2$4 indicate the model contradicts the rule less often than the dataset itself, $R^2$5 indicates dataset-equivalent behavior, and above $R^2$6 indicates worse-than-dataset compliance.

Global aggregates combine hard scores via user-specified importance weights $R^2$7 and soft scores weighted by each rule's share of evaluable groundings. Refutation is mechanism-dependent: closed-world settings refute by absence, while open-world settings require explicit negative facts, incompatibility constraints, or auxiliary rules supplied by domain experts.

Computation is implemented as relational queries over a database storing $R^2$8 and $R^2$9 (one table per predicate, with tags distinguishing observations from predictions). For positive Horn rules, SQL generation is fully automatic—body atoms compile to joins, shared variables to equality constraints, and head confirmation/refutation to lookups or incompatibility checks. The authors verify that RVS satisfies all four desiderata by construction.

Experimental evaluation

The evaluation addresses two research questions: RQ1, whether models with similar predictive accuracy exhibit different logical compliance, and RQ2, whether $$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$0 diagnoses ill-posed rules. Three benchmarks are used: the Family knowledge graph (3,007 entities, 37 rules), FB15k-237 (14,541 entities, 509 NeuralLP rules), and DV3F, a French real-estate relational database (392,809 entities, 4.8M tuples) with price-proximity rules for regression. Six models spanning rule-based (AnyBURL), embedding (CompGCN), neuro-symbolic (UniKER, ExpressGNN), GNN (GraphSAGE), and LLM-enhanced (Rel-LLM) families are evaluated.

Family. The $$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$1 diagnostic immediately flags the deliberately imperfect rule $$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$2 with $$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$3, confirming its role as a model-independent rule auditor. The central result concerns CompGCN versus UniKER: CompGCN achieves higher ranking performance (MRR 0.923 vs. 0.867) yet produces 425 hard-rule violations against UniKER's 121—a model with stronger predictive performance is substantially less logically compliant, a difference invisible to MRR and Hits@$$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$4. Rule-level analysis further shows the violation profiles are qualitatively different: CompGCN produces 1,160 violations on one brotherhood rule (versus UniKER's 29) but only 6 on an aunt rule (versus 30), indicating non-uniform compliance across constraints.

FB15k-237. The $$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$5 distribution is far wider than on Family, with many rules at $$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$6—contradicted in every evaluable grounding—showing that many rules for this benchmark are weak statistical regularities rather than constraints. The key comparison is AnyBURL versus CompGCN, whose MRR scores are nearly indistinguishable (0.327 vs. 0.335), yet whose hard-rule scores differ by more than an order of magnitude (1.44 vs. 49.8). ExpressGNN achieves the best of both: highest MRR (0.490), zero hard-rule violations, and $$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$7, meaning its predictions contradict soft rules less often than the dataset itself. This is plausibly attributable to its probabilistic-logic training, though the paper does not establish causality.

DV3F. The $$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$8 values vary sharply across property types (House 0.019, Apartment 0.203, Industrial 0.141, Outbuilding 0.257), showing that an identical rule structure has different empirical validity per category and guiding rule refinement. The model comparison again inverts the accuracy ordering: GraphSAGE achieves better regression performance ($$\bot \leftarrow \text{sister}(x,y) \land \text{wife}(x,y)$$9 0.931 vs. 0.915, MAE 31.9 vs. 40.5), but Rel-LLM achieves better aggregate logical compliance (RVS 0.91 vs. 1.17). Rule-level scores show the trade-off is not uniform—GraphSAGE is more compliant on houses, Rel-LLM on outbuildings—so the LLM-enhanced model preserves different regularities rather than uniformly better or worse ones.

Across all three benchmarks, the answer to RQ1 is affirmative: predictive accuracy and logical compliance are empirically distinct dimensions, and RVS supplies the missing axis. The answer to RQ2 is likewise affirmative: $\mathcal{E} = (D, \widehat{D}, R)$0 identifies overly broad, weakly supported, or inconsistent rules before any model is evaluated.

Relation to prior work

The paper positions RVS against two lines of work. Semantic loss functions encode logical constraints into training objectives, but are training-time mechanisms rather than evaluation metrics (2606.20208). The Sem@$\mathcal{E} = (D, \widehat{D}, R)$1 metric checks domain/range validity of link-prediction candidates, but is restricted to knowledge graphs and type constraints; the authors show Sem@$\mathcal{E} = (D, \widehat{D}, R)$2 is a strict special case of RVS under two domain/range rules. RVS generalizes along three axes: rule expressiveness, task (link prediction, classification, regression), and data model (relational databases beyond knowledge graphs).

Limitations and open questions

Several limitations are conceded or implicit. The body-evaluability assumption restricts RVS to finite settings with decidable rule bodies; rules outside this class are out of scope. Automatic SQL generation covers only positive Horn rules—more general rules require manually written templates, which limits scalability of the approach to arbitrary rule sets. The hard/soft partition is determined by observed $\mathcal{E} = (D, \widehat{D}, R)$3 values by default, so a genuinely hard constraint that happens to be violated in noisy data would be misclassified as soft unless manually overridden; conversely, the incompatibility constraints used for refutation on FB15k-237 rest on the assumption that never-co-occurring relations are incompatible, an approximation that could yield spurious refutations under open-world semantics. The aggregation weights $\mathcal{E} = (D, \widehat{D}, R)$4 are user-defined and set uniformly to 1 in the experiments, leaving the sensitivity of global scores to weighting choices unexamined. Finally, the paper evaluates RVS only as a diagnostic; whether RVS can serve as a training signal—jointly optimizing accuracy and compliance while respecting the hard/soft distinction—remains an open question the authors explicitly defer.

Conclusion

The paper contributes a formally grounded, automatically computable metric for logical compliance that is complementary to predictive accuracy. Its main empirical finding is concrete and reproducible across three heterogeneous benchmarks: models with near-identical MRR, Hits@$\mathcal{E} = (D, \widehat{D}, R)$5, or $\mathcal{E} = (D, \widehat{D}, R)$6 can differ by an order of magnitude in rule violations, and in some cases the more accurate model is the less compliant one. Beyond model comparison, the dataset contradiction rate $\mathcal{E} = (D, \widehat{D}, R)$7 functions as a model-independent auditor of both datasets and rule sets. RVS is a useful addition to the evaluation toolkit for constraint-sensitive applications, with its principal open problem being its use as an optimization target rather than a post-hoc diagnostic.