Counterfactual Trust Score Overview

• Counterfactual Trust Score is a family of metrics that quantify trust by measuring the minimal counterfactual perturbations (e.g., minimal Hamming distance) needed to flip a model’s decision.
• The framework also includes credibility profiles from constrained risk–fit optimization and gradient-based methods that assess parameter-space shifts and robustness under various scenarios.
• These diverse measures support interventions in model evaluation, recourse generation, and human-centered explanation by highlighting trade-offs in stability, uncertainty, and fairness.

Counterfactual Trust Score denotes a family of trust, credibility, and satisfaction measures that use counterfactual reasoning to assess either a model decision, a recourse candidate, or an explanation. In one formulation, trust is the normalized minimal distance from an instance $x$ to the nearest $x'$ that flips a binary classifier’s label; in other formulations it is a credibility profile derived from a constrained risk–fit optimization, a gradient-based ratio over alternative labels, a rule-level stability statistic for counterfactual recourse, a human rating that the suggested changes would lead to the desired outcome, or a batch-level aggregate over causal shifts, uncertainty, fairness violations, and temporal instability (Bertossi, 2021, Chamon et al., 2020, Prabhushankar et al., 2024, Abeyrathna et al., 17 Jun 2026, Domnich et al., 7 Apr 2025, Mathew et al., 27 Aug 2025).

1. Minimal-distance counterfactual trust for classifier decisions

A direct formalization defines trust for a single binary classification decision by the distance to the nearest label-flipping counterfactual. Let $F=\{F_1,\dots,F_n\}$ be the feature set, $x=\langle x_1,\dots,x_n\rangle$ an instance with $x_i\in Dom(F_i)$, and $C$ a binary classifier with output $y=C(x)$. With Hamming distance

$dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$

the counterfactual distance is

$$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$

If no $x'$ flips the label then one may set $x'$0. Two normalizations are given: $x'$1 Under $x'$2, a one-feature flip gives $x'$3, while changing all $x'$4 features gives $x'$5 (Bertossi, 2021).

The associated counterfactual semantics is a minimal-perturbation problem: $x'$6 The same exposition allows domain constraints and weighted perturbations through

$x'$7

This makes the score sensitive to feasibility constraints or asymmetric feature costs. The procedural algorithm enumerates Hamming layers by subset size $x'$8, searching for the first counterfactual assignment that changes the prediction; its worst-case search cost is stated as $x'$9, with pruning by early stopping when a small-$F=\{F_1,\dots,F_n\}$0 flip is found.

The declarative realization uses answer-set programming. Feature domains and the original entity are encoded as EDB facts; a classifier is encoded as rules or an external predicate; counterfactuals are generated level by level by changing exactly one feature at a time; and the first round $F=\{F_1,\dots,F_n\}$1 for which the label flips is extracted by a weak constraint of the form :~ flipped(K). [K@1]. The minimal $F=\{F_1,\dots,F_n\}$2 is then taken as $F=\{F_1,\dots,F_n\}$3, and $F=\{F_1,\dots,F_n\}$4. In the worked decision-tree example with $F=\{F_1,\dots,F_n\}$5, $F=\{F_1,\dots,F_n\}$6, and $F=\{F_1,\dots,F_n\}$7, changing only Humidity to high yields $F=\{F_1,\dots,F_n\}$8, so $F=\{F_1,\dots,F_n\}$9 and $x=\langle x_1,\dots,x_n\rangle$0. The interpretation given there is “relatively low trust,” because only a one-feature tweak is needed to reverse the decision.

2. Credibility profiles from counterfactual constrained learning

A broader counterfactual notion of trust appears as prediction credibility in a risk–fit trade-off. Let $x=\langle x_1,\dots,x_n\rangle$1 be a pre-trained differentiable classifier, $x=\langle x_1,\dots,x_n\rangle$2 a test input, $x=\langle x_1,\dots,x_n\rangle$3 the class set, and $x=\langle x_1,\dots,x_n\rangle$4 a nonnegative loss for class $x=\langle x_1,\dots,x_n\rangle$5. The fit of $x=\langle x_1,\dots,x_n\rangle$6 to class $x=\langle x_1,\dots,x_n\rangle$7 is defined as $x=\langle x_1,\dots,x_n\rangle$8, while perturbation risk is measured by

$x=\langle x_1,\dots,x_n\rangle$9

For a candidate credence profile $x_i\in Dom(F_i)$0, the minimum risk needed to attain that profile is

$x_i\in Dom(F_i)$1

The credibility profile $x_i\in Dom(F_i)$2 is characterized as a saddle-point solution of $x_i\in Dom(F_i)$3 (Chamon et al., 2020).

The Lagrangian formulation introduces multipliers $x_i\in Dom(F_i)$4: $x_i\in Dom(F_i)$5 The KKT system contains stationarity,

$x_i\in Dom(F_i)$6

primal feasibility, and complementary slackness

$x_i\in Dom(F_i)$7

The counterfactual fixed-point condition chooses credences proportional to the duals: $x_i\in Dom(F_i)$8 This is presented as the Counterfactual Trust Score or credibility profile: once the KKT multipliers are known, the full $x_i\in Dom(F_i)$9-dimensional trust profile is obtained “in one shot,” without solving the constrained problem separately for many credence vectors.

A modified Arrow–Hurwicz saddle-point algorithm jointly updates $C$0 and $C$1, and on convergence returns $C$2. The reported guarantees include local existence under differentiability and local regularity assumptions, local optimality of limit points of the algorithm, and global convergence in the convex case when each $C$3 is convex and step sizes are sufficiently small. The same $C$4 is also stated to arise as a MAP estimate of a generative model. Experimentally, credibility-based filtering on CIFAR-10 yields a smoother trade-off between coverage and accuracy than naive softmax-gap filtering, and credibility thresholding under PGD attacks reduces the accuracy drop to approximately $C$5, whereas the vanilla softmax classifier on ResNet loses up to $C$6 accuracy.

3. Inference-time and recourse-oriented variants

For large neural networks, counterfactual trust can be defined through gradients with respect to alternative labels. Let $C$7 be the pre-softmax logits, $C$8, and $C$9 a differentiable loss. For each counterfactual label $y=C(x)$0, the counterfactual gradient is

$y=C(x)$1

Over a counterfactual candidate set $y=C(x)$2, one computes

$y=C(x)$3

with variance taken over gradient coordinates or a partition such as layers. Letting $y=C(x)$4 be the largest such variance and $y=C(x)$5 the mean of the rest, GradTrust is

$y=C(x)$6

Large values indicate that one counterfactual class requires a much larger parameter shift than the others; small values signal ambiguity or low trust. The method uses one forward pass, top-$y=C(x)$7 counterfactual label selection, and $y=C(x)$8 backward passes, with the recommendation $y=C(x)$9. The stated overhead is at most one additional backward pass through the final layer per sample when implemented efficiently, and the score is explicitly unbounded, so it is to be interpreted in a within-model percentile-ranking setting (Prabhushankar et al., 2024).

The empirical evaluation covers 50,000 ImageNet validation images and 19,000 Kinetics-400 validation videos. On ImageNet, GradTrust ranks in the top-2 methods on 13 of 14 architectures for AUAC and AUFC; on Kinetics-400 it is best on 3 of 5 models and close second on the remainder. The reported failure mode is severe undertraining or severe corruption, where gradients lose discriminatory power and $dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$0 becomes noisy.

A distinct recourse-oriented variant appears in target-confidence recourse using Probabilistic Tsetlin Machines. Here the objective is not merely to cross a decision boundary but to satisfy a user-specified target confidence $dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$1: $dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$2 or equivalently $dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$3. The PTM estimates $dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$4 by Monte Carlo averaging over sampled clause activations. Each clause $dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$5 has an empirical activation probability

$dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$6

and clause stability is defined as

$dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$7

The rule-level Counterfactual Trust Score is then

$dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$8

where $dist(x,x') = |\{\, i \mid x_i \neq x'_i \,\}|,$9 is the set of clauses that contributed to the positive vote. A high $$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$0 close to $$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$1 means that all supporting rules are robustly active; a low score means that at least one clause is barely satisfied and may flip under small perturbations. The constrained search is handled by Bayesian optimization using expected improvement times feasibility probability. Across Synthetic 2D, Synthetic 5D, Iris, and Haberman, the $$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$2 setting attains perfect robustness with modest cost, including an $$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$3 distance of $$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$4 on Haberman at $$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$5 confidence, and average $$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$6 (Abeyrathna et al., 17 Jun 2026).

4. Trust in counterfactual explanations and human evaluation

In human-centered evaluation, trust refers to belief in the efficacy of the proposed counterfactual change rather than geometric robustness of a model decision. One study defines Trust as “the belief that the suggested changes would lead to the desired outcome,” measured on a 1–6 Likert-type scale, with 1 meaning “strongly disagree” and 6 meaning “strongly agree.” In an ordinary least squares model of overall Satisfaction,

$$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$7

the Trust coefficient is $$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$8 with standard error $$\delta_C(x) := \min\{ dist(x,x') \mid C(x') \neq C(x) \}.$$9, $x'$0-value $x'$1, $x'$2, and $x'$3 confidence interval $x'$4. Holding all other metrics constant, a one-point increase in Trust predicts a $x'$5 increase in the 1–6 Satisfaction rating. The overall model fit is $x'$6 on 5-fold CV, and omitting Trust together with Feasibility reduces $x'$7 from approximately $x'$8 to approximately $x'$9. Factor analysis assigns Trust a loading of $x'$00 on Factor 1, which explains $x'$01 of variance, and SHAP analysis ranks Trust as having the largest mean absolute impact on predicted Satisfaction. The paper does not propose a new composite trust score beyond the raw 1–6 rating, though it notes that a straightforward normalization is $x'$02 (Domnich et al., 7 Apr 2025).

A separate line of work studies trust gains from counterfactual explanations in image recognition through CX-ToM. In that framework, a counterfactual “fault-line” identifies the minimal semantic-level features, or explainable concepts, that need to be added to or deleted from an image in order to alter the classification category to another specified class. The optimization combines a hinge-style decision loss with $x'$03 penalties on removed and added concepts, under binary masks $x'$04 and $x'$05, and is solved by projected FISTA. Explanation is embedded in a Theory-of-Mind dialog with parse-graphs for the machine’s true mind, the human’s inference of the machine, and the machine’s inference of the human (Akula et al., 2021).

The trust-scoring metrics in CX-ToM are Justified Positive Trust (JPT), Justified Negative Trust (JNT), and Reliance. For image $x'$06 and inference process $x'$07,

$x'$08

with dataset averages over $x'$09 images and $x'$10. In the ToM game, X-ToM yields JPT of approximately $x'$11, versus $x'$12 for QA and $x'$13 for saliency, with JNT and Reliance showing the same large-effect advantage. In the VGG-16 fault-line user study, non-expert results include fault-lines without ToM at $x'$14 and CX-ToM at $x'$15; expert results include $x'$16 and $x'$17, respectively. Likert ratings are also highest for CX-ToM in Confidence, Usefulness, Detail, Understandability, and Sufficiency, and reported significance tests give $x'$18 or $x'$19 with large effect sizes $x'$20.

5. Batch-level Counterfactual Trust Score in multimodal RLHF

In multimodal reinforcement learning with human feedback, Counterfactual Trust Score is defined as a batch-level aggregate intended to mitigate bias in reward models. Let $x'$21 be the multimodal input, $x'$22 the reward-model output, $x'$23 a counterfactual, $x'$24 the topical embedding, $x'$25 the framing embedding, and $x'$26 the protected attributes. The score has four components (Mathew et al., 27 Aug 2025).

The first component is the counterfactual-shift score. Two specialized counterfactuals are constructed: one replacing only the topic embedding, and one replacing only the framing embedding. The shifts are

$x'$27

combined as

$x'$28

The second component is reconstruction uncertainty under counterfactual perturbations. A Transformer-Autoencoder is trained with

$x'$29

and evaluation uses noisy and dropout-perturbed reconstructions to compute

$x'$30

which is then normalized to $x'$31. The third component measures fairness-rule violations through Statistical Parity Difference and Disparate Impact: $x'$32

$x'$33

aggregated as

$x'$34

The fourth component measures temporal reward shifts. With $x'$35 and $x'$36,

$x'$37

The overall batch trust score is

$x'$38

with higher values meaning more trustworthy: low bias, low uncertainty, fair, and stable. The algorithmic pipeline pretrains the TAE on $x'$39, trains a CatBoost reward model on $x'$40, and for each batch $x'$41 computes feature embeddings, counterfactual shifts, uncertainty, fairness violations, temporal shift, and then $x'$42. If $x'$43, the system intervenes; $x'$44, and $x'$45.

The reported experiment uses 12,000 multimodal news items with synthetic bias injected across batches 6–10 through topical skew, framing swaps, and label drift. Relative to a CatBoost reward model without CTS, the full CTS system achieves $x'$46 accuracy versus $x'$47, AUC-ROC $x'$48 versus $x'$49, average SPD reduced from $x'$50 to $x'$51, and Disparate Impact moved from $x'$52 to $x'$53. On batches 8–10, the ablations report: counterfactual shifts only, $x'$54 and $x'$55; uncertainty only, $x'$56 and drift-warning recall $x'$57; fairness only, $x'$58 and $x'$59; temporal only, $x'$60 and $x'$61; and full CTS with equal weights, $x'$62, $x'$63, and drift detection $x'$64. The trust-versus-time curve reportedly drops $x'$65 from approximately $x'$66 to $x'$67 when bias is injected and recovers after retraining, while SHAP identifies $x'$68 and $x'$69 as the two largest contributors.

6. Comparative interpretation and scope

Across these works, the phrase denotes several non-equivalent constructions. Some are instance-level and input-space based, such as the minimal Hamming-distance score $x'$70 and its normalizations. Some are class-profile or parameter-space objects, such as the credibility vector $x'$71 or the gradient ratio $x'$72. Some are recourse-oriented and rule-based, such as $x'$73. Others are explicitly human-centered, where trust is a survey rating or a justified-alignment metric between the human’s model of the system and the system’s actual behavior (Bertossi, 2021, Chamon et al., 2020, Prabhushankar et al., 2024, Abeyrathna et al., 17 Jun 2026, Domnich et al., 7 Apr 2025, Akula et al., 2021).

This suggests that “trust” is being operationalized along at least four axes: distance to a counterfactual boundary, cost–risk compromise, stability of alternative explanations or rules, and human belief that a counterfactual recommendation will work. The range and interpretation therefore differ materially by formulation. $x'$74 and the RLHF $x'$75 are normalized to $x'$76; $x'$77 lies in $x'$78; the human Trust criterion is a raw 1–6 rating; GradTrust is unbounded and must be interpreted by within-model percentile; and the credibility profile $x'$79 is an $x'$80-dimensional vector rather than a scalar. A common misconception is to treat these quantities as interchangeable. The definitions show otherwise: larger trust means larger minimal flip distance in the Hamming-based score, larger parameter-space separation in GradTrust, smaller aggregated penalties in the RLHF score, larger clause margins in PTM-based recourse, or stronger human belief in the success of a suggested intervention.

A plausible implication is that the choice of Counterfactual Trust Score should be driven by the intervention one intends to support. Filtering and adversarial defense align with credibility profiles; low-trust percentile triage aligns with gradient-based trust; recourse under explicit confidence constraints aligns with rule-level stability; explanation evaluation aligns with human Trust, JPT, JNT, and Reliance; and fairness-aware online policy updates align with the four-component RLHF aggregator. The literature therefore supports using “Counterfactual Trust Score” as a family resemblance term rather than as a single canonical metric.