Defensibility Index (DI): Metrics & Analysis

• Defensibility Index (DI) is a metric that measures the defendability of AI decisions by assessing their logical derivability from explicit policy rules.
• It utilizes a three-level framework to differentiate robust, plausible, and indefensible outcomes, incorporating complementary measures like Ambiguity Index and Probabilistic Defensibility Signal.
• The DI is applied in contexts such as rule-governed AI, risk analysis, and cybersecurity, and is distinct from other unrelated DI metrics in bibliometrics and software quality.

to=arxiv_search 天天中彩票篮球json {"query":"\"Defensibility Index\" OR \"defensibility\" DI arXiv", "max_results": 10, "sort_by": "relevance"}Japgollyassistant to=arxiv_search commentary ส่งเงินบาทไทย 全民彩票天天送json {"query":"\"Defensibility Index\" arXiv", "max_results": 10, "sort_by": "relevance"} Defensibility Index (DI) is not a universally standardized term across the arXiv literature. In its most explicit recent usage, it denotes a policy-grounded metric for rule-governed AI: the fraction of decisions that are defensible under an explicit rule hierarchy rather than merely aligned with historical human labels (O'Herlihy et al., 22 Apr 2026). In adjacent security and risk-analysis work, closely related formulations of defensibility quantify how much a defensive intervention improves post-attack outcomes, emphasizing marginal defensive leverage rather than baseline vulnerability alone (Bier et al., 2019). Across arXiv more broadly, however, the abbreviation “DI” also denotes unrelated constructs such as discrimination information, disruptive index, and depth of inspection, making domain-specific disambiguation essential.

1. Terminological scope

Across the cited literature, “DI” is a polysemous abbreviation rather than a single established technical object. The most direct use of the full term “Defensibility Index” appears in rule-governed AI evaluation, while several neighboring literatures either use “defensibility” without that exact label or reserve “DI” for entirely different metrics.

Domain	Meaning of “DI”	Relation to “Defensibility Index”
Rule-governed AI	Defensibility Index	Explicit named usage
Critical-system risk analysis	Defensibility $D(a,b)$	Closely related, but not labeled “DI”
Statistical inference	Discrimination information	Unrelated usage
Bibliometrics	Disruptive Index	Unrelated usage
Software quality	Depth of Inspection	Unrelated usage
Quantum cryptography	Device-independent	Abbreviation only, not an index

This distribution of meanings appears directly in the cited corpus and underlies a recurring methodological issue: cross-domain transfer of the label “DI” is unsafe unless the underlying object, normalization, and decision context are specified (O'Herlihy et al., 22 Apr 2026, Bier et al., 2019, Bickel, 2010, Liang et al., 2020, Nair et al., 2012, Tan et al., 2019).

2. Policy-grounded DI in rule-governed AI

In "Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI" (O'Herlihy et al., 22 Apr 2026), DI is defined for environments such as content moderation in which multiple outcomes may be logically consistent with the governing rules. The paper’s motivating claim is that agreement-based evaluation conflates three distinct phenomena—true model error, moderator divergence from written policy, and policy ambiguity—and therefore can penalize valid decisions. Its alternative is policy-grounded correctness.

The formal object is a moderation decision $y \in \{remove, approve\}$ evaluated against content $C$, rule hierarchy $R = (R_G, R_C)$, and precedent corpus $P$. A decision is defensible if a valid logical derivation exists from $(C,R,P)$ to $y$. The framework distinguishes three levels: L1 — Robustly Defensible, where an explicit rule directly and unambiguously authorizes $y$; L2 — Plausibly Defensible, where the rules are genuinely ambiguous but could reasonably support $y$; and L3 — Indefensible, where no explicit rule authorizes $y$, the reasoning imports concepts absent from the rule set, or the content actually complies with the cited rule. The Defensibility Index is then

$y \in \{remove, approve\}$0

A companion metric, the Ambiguity Index (AI), measures whether the opposite decision is also defensible:

$y \in \{remove, approve\}$1

Within this formulation, DI does not measure agreement with labels and does not measure ambiguity directly. It measures whether the chosen decision is defensible at all. AI measures whether the rule set is underspecified enough that the inverse outcome is also defensible. The distinction is central: high DI with low AI corresponds to clear policy-grounded automation; high DI with high AI corresponds to defensible but ambiguous decisions; low DI indicates indefensible reasoning regardless of ambiguity. The paper further decomposes ambiguity into $y \in \{remove, approve\}$2 for platform-wide underspecification in $y \in \{remove, approve\}$3 and $y \in \{remove, approve\}$4 for community-specific underspecification in $y \in \{remove, approve\}$5.

3. Audit architecture, probabilistic signals, and empirical behavior

The same paper operationalizes DI through a two-model architecture. The moderation model $y \in \{remove, approve\}$6 produces decision $y \in \{remove, approve\}$7, while an audit model $y \in \{remove, approve\}$8 evaluates whether $y \in \{remove, approve\}$9 is derivable from $C$0. The audit model does not decide whether content violates policy; it emits a structured JSON reasoning trace in the order

$C$1

The ordering is intended to force commitment to explicit policy citation before the defensibility label is assigned. To estimate reasoning stability without additional audit passes, the paper introduces the Probabilistic Defensibility Signal (PDS), derived from audit-model token logprobs. PDS combines three signals: label log-confidence at the defensibility-level token; citation span entropy during policy-citation generation; and inverse-check log-odds, $C$2, where $C$3 is the prefix before the inverse check. The calibrated weights learned on the balanced sample are approximately $C$4 for label confidence, $C$5 for entropy, and $C$6 for inverse-check signal, indicating that the binary defensibility target is driven primarily by final verdict confidence and inverse-check behavior rather than citation entropy.

Empirically, the framework was validated on 193,000+ Reddit moderation decisions. The reported gap between agreement-based metrics and DI is 33–46.6 percentage points. On the random sample, $C$7 while DI $C$8; on the balanced sample, $C$9 while DI $R = (R_G, R_C)$0. The paper further reports that 79.8–80.6% of the model’s false negatives were actually defensible. Across 270 communities, the “Earned Autonomy” cohort had DI $R = (R_G, R_C)$1 and AI $R = (R_G, R_C)$2, “Policy Gaps” had DI $R = (R_G, R_C)$3 and AI $R = (R_G, R_C)$4, and “Normative Complexity” had DI $R = (R_G, R_C)$5 and AI $R = (R_G, R_C)$6. In a rule-specificity experiment on 37,286 identical decisions from r/AskReddit, auditing under “Title Only,” “Sidebar (+ descriptions),” and “Wiki (+ examples/exceptions)” rules produced DI values of 97.4%, 98.0%, and 98.1%, while AI fell from 18.2% to 8.8% to 7.4%; the paper interprets the 10.8 percentage-point drop in AI as an empirical estimate of community-level underspecification, with the dominant effect being $R = (R_G, R_C)$7 conversion rather than a reduction in indefensible cases. Repeated-sampling analysis attributes PDS variance primarily to governance ambiguity rather than decoding noise; DI is reported as temperature-invariant, and standard confidence remains high even when reasoning is unstable. A Governance Gate built on these signals achieves 78.6% automation coverage with 64.9% risk reduction. The stated caveats are substantial: audit-model dependence, the same-backbone condition (SBC), the Audit Independence Assumption (AIA), imperfect calibration, a human validation study with $R = (R_G, R_C)$8, a structural escape rate of about 29.9% for hallucinated-grounding attacks, and domain specificity to Reddit moderation (O'Herlihy et al., 22 Apr 2026).

4. Defensibility as marginal value of defense

A different but conceptually adjacent formulation appears in "Risk analysis beyond vulnerability and resilience - characterizing the defensibility of critical systems" (Bier et al., 2019). There, defensibility is a dimensionless measure of how much a modest defensive investment can improve the outcome of an attack or disruption. The paper’s core quantity is

$R = (R_G, R_C)$9

where $P$0 is residual system value after attack strength $P$1 and defense investment $P$2, $P$3 is residual value without defense, and $P$4 is initial system value. The normalization by $P$5 makes the quantity dimensionless and is explicitly intended to allow comparison across incommensurable systems such as electrical grids and water-distribution networks. In the discrete-asset formulation, the paper models survival through $P$6, asset-level residual value through $P$7, and total residual value through $P$8.

The paper distinguishes defensibility from vulnerability, robustness, resilience, security, and redundancy by focusing on the marginal benefit of defense rather than baseline exposure or recovery alone. Its results include: reflexive defense is optimal when attacker and defender rank assets the same way and the attacker is optimizing; residual value is decreasing in attack effort $P$9 and increasing in defense effort $(C,R,P)$0; defensibility is increasing in both $(C,R,P)$1 and $(C,R,P)$2 under the same conditions; and DI is more sensitive to increasing $(C,R,P)$3 than to increasing $(C,R,P)$4 iff $(C,R,P)$5. Against a uniform random attacker, reflexive defense also remains optimal, and the paper derives corresponding residual-value and defensibility expressions. Under stated conditions, defensibility against an optimal attacker can exceed defensibility against a uniform random attacker, particularly when asset values are highly skewed. The examples illustrate the dependence on value distribution and threat model: in a property-losses dataset, average defensibility over $(C,R,P)$6 is reported as more than 53% for an optimal attacker with $(C,R,P)$7; in an air-departures dataset, the corresponding figure is about 11%; and in a synthetic negative-skew dataset, average defensibility is only about 2.5% against optimal attack but about 5.9% against a random attacker.

A later network-defense paper generalizes the same design question away from a scalar index toward a verdict-and-fingerprint architecture (Hsain et al., 11 Jun 2026). "Beyond Runtime Enforcement: Shield Synthesis as Defensibility Analysis for Adversarial Networks" does not define a single scalar DI; instead it outputs a defensibility verdict—whether the initial product state $(C,R,P)$8 lies in the winning region $(C,R,P)$9—along with a winning region, a shield $y$0, and a six-axis defensibility fingerprint. The formal layer is built from a constrained two-player safety game on a network-defense topology, with defender actions $y$1, attacker actions $y$2, a defender safety specification requiring that DB and BK are never both in a bad state and that at least 3 hosts remain active, and an attacker specification of at most 2 Destroy actions per engagement. The fingerprint combines five attractor-based metrics—Attackability (ATK), Sinking Ratio (SNK), Shield Friction (FRC), Attractor Steepness (STP), and Violation Proximity (VPX)—with the MARL-based Defender Dominance Ratio (DDR), represented in danger-oriented form as ADR $y$3. The what-if analysis shows that formal defensibility and operational effectiveness are distinct: fully connecting the topology changed $y$4 only from 15.8% to 15.5% but reduced DDR from 53.9% to 22.7%, while removing a VPN bypass changed $y$5 only from 15.8% to 15.9% but raised DDR to 80.7%. This suggests that, in adversarial networks, a scalar “defensibility index” may be too coarse to capture the separation between formal safety margins and operational behavior.

5. Other formal meanings of “DI”

In statistical inference, "Statistical inference optimized with respect to the observed sample for single or multiple comparisons" (Bickel, 2010) uses DI to mean discrimination information. The measure is defined as the logarithm of a normalized maximum likelihood (NML) ratio comparing alternative and null hypotheses on the observed sample. For a family $y$6, the NML is

$y$7

and the information favoring the alternative over a point null $y$8 is

$y$9

In the weighted generalization, the paper defines

$y$0

The result is expressed in bits, and the paper emphasizes several evidential properties: asymptotic vanishing of misleading evidence, support for simple null hypotheses, minimax optimality with respect to observed-sample loss, and the absence of a prior-distribution requirement. Because ordinary NML is often undefined for common families, the paper introduces weighted likelihood and normalized maximum weighted likelihood (NMWL). This DI is therefore an information-theoretic evidence measure rather than a governance or defense metric.

In bibliometrics, "Same data may bring conflict results: a caution to use the disruptive index" (Liang et al., 2020) uses DI to denote the Disruptive Index, intended to capture whether a scientific paper displaces the prior literature it built upon or instead consolidates it. In the standard form reported there,

$y$1

where $y$2 counts later papers citing the focal paper without its references, $y$3 counts later papers citing both the focal paper and its references, and $y$4 counts later papers citing the references but not the focal paper. The paper’s central caution is methodological: DI is highly sensitive to citation-window choice, discipline, time period, and bibliographic database coverage. It reports that the threshold at which the median DI of Nobel papers becomes lower than benchmark papers occurs at about 10 years in Medicine, 6 years in Chemistry, and 4 years in Physics. It also warns that DI $y$5 does not necessarily indicate a breakthrough-class achievement, because the score is mechanically determined by local citation counts and reference coverage. This use of DI is entirely separate from defensibility.

6. Software quality, cybersecurity, and interpretive cautions

In software engineering, "Defect Management Using Depth of Inspection and the Inspection Performance Metric" (Nair et al., 2012) defines DI as Depth of Inspection, a process metric for the fraction of detected defects found by inspection:

$y$6

Here $y$7 is the number of defects captured by inspection and $y$8 is the total defects captured by inspection and testing. DI ranges from 0 to 1, can be measured phase-wise or at the project level, and can be predicted by a multiple linear regression model,

$y$9

using inspection time, preparation time, number of inspectors, and experience level of inspectors. The paper gives a qualitative scale from “Worse” $y$0 through “Ideal” $y$1, treats $y$2 to $y$3 as a normal inspection process, states that $y$4 onwards requires high competency, and reports that CMMI Level 4 and above organizations can achieve an average DI of about $y$5 to $y$6. This DI measures inspection effectiveness, not defensibility.

Several cybersecurity papers define adjacent but differently named measures. "Security Index from Input/Output Data: Theory and Computation" introduces a model-based security index $y$7 and a data-driven index $y$8, each quantifying the minimum number of components that must be compromised to carry out a perfectly undetectable attack involving component $y$9 (Shinohara et al., 12 Nov 2025). Smaller values indicate greater vulnerability; infeasibility yields $y$0. Under persistent excitation of order $y$1 and $y$2, the paper proves $y$3, shows exact computation is NP-hard, and provides a polynomial-time computable upper bound. A plausible implication is that this security index functions as a component-wise defensibility threshold, even though the paper does not use that label. "Threat-Informed Cyber Resilience Index: A Probabilistic Quantitative Approach to Measure Defence Effectiveness Against Cyber Attacks" introduces CRI rather than DI, but explicitly frames it as a single quantitative score of defence effectiveness against a campaign, built from attack-flow probabilities, POMDP state transitions, and normalized cumulative rewards (Alevizos et al., 2024). Finally, in quantum cryptography, "Computing secure key rates for quantum key distribution with untrusted devices" uses “DI” exclusively to mean device-independent; it defines no Defensibility Index and no scalar metric by that name (Tan et al., 2019).

Taken together, these usages establish two points. First, “Defensibility Index” in the strict sense presently refers most clearly to the policy-grounded metric for rule-governed AI in which defensibility is judged by logical derivability from explicit rules. Second, the broader defensibility family includes related constructs in risk analysis and cybersecurity that measure the marginal efficacy of defense, but the abbreviation “DI” itself remains strongly context-dependent and frequently denotes unrelated quantities.