Minimum Specification Perturbation: Robustness as Distance-to-Falsification in Causal Inference

Abstract: Empirical causal claims depend on many analyst decisions, from selecting covariates to choosing estimators. Existing robustness tools summarize how results vary across these choices, but, to the best of our knowledge, do not answer: \textbf{How many analyst decisions must change to reach a specification, which is a set of choices, whose confidence interval (CI) contains zero?} We introduce \emph{Minimum Specification Perturbation (MSP)}, the smallest number of changes. MSP is small under the null, grows with effect strength and captures distance-to-falsification information that dispersion-based summaries cannot report; when making decisions under weak effects, an MSP-based rule yields lower false-positive rates than dispersion-based rules. We show that Fragility Index and MSP measure orthogonal vulnerabilities: fragility to influential observations need not imply fragility to specification choices. On the LaLonde benchmark, MSP = 1 implies that one decision change makes the CI contain zero. We further provide exact permutation calibration under randomization and characterize computation, showing tractable cases under additive structure and NP-hardness in general.

• The paper introduces MSP as a measure of falsification complexity, quantifying the minimum analyst decision flips needed to render a causal effect null-compatible.
• It details the computational approach for efficient MSP estimation under additive structure and examines its statistical calibration via permutation tests.
• Empirical illustrations on the LaLonde benchmark reveal MSP’s ability to detect specification fragility distinct from data fragility metrics.

Minimum Specification Perturbation: Robustness as Distance-to-Falsification in Causal Inference

Motivation and Conceptual Framework

Empirical causal inference routinely depends on a sequence of defensible analyst decisions, including estimator selection, covariate adjustment, functional form specification, treatment of outliers, and sample restrictions. Conventional robustness checks typically summarize the variation of causal estimates across this multiverse of specifications, emphasizing dispersion and instability. However, such summaries do not quantify how many coordinated changes to analyst choices are required to overturn a declared causal finding.

This paper introduces the Minimum Specification Perturbation (MSP) as a metric for falsification complexity within a declared specification space. MSP is defined as the smallest number of binary decision-axis flips (Hamming distance) required to reach a specification whose effect confidence interval (CI) contains zero. Small MSP values indicate fragility: the causal conclusion can be overturned by a few changes. Large MSP values imply robustness: multiple coordinated perturbations are necessary for falsification. MSP thus directly captures distance-to-falsification information.

Relation to Existing Robustness and Fragility Tools

MSP is orthogonal to, and not directly comparable with, three canonical families of robustness tools:

• Specification curve and multiverse analyses: These enumerate estimates across defensible specifications, tracking estimate dispersion but not the proximity to falsification.
• Sensitivity analysis for unmeasured confounding: Approaches such as the E-value and OVB methods evaluate how strong hidden bias must be to nullify a result, focusing on model-based perturbations rather than auditable analyst choices.
• Data-fragility measures: Metrics like the Fragility Index quantify the minimal data perturbation (e.g., observation removal or outcome modification) required to change a claim, but do not describe fragility vis-à-vis specification space.

Strong empirical divergence of MSP and data-fragility metrics is established: conclusions can be data-fragile but specification-robust and vice versa.

Formal Definition and Structural Properties

Given a specification space $S \subseteq \{0,1\}^K$ (where each axis encodes a binary analyst decision), MSP is formally:

$$\mathrm{MSP}(S) = \min\{\|s\|_0 : s \in \mathcal{N}(S)\},$$

where $\mathcal{N}(S)$ denotes configurations whose CI contains zero.

Key structural results include:

• Refinement monotonicity: Feasibility-preserving refinement of specification spaces weakly decreases MSP, making coarse grids conservative upper bounds.
• Permutation calibration: Under complete randomization and sharp-null, the permutation-based p-value for MSP is valid, establishing statistical calibration of the metric.
• Weighted MSP sandwich: Weighted axis perturbations yield bounds $$w_{\min}\,\mathrm{MSP}(S) \leq \mathrm{wMSP}(S, F, w) \leq w_{\max}\,\mathrm{MSP}(S)$$, quantifying discrepancies between weighted and unweighted MSP.

Computational Characterization

Additive structure in the specification space permits efficient computation:

• For additive point estimates and constant CI width, MSP is computable via a greedy $O(K\log K)$ algorithm.
• Variable-width CIs admit tractable extensions given structural sufficient conditions; otherwise, branch-and-bound search recovers exact MSP.
• Without bounded-step conditions or additive structure, MSP computation is NP-hard (via reduction from SUBSET SUM), but applicability to auditable specification spaces (moderate $K$) remains practical.

Empirical Illustration: LaLonde Benchmark

MSP is empirically applied to the LaLonde NSW–CPS benchmark with $K=4$ axes (estimator, covariate choice, functional form, trimming). The baseline configuration (OLS, basic covariates, linear outcome, no trimming) yields a significantly negative effect; however, MSP = 1—a single analyst decision (switching covariates to full) suffices to render the CI null-compatible (contains zero), indicating extreme structural fragility.

Figure 1: LaLonde illustration of all $2^4$ binary configurations, with MSP = 1. The baseline specification is one axis-flip away from a configuration whose CI contains zero (red intervals).

The fragility revealed by MSP is not made explicit by dispersion-based summaries. Reporting MSP with a transparent audit trail for axes is recommended to prevent selective post hoc specification space inflation.

Simulation and Decision-Rule Analysis

Simulation results elucidate that:

• Under the null, MSP is concentrated at zero (high proportion of specifications are null-compatible).
• MSP increases with effect strength, but exhibits high discriminatory power as a classifier between weak and strong effect regimes.
• In overshoot regimes (weak effects with significant confounding-induced bias), all configurations may be significant in some direction, yet MSP is infinite—no admissible perturbation yields a null-compatible CI. Conventional dispersion-based thresholds are anti-informative here.

Figure 2: ROC curves for MSP and summary-based classifiers. MSP achieves AUC = 0.941; share-significant-any is anti-informative in overshoot regimes.

Threshold rules based on dispersion-based summaries attain false-positive rates above 0.86, whereas MSP-based rules ($\mathrm{MSP} \ge 2$) achieve FPR = 0.010. MSP thus supplies actionable decision-theoretic advantages over standard specification-curve diagnostics.

Comparison with Fragility Index and Data Perturbation

A systematic comparison demonstrates that MSP and Fragility Index probe orthogonal robustness dimensions. On the NSW randomized benchmark, $\widetilde{\mathrm{FI}}_{\mathrm{adv}} = 3$ (three treated outcomes set to zero achieve nonsignificance), while MSP is infinite ($$\mathrm{MSP}(S) = \min\{\|s\|_0 : s \in \mathcal{N}(S)\},$$0): the result is data-fragile but specification-robust. The empirical and synthetic reversals highlight the necessity of reporting both metrics for comprehensive robustness auditing.

Practical Implications and Future Directions

MSP delivers a transparent, auditable summary of robustness conditional on declared analyst choice axes and inference rules. Applications include empirical economics, medicine, and public policy, where specification fragility—distinct from data fragility and model-based sensitivity—has direct implications for policy recommendations.

Key practical guidance is to pre-specify axes defensibly, maintain fixed estimand scales, and report both unweighted and weighted MSP versions. In observational settings, MSP should not be over-interpreted as causal certification absent valid permutation calibration.

MSP exposes avenues for future theoretical development:

• Standardization of axis-weight protocols for weighted MSP.
• Algorithmic recourse analysis leveraging MSP in specification-space navigation.
• Extensions to specification spaces with mixed scales and nonbinary axis encoding.
• Integration of MSP with joint permutation-based specification curve tests.

Conclusion

Minimum Specification Perturbation quantifies falsification complexity as the distance in a declared specification space to a null-compatible configuration. It reveals structural fragility or robustness missed by dispersion-based summaries, delivers strong empirical discriminatory power, and exposes the orthogonality of data and specification fragility. Computational tractability is ensured under additive structure, but general cases are NP-hard.

MSP's auditability and distance-to-falsification interpretation make it a substantive tool for robust causal inference and specification-space transparency. Its integration with existing robustness metrics and decision rules is essential for comprehensive empirical analysis.