Binary Choice Model Diagnostics

Updated 2 April 2026

Binary choice model diagnostics are tests designed to assess model specification, identification, and stability through various nonparametric and structural approaches.
They employ techniques such as local smoothing, numerical differentiation, and bootstrap simulations to evaluate rationality, tail behavior, and goodness-of-fit.
Score stability and discrimination diagnostics, including PSI and KS statistics, help detect population shifts and ensure robust classifier performance.

A diagnostic test for binary choice models is any statistical or structural procedure designed to assess, validate, or falsify the adequacy of a binary choice model’s specification, identification, empirical content, or stability. These diagnostics encompass tests for nonparametric rationalizability, distributional assumptions (including error tail behavior), score stability, goodness-of-fit, identification conditions, and confusion matrix recovery in the absence of ground-truth labels. The recent literature provides a suite of theoretically grounded and computationally implementable diagnostics tailored to various parametric, semiparametric, and nonparametric forms of binary choice models, with substantial implications for applied demand analysis, counterfactual prediction, and classifier evaluation.

1. Nonparametric Shape and Rationality Diagnostics

A foundational advance is the nonparametric shape-restriction test derived by Bhattacharya for binary choice models with general unobserved heterogeneity (Bhattacharya, 2019). Let $Q \in \{0,1\}$ be the observed outcome (e.g., purchase/no-purchase), priced at $P=p$ and income $Y=y$ . The conditional choice probability function $q(p,y) := Pr(Q=1 | P=p,Y=y)$ must satisfy two “Slutsky-like” restrictions for consistency with utility maximization:

$\frac{\partial q(p,y)}{\partial p} \leq 0$ (monotonicity in price)
$\frac{\partial q(p,y)}{\partial p} + \frac{\partial q(p,y)}{\partial y} \leq 0$ (joint monotonicity under price–income shifts keeping $y-p$ constant)

These closed-form, global constraints are necessary and sufficient for rationalizability of the observed binary demand under unrestricted heterogeneity and arbitrary income effects. The applied diagnostic workflow involves (i) nonparametric estimation of $q(p,y)$ via local smoothing or splines, (ii) numerical differentiation on a regular $(p,y)$ grid, (iii) enforcement or post-estimation projection to handle noise, and (iv) hypothesis testing using test statistics based on the supremum of violations, with bootstrap-based critical values.

This diagnostic is applicable in parametric (e.g., probit/logit), semiparametric (monotone-index), and nonparametric bases by expressing the shape constraints as linear (sign) restrictions on parameter or basis coefficients. Its empirical content enables construction of sharp theory-consistent bounds on demand and welfare at unobserved counterfactuals, leveraging only the shape and observed data (Bhattacharya, 2019).

2. Tail Diagnostics for Error Distribution Misspecification

A recent class of model-free specification tests targets the tail structure of the unobserved error distribution, exploiting observable extremes in the covariates (Ji et al., 29 Mar 2026). For the unit-threshold model $Y_i = I(X_i - \varepsilon_i \ge 0)$ , under domain-of-attraction (DoA) conditions, the tail index $P=p$ 0 of the error can be translated into a tail index $P=p$ 1 of the observed $P=p$ 2 via $P=p$ 3. The null hypothesis $P=p$ 4 encompasses all thin-tailed (Gumbel-type) errors, i.e., both probit and logit models.

Operationally, the test proceeds by:

Extracting the largest $P=p$ 5 order statistics in $P=p$ 6 among observations with $P=p$ 7.
Constructing a self-normalized, affine-invariant test statistic based on their spacings.
Calculating a likelihood ratio with reference distribution simulated under the thin-tail null.
Extending this to the left tail $P=p$ 8 and combining via Bonferroni for two-sidedness.
Using critical values tabulated via simulation.

Monte Carlo studies establish size control and power for moderate $P=p$ 9 and realistic samples; empirical analyses reveal frequent rejection of thin-tail assumptions in economic applications—implying that conventional probit/logit may understate incidence of large shocks (Ji et al., 29 Mar 2026).

3. Score Stability and Discrimination Diagnostics

Diagnostics addressing model stability and practical discrimination leverage comparison between baseline and deployment populations (Pomazanov, 7 Jul 2025). Three key indices are central:

Population Stability Index (PSI):

$Y=y$ 0

Quantifies distributional shift in score indicator between two samples; thresholds ( $Y=y$ 1) indicate instability.

Kolmogorov–Smirnov (KS) Statistic:

$Y=y$ 2

Measures maximal CDF divergence; values above $Y=y$ 3– $Y=y$ 4 flag material drift.

Stability-Corrected (“Real”) Gini:

$Y=y$ 5

Assesses effective discriminatory power after accounting for population shift (via observed KS shift $Y=y$ 6). This adjusted estimate can fall substantially below the ROC-derived Gini, revealing practical degradation in real-world discrimination (Pomazanov, 7 Jul 2025).

Combined application of PSI and KS guides model revalidation, while the real Gini provides a conservative lower bound for performance under distributional change.

4. Goodness-of-Fit Diagnostics via Probability Integral Transform

Kheifets and Velasco (Kheifets et al., 2017) propose a specification test based on non-randomized probability integral transforms (PIT) for discrete-response (including binary choice) models. For fitted parametric models (such as probit/logit), they construct a deterministic transform $Y=y$ 7 for each observation, which under the model yields a martingale-difference process in $Y=y$ 8.

The empirical process

$Y=y$ 9

with test statistics based on the Kolmogorov–Smirnov ( $q(p,y) := Pr(Q=1 | P=p,Y=y)$ 0) or Cramér–von Mises (integral quadratic) functionals over $q(p,y) := Pr(Q=1 | P=p,Y=y)$ 1 is compared to bootstrap-based critical values. The non-randomized transform offers strictly smaller variance and higher power compared to jittered PIT alternatives, especially in small samples and under local misspecification (Kheifets et al., 2017).

5. Identification Diagnostics: Sign Saturation and Fixed Effects

Zhu introduces a nonparametric diagnostic addressing identification in binary choice models with fixed effects, via the “sign saturation” condition (Zhu, 2022). The key requirement is that the distribution of conditional average treatment effects $q(p,y) := Pr(Q=1 | P=p,Y=y)$ 2 be nontrivial in sign:

$q(p,y) := Pr(Q=1 | P=p,Y=y)$ 3

This ensures that the “separating hyperplane” defined by the index $q(p,y) := Pr(Q=1 | P=p,Y=y)$ 4 is uniquely identified (up to scale) by the observed data. The diagnostic is operationalized by maximizing the sample average of $q(p,y) := Pr(Q=1 | P=p,Y=y)$ 5 over directions $q(p,y) := Pr(Q=1 | P=p,Y=y)$ 6. The test statistic is the rescaled minimum of supremum and infimum values, and bootstrap-based quantiles provide significance thresholds. Failure of sign saturation implies generic nonidentification under bounded regressors or discrete covariates (Zhu, 2022).

6. Model Assumption Diagnostics via Interquantile Range

A novel diagnostic addresses the validity of the commonly imposed Type I extreme value (EV1) distribution for “resolvable uncertainty” in random-coefficient models. The methodology tests the null that the interquantile range of the latent transfer variable, scaled appropriately, is invariant across quantile levels under the EV1 assumption (Meango, 18 Mar 2025).

Implementation proceeds by:

Estimating the quantile-treatment-response function $q(p,y) := Pr(Q=1 | P=p,Y=y)$ 7 via quantile regressions.
Computing the distribution of interquantile ranges $q(p,y) := Pr(Q=1 | P=p,Y=y)$ 8 at multiple quantiles.
Constructing a multivariate test statistic reflecting the invariance of this distribution across quantiles, with a covariance-adjusted norm.
Simulating the null distribution via bootstrap.

Empirical rejection signals that the unobserved uncertainty is not EV1, thereby questioning the appropriateness of standard mixed-logit or multinomial logit formulations (Meango, 18 Mar 2025).

7. Diagnostic Test Approach for Confusion Matrices Without Labeled Data

Evans (Evans, 2022) adapts classical diagnostic testing theory for estimating binary classifier confusion matrices and accuracy statistics using only unlabeled data. The core setting involves running two independently erring classifiers on two populations of differing prevalences, solving a moment system for the unknown sensitivities, specificities, and prevalences. Both frequentist and Bayesian inference (method of moments, MCMC) are feasible. Once operating characteristics are recovered, all standard accuracy indices (PPV, NPV, accuracy, F1) follow directly from their functional form.

The two-population, two-test approach hinges on the conditional independence of classifier errors and sufficient contrast in class prevalences. Violation of these conditions or sample size constraints can render the system weakly identified or numerically unstable (Evans, 2022).

References: