Reject Inference: Techniques and Applications

• Reject Inference is a set of techniques addressing sample selection bias by adjusting statistical models when target labels are observed only in selected cases.
• It employs methods such as reweighting, extrapolation, and self-learning to infer outcomes for rejected cases in domains like credit risk and safety-critical applications.
• Reject Inference improves model calibration and reliability by mitigating distribution shifts, enabling more robust predictions in high-stakes environments.

Reject inference refers to a family of statistical and algorithmic techniques for addressing sample selection bias in situations where observed target labels are only available for a selected (non-random) subset of the population. Most prominently encountered in credit risk modeling, reject inference is deployed to estimate or adjust for the unknown outcomes of cases that have been systematically “rejected” in the data-collection process—such as applicants denied credit, medical patients outside inclusion criteria, or any population for which measurement is censored by deliberate selection policies. The central challenge is correction of the distribution shift induced by selection, as models trained only on accepted samples generally provide miscalibrated or biased predictions when deployed on the broader population.

1. The Selection Bias Problem and Statistical Formulation

Reject inference arises in domains where an accept/reject policy $A \in \{0,1\}$ determines which input instances $x \in \mathcal{X}$ have observed target labels $y \in \mathcal{Y}$, resulting in selection bias. In credit scoring, only applicants who pass the lender’s initial screening (accepted applicants, $A=1$) are observed with repayment outcomes $y$, while rejected applicants’ $y$ remain unobserved. The observed data distribution $p(x, y \mid A=1)$ differs from the full population $p(x, y)$, violating covariate shift and missing not at random (MNAR) assumptions. When a scoring model $f(x)$ is trained on the accepted set $\{(x_i, y_i): A(x_i) = 1\}$, its empirical risk $$R_s(f) = \mathbb{E}_{(X, Y) \mid A=1}[\ell(Y, f(X))]$$ does not estimate the target risk $R(f) = \mathbb{E}_{(X, Y)}[\ell(Y, f(X))]$, inducing systematic underestimation of risk in unobserved regions (Ribeiro et al., 14 Oct 2025, Mancisidor et al., 2019).

2. Classical and Modern Reject Inference Methodologies

Reject inference methods aim to leverage rejected cases $(x_j : A(x_j) = 0)$ to reduce model bias. The literature distinguishes several categories:

2.1. Augmentation and Re-weighting

• Upward/Downward Augmentation: Compute acceptance model $g(x) = P(A=1|x)$, reweight accepted samples as $w_i \leftarrow w_i / g(x_i)$ or $w_i \leftarrow w_i \times (1-g(x_i))$ to correct for selection probabilities.
• Soft Cut-Off & Fuzzy Augmentation: Partition data by acceptance probability or duplicate rejected cases with weighted labels to reflect estimated acceptance and default rates (Ribeiro et al., 14 Oct 2025).

2.2. Extrapolation and Self-learning

• Extrapolation (EX, BE, E-C): Assign pseudo-labels to rejected samples based on the current classifier predictions, potentially restricting to “most confident” instances. Parcelling assigns probabilistic labels to rejected samples within score bins based on an adjusted bad-rate (Ribeiro et al., 14 Oct 2025, Kozodoi et al., 2019).
• Label Propagation/Spreading: Propagate labels via similarity graphs, allowing for label diffusion from accepted to rejected cases (Ribeiro et al., 14 Oct 2025).
• Iterative Self-Learning: Alternating labeling of high-confidence rejects with model re-training, often using outlier filtering (Isolation Forest), and separating low-complexity labeling steps from strong model training (e.g., gradient boosting) (Kozodoi et al., 2019).

2.3. Model-based and Bayesian Techniques

• Deep Generative Semi-supervised Models: Model joint $p(x, y, z)$ with latent variables and fit all available (accepted and rejected) data as either labeled or unlabeled, often with class-conditional Gaussian Mixtures and variational inference. Exact enumeration over possible outcomes for rejected samples avoids ad-hoc heuristics (Mancisidor et al., 2019).
• Modern Transductive and Multi-task Approaches: Networks that jointly model acceptance and default or propagate metrics within learned embedding spaces can further reduce bias (Ribeiro et al., 14 Oct 2025).

2.4. Distributional Filtering and Confident Inlier Extrapolation

• Confident-Inlier Extrapolation (CI-EX): Only include rejected samples that are both distributionally close to the accepted cases (inliers under an outlier detector) and classified with high pseudo-label confidence at each iteration, thereby limiting extrapolation risk (Ribeiro et al., 14 Oct 2025).

3. Reject Inference in Explainable and Robust Machine Learning

Reject inference is not confined to credit scoring. In explainable AI and robust classification for safety-critical applications, the reject option is a principled output state triggered by logical or probabilistic conflict in the predicted outputs and their explanations. Models employing Explanation Dependency Decomposition (EDD) represent the joint $p(y, z|x; \theta)$ with various factorizations (e.g., $p(y|x;\theta)p(z|y,x;\theta)$ or $p(z|x;\theta)p(y|z,x;\theta)$) and, at inference, accept only those predictions for which the predicted label $y$ and attribute pattern $z$ uniquely satisfy manually specified necessary and sufficient criteria $C(y)$ (Kronenberger et al., 2020). If conflicting or ambiguous, the classifier outputs "reject," providing an explicit uncertainty-aware mechanism integral to safety and trustworthiness.

4. Mathematical and Algorithmic Frameworks

Reject inference methods instantiate a variety of learning and inferential frameworks, nearly always formalized through probabilistic modeling, sampling, or semi-supervised estimation.

4.1. Probabilistic and Variational Approaches

• Variational Rejection Sampling (VRS): Augments variational inference with a resampling step that rejects candidate latent samples from the inference network $q_\phi(z|x)$ with probability proportional to likelihood under the generative model $p_\theta(x, z)$ and a tunable threshold $T$. This creates an intermediate proposal $r_{\theta, \phi}(z|x, T)$ between $q_\phi$ and the exact posterior, tightening the evidence lower bound (ELBO) as $T$ decreases and controlling the bias-variance-computation trade-off (grover et al., 2018).

4.2. Semi-supervised Learning Objectives

• Deep Semi-supervised ELBO: Combine labeled (accepted) and unlabeled (rejected) objectives, e.g.,

$$\mathcal{L} = \sum_{(x, y) \in \text{accepted}} [-\mathcal{L}_\text{accept}(x, y)] + \sum_{x \in \text{rejected}} [-\mathcal{L}_\text{reject}(x)] + \alpha \sum_{(x, y) \in \text{accepted}} \log q(y|x)$$

with exact marginalization over label possibilities for rejects (Mancisidor et al., 2019).

4.3. Logical Verification for Reject Option

• Belnap Logic Criterion: For explainable classification, accept a prediction $\hat{y}$ only if the predicted attribute vector $\hat{z}$ uniquely matches class $C(\hat{y})$, i.e., the compatible class set $\hat{\mathcal{Y}}$ constructed from $\hat{z}$ and $C(y)$ equals $\{\hat{y}\}$. All other situations, including ambiguous, conflicting, or missing attributes, trigger a "reject" decision (Kronenberger et al., 2020).

5. Metrics and Evaluation Methodologies

Standard metrics (AUC, Brier score, F1) are insufficient to assess reject inference as they only measure performance on the accepted set. Dedicated RI metrics include:

• Kickout: Quantifies net improvement in the rejection of bad payers versus good payers after model update. For acceptance rate $\alpha$,

$\text{kickout} = \frac{K_B}{S_B} - \frac{K_G}{S_G}$

where $K_B$ and $K_G$ are the counts of bad and good loans, respectively, that are newly "kicked out" by the post–RI model (Kozodoi et al., 2019, Ribeiro et al., 14 Oct 2025).

• Area Under the Kickout (AUK): Aggregates kickout improvements over the full range of acceptance rates, providing a holistic frontier for RI performance (Ribeiro et al., 14 Oct 2025).

A key finding is that the correlation between AUC on accepted samples and actual performance on an unbiased population is near zero ($\approx 0$), while kickout correlates moderately with out-of-sample performance ($r \approx 0.41$), making it a more operational and production-aligned metric (Kozodoi et al., 2019).

6. Empirical Performance and Trade-offs

Empirically, incorporating reject inference can yield measurable—even if modest—gains in out-of-sample discrimination, calibration, and reduced risk of underestimated default. For example, semi-supervised deep generative models achieve AUC improvements ($0.636$ vs. $0.629$) and superior Gini coefficients relative to classic baselines in credit scoring, with scalability to hundreds of thousands of samples (Mancisidor et al., 2019). Shallow self-learning with calibrated filtering yields the highest R-Precision and Brier score on unbiased sets, outperforming classical and regular self-learning strategies (Kozodoi et al., 2019). CI-EX achieves a strong Pareto frontier: on LendingClub and HomeCredit, it attains near-benchmark AUC with maximal kickout/AUK values, outperforming alternatives on RI-specific measures (Ribeiro et al., 14 Oct 2025).

A characteristic trade-off is observed—methods optimizing for high AUC on the accepted set may underperform on AUK, overstating in-sample performance, while aggressive RI strategies can degrade AUC but provide more equitable and robust out-of-sample calibration.

7. Practical Guidance and Deployment Considerations

• Filtering and Confidence Control: Employ outlier detection (e.g., Isolation Forest) and confidence thresholds to minimize harmful extrapolation into unrepresented feature regions (Ribeiro et al., 14 Oct 2025, Kozodoi et al., 2019).
• Model Selection: Use evaluation measures tailored to RI (kickout, AUK), as conventional holdout AUC is not diagnostic of population-level error (Ribeiro et al., 14 Oct 2025, Kozodoi et al., 2019).
• Iterative Expansion and Stopping Criteria: Gradually add confidently inferred rejects, monitoring the AUC-AUK trade-off to optimize the Pareto frontier (Ribeiro et al., 14 Oct 2025).
• Manual Specification for Safety: In explainable models with reject options, explicitly specify class-attribute rules $C(y)$, calibrate attribute thresholds, and design for human-in-the-loop fallback on reject cases (Kronenberger et al., 2020).
• Integration: RI methods such as shallow self-learning, deep generative models, or CI-EX integrate with standard ML pipelines and do not require labeled reject data, enabling production deployment without "random-lend" experiments (Kozodoi et al., 2019, Ribeiro et al., 14 Oct 2025).

A plausible implication is that, in high-stakes or safety-critical scenarios, principled reject inference—whether via statistical, algorithmic, or logical means—enables more reliable, calibrated decisions at the cost of (controllable) prediction coverage, aligning automated inference with robust risk management and regulatory requirements.