Binary Classifier P(IK): Methods & Calibration

• Binary classifier P(IK) is a method that assigns events to positive (I=1) or negative (I=0) classes by estimating conditional probabilities given input features.
• It employs kernel-based techniques, Bayesian calibration, and maximum-likelihood strategies to achieve robust probability estimation, even under class imbalance and label scarcity.
• The framework integrates analytic decision rules, threshold optimization, and information-based metrics to ensure scalability and reliable performance in real-world applications.

A binary classifier P(IK) produces a rule that assigns events or observations to one of two classes, typically labeled as positive (I=1) or negative (I=0), with the output representing either a class label or a conditional probability P(I|K)—the probability that I=1 given input K. Modern approaches employ various learning strategies, calibration techniques, and theoretical frameworks for robust and interpretable probability estimation, notably under practical constraints such as class imbalance, label scarcity, and large-scale computation. This article presents a comprehensive technical overview of P(IK), its analytic foundations, algorithmic procedures, calibration principles, theoretical guarantees, and performance metrics.

1. Analytic Binary Classification via Weighted Integral Probability Metrics

The "Principled analytic classifier for positive–unlabeled learning via weighted integral probability metric" (Kwon et al., 2019) establishes a kernel-based PU classifier for the scenario where only positive and unlabeled samples are available. Suppose $n_+$ positive instances $X^+_i \sim P_{X|Y=1}$ and $n_u$ unlabeled instances $$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$ are observed, with the aim to construct a sign-based binary decision rule $f: \mathcal{X} \rightarrow \mathbb{R}$.

The algorithm minimizes the hinge risk $R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$ using the weighted integral probability metric (WIPM)

$$\text{WIPM}(P, Q; w, \mathcal{F}) := \sup_{f \in \mathcal{F}} [ \mathbb{E}_{P}[f(X)] - w \mathbb{E}_{Q}[f(X)] ],$$

where $w = 2\pi_+$, $P = P_X$, $Q = P_{X|Y=1}$, and $X^+_i \sim P_{X|Y=1}$0 is a closed ball in RKHS $X^+_i \sim P_{X|Y=1}$1 for reproducing kernel $X^+_i \sim P_{X|Y=1}$2.

The optimal classifier is determined analytically as

$X^+_i \sim P_{X|Y=1}$3

where $X^+_i \sim P_{X|Y=1}$4, $X^+_i \sim P_{X|Y=1}$5 are kernel-mean embeddings, which leads directly to a test rule based on the WMMD score

$X^+_i \sim P_{X|Y=1}$6

Classification is performed via a threshold at $X^+_i \sim P_{X|Y=1}$7:

• If $X^+_i \sim P_{X|Y=1}$8, assign $X^+_i \sim P_{X|Y=1}$9; otherwise, $n_u$0.

This method avoids matrix inversion and relies only on kernel sums, making it highly scalable for large $n_u$1.

2. Calibration of Binary Classifiers: Bayesian and Local Regression Approaches

Accurate probability interpretation of classifier outputs demands calibration such that, for predicted probability $n_u$2, the empirical frequency of $n_u$3 approaches $n_u$4. The Bayesian nonparametric binnings framework ("Binary Classifier Calibration: Bayesian Non-Parametric Approach" (Naeini et al., 2014)) defines two methods:

• SBB (Selection over Bayesian Binnings): Selects the optimal binning model for mapping raw scores to calibrated probabilities using Beta priors and exact Bayesian model selection.
• ABB (Averaging over Bayesian Binnings): Averages posterior means of all possible binnings to smooth the calibration.

Both employ histogram binning, Beta-binomial integral computation, and dynamic programming for $n_u$5 training time, providing well-calibrated scores $n_u$6 via posterior estimates.

The "Local Calibration Score" (LCS) (Machado et al., 2024) is introduced as a differentiable, locally sensitive measure of calibration error by LOESS/regression of $n_u$7 on $n_u$8. Local regression recalibration (LOESS) further refines calibration by fitting a polynomial function $n_u$9 across the score axis.

Key global calibration metrics include:

• Brier Score: Mean squared error between predicted probability and observed class.
• ECE (Expected Calibration Error): Average deviation between empirical accuracy and predicted confidence across bins.
• LCS: Weighted square deviation of smoothed calibration curve from the identity.

3. Class-Prior Shift and Maximum-Likelihood Estimation

Classifiers trained on one class balance may exhibit bias if deployed under a different class-prior. The maximum-likelihood method for prior estimation (Puts et al., 2021) corrects for such bias by solving

$$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$0

where $$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$1, $$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$2 are the densities of classifier outputs $$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$3 under the positive and negative classes as learned on training data.

The MLE $$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$4 satisfies

$$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$5

For two score-values, a closed-form solution exists; otherwise, numerical optimization (e.g., Newton-Raphson) is required.

Empirical studies demonstrate robust and unbiased estimation of the true positive rate using only unlabeled classifier outputs and the score-density separation learned previously.

4. Posterior Probability Estimation via Class-Prior Reweighting

A classifier need not output calibrated scores; one can estimate $$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$6 via "prior-variation" (Nalbantov et al., 2019). By varying the assumed class priors $$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$7 and retraining the classifier, the value at which $$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$8 lies on the decision boundary $$X^u_j \sim P_X = \pi_+ P_{X|Y=1} + \pi_- P_{X|Y=-1}$$9 yields the density ratio $f: \mathcal{X} \rightarrow \mathbb{R}$0.

The posterior probability under the original priors is computed as

$f: \mathcal{X} \rightarrow \mathbb{R}$1

This approach is agnostic to model type and score calibration, relying solely on the classifier's capacity to identify the 50/50 classification under prior reweighting.

Computational cost is $f: \mathcal{X} \rightarrow \mathbb{R}$2 per test point (for classifier retraining time $f: \mathcal{X} \rightarrow \mathbb{R}$3 and accuracy $f: \mathcal{X} \rightarrow \mathbb{R}$4).

5. Optimality and Threshold Estimation under Non-Decomposable Performance Metrics

"Binary Classification with Karmic, Threshold-Quasi-Concave Metrics" (Yan et al., 2018) generalizes binary classifier design beyond accuracy to complex, possibly non-decomposable metrics. Let $f: \mathcal{X} \rightarrow \mathbb{R}$5, and measure utility $f: \mathcal{X} \rightarrow \mathbb{R}$6 as a function of confusion matrix elements.

If the metric $f: \mathcal{X} \rightarrow \mathbb{R}$7 is Karmic (utility strictly increases with TP/TN) and satisfies threshold quasi-concavity (utility vs. threshold $f: \mathcal{X} \rightarrow \mathbb{R}$8 is unimodal), the Bayes-optimal classifier is $f: \mathcal{X} \rightarrow \mathbb{R}$9 with unique $R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$0 determined by the fixed-point equation involving $R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$1.

A two-step plug-in estimator trains a regression for $R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$2 (e.g., logistic regression, kernel smoothing), then numerically optimizes $R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$3 on held-out data to maximize empirical utility $R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$4. Statistical error bounds depend on the estimator's rate ($R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$5) and margin exponent ($R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$6), yielding excess utility bounds of $R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$7.

6. Performance Metrics and Information-Based Evaluation

Normalized Mutual Information (NI) (0711.3675) quantifies the informativeness of binary classifiers relative to class entropy. For classes $R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$8 and predictions $R_{\text{hinge}}(f) = \mathbb{E}[L_h(Yf(X))]$9, the asymmetric NI normalization is

$$\text{WIPM}(P, Q; w, \mathcal{F}) := \sup_{f \in \mathcal{F}} [ \mathbb{E}_{P}[f(X)] - w \mathbb{E}_{Q}[f(X)] ],$$0

with $$\text{WIPM}(P, Q; w, \mathcal{F}) := \sup_{f \in \mathcal{F}} [ \mathbb{E}_{P}[f(X)] - w \mathbb{E}_{Q}[f(X)] ],$$1 and $$\text{WIPM}(P, Q; w, \mathcal{F}) := \sup_{f \in \mathcal{F}} [ \mathbb{E}_{P}[f(X)] - w \mathbb{E}_{Q}[f(X)] ],$$2 computed from empirical counts (TP, FP, TN, FN). Closed-form expressions for NI in terms of accuracy ($$\text{WIPM}(P, Q; w, \mathcal{F}) := \sup_{f \in \mathcal{F}} [ \mathbb{E}_{P}[f(X)] - w \mathbb{E}_{Q}[f(X)] ],$$3), precision ($$\text{WIPM}(P, Q; w, \mathcal{F}) := \sup_{f \in \mathcal{F}} [ \mathbb{E}_{P}[f(X)] - w \mathbb{E}_{Q}[f(X)] ],$$4), recall ($$\text{WIPM}(P, Q; w, \mathcal{F}) := \sup_{f \in \mathcal{F}} [ \mathbb{E}_{P}[f(X)] - w \mathbb{E}_{Q}[f(X)] ],$$5), and class-imbalance $$\text{WIPM}(P, Q; w, \mathcal{F}) := \sup_{f \in \mathcal{F}} [ \mathbb{E}_{P}[f(X)] - w \mathbb{E}_{Q}[f(X)] ],$$6 are

$$\text{WIPM}(P, Q; w, \mathcal{F}) := \sup_{f \in \mathcal{F}} [ \mathbb{E}_{P}[f(X)] - w \mathbb{E}_{Q}[f(X)] ],$$7

and equivalently in terms of false alarm (F), hit rate (H).

NI penalizes unbalanced mistake patterns that may inflate accuracy without true information gain, providing a summary that reflects both discrimination and balance in class decisions.

Table: Key Metrics and Evaluation Criteria

Metric	Definition/Computation	Typical Usecase
WMMD Score	Ratio of positive/unlabeled kernel means	PU classification (Kwon et al., 2019)
Brier Score	Mean squared error of predicted probabilities	Calibration (Machado et al., 2024)
ECE	Mean absolute bin-wise calibration error	Calibration evaluation
LCS	Weighted squared deviation of calibration curve	Local calibration sensitivity
NI	Normalized mutual information from confusion	Informativeness assessment

7. Practical Considerations and Empirical Insights

Computational scalability, robustness to misspecified class priors, and calibration integrity are essential for real-world deployment. WMMD-based classifiers (Kwon et al., 2019) and prior-variation methods (Nalbantov et al., 2019) offer high efficiency and bypass costly hyperparameter optimization. LOESS calibration (Machado et al., 2024) provides both visualization and effective recalibration.

Empirical benchmarks demonstrate:

• WMMD classifier achieving top accuracy and AUC with orders-of-magnitude speedup compared to PU-SVM, logistic, and double-hinge baselines, as well as robustness to $$\text{WIPM}(P, Q; w, \mathcal{F}) := \sup_{f \in \mathcal{F}} [ \mathbb{E}_{P}[f(X)] - w \mathbb{E}_{Q}[f(X)] ],$$8 estimation errors.
• Maximum-likelihood prior estimation correcting bias from class proportion shifts with small variance.
• NI highlighting classifier designs that balance both accuracy and class information.

A plausible implication is that optimal binary classification, probability estimation, and calibration increasingly rely on analytic, computationally tractable solutions that integrate kernel methods, Bayesian nonparametrics, and performance-oriented threshold estimation, especially in high-dimensional or weakly labeled settings.