K-fold Personalization Test (KPT)

• The paper introduces KPT as a rigorous inferential method that quantifies the value gap between optimal personalized policies and constant actions.
• KPT relies on a repeated K-fold, cross-fitted augmented inverse propensity weighted scheme to ensure unbiased estimation and strong type-I error control.
• Its design segregates training, evaluation, and nuisance estimation, leading to improved stability and power compared to simpler evaluation methods.

K-fold Personalization Test (KPT) is a statistical hypothesis test for whether a learned personalized intervention policy improves expected outcome relative to the best single intervention applied to everyone. In its formal version, KPT is a cross-fitted, doubly robust, repeated K-fold sample-splitting test developed for i.i.d. historical data with covariates, actions, and outcomes, and it targets the value gap between the best policy in a chosen personalization class and the best constant action (Li et al., 9 Jul 2026). Earlier personalization literature supplied several ingredients later associated with KPT—most notably a weighted local/global personalization objective under privacy constraints, subject-specific within-subject splitting protocols, and on-device holdout evaluation—but did not define KPT explicitly (Brasher et al., 2018, Zunino et al., 2016, Wang et al., 2019).

1. Conceptual scope and historical antecedents

KPT is not a generic synonym for cross-validation under personalization. Its direct formulation is inferential: given a policy class $\Pi$, the question is whether the best policy in that class has strictly higher expected value than the best constant action. The test is therefore about the benefits of personalization, not merely about predictive accuracy or the existence of heterogeneous treatment effects. The underlying paper emphasizes that heterogeneous treatment effects are necessary, but not sufficient: personalization helps only if different interventions are optimal for different subgroups (Li et al., 9 Jul 2026).

Earlier work anticipated distinct parts of this logic without defining KPT itself. In a privacy-constrained personalization setting, one paper defined personalization as the relative weighting between performance on user-specific data and performance on a large, multi-user global dataset, with the global term serving as regularization against overfitting small per-user datasets (Brasher et al., 2018). In human action recognition, a distinct line of work introduced a subject-specific “Personalization” strategy based on repeated random $2/3$–$1/3$ splits within each subject’s own repetitions, but it was not a genuine K-fold protocol (Zunino et al., 2016). In federated learning, Federated Personalization Evaluation (FPE) operationalized on-device personalization as a single temporal 80/20 train/test split on each device, again without K-fold repetition (Wang et al., 2019). This suggests that KPT emerged as a formal inferential consolidation of earlier evaluation ideas rather than as an isolated invention.

2. Formal estimand and null hypothesis

The formal KPT setup assumes i.i.d. observations

$(X_i, A_i, Y_i), \quad i=1,\dots,n,$

drawn from a distribution $\nu$, where $X_i \in \mathcal X$ are covariates, $A_i \in \mathcal A$ are actions from a finite action set, and $Y_i \in \mathcal Y$ is the observed outcome. The conditional mean outcome is

$\mathbb E[Y_i \mid X_i=x, A_i=a] = r(x,a).$

A personalized policy is a map $\pi:\mathcal X \to \mathcal A$, and $2/3$0 denotes the policy class under consideration. The value of a policy is

$2/3$1

while the utility of a constant action $2/3$2 is

$2/3$3

The best constant action and the optimal policy in the class are

$2/3$4

The target estimand is the personalization effect

$2/3$5

Equivalently, the main text states the same quantity as

$2/3$6

KPT tests

$2/3$7

This formulation separates KPT from broader policy-learning or uplift-modeling frameworks. In particular, KPT is not simply evaluating whether a learned personalized rule performs well in isolation; it asks whether personalization outperforms deploying the best single intervention to all individuals (Li et al., 9 Jul 2026).

3. Cross-fitted repeated K-fold construction

The operative mechanism of KPT is a cross-fitted, repeated K-fold procedure. For one random split $2/3$8, the data are partitioned into $2/3$9 folds $1/3$0. For each held-out fold $1/3$1, some other folds are used to learn the personalized policy $1/3$2 and the best constant action $1/3$3; a disjoint set of folds is used to learn the outcome model $1/3$4 and the propensity model $1/3$5; and fold $1/3$6 itself is used only for final effect estimation. In the reported experiments, $1/3$7, with half the folds for policy/best-arm learning, two folds for outcome/propensity learning, and one fold held out for evaluation (Li et al., 9 Jul 2026).

The fold-level estimator is an AIPW-style contrast between the learned personalized policy and the learned best constant action: $1/3$8 The supplement expresses the same score as

$1/3$9

where $(X_i, A_i, Y_i), \quad i=1,\dots,n,$0 and

$(X_i, A_i, Y_i), \quad i=1,\dots,n,$1

For one split $(X_i, A_i, Y_i), \quad i=1,\dots,n,$2, foldwise estimates are aggregated as

$(X_i, A_i, Y_i), \quad i=1,\dots,n,$3

The procedure is then repeated over $(X_i, A_i, Y_i), \quad i=1,\dots,n,$4 random reorderings or partitions. The repeated-split estimator and variance estimator are

$(X_i, A_i, Y_i), \quad i=1,\dots,n,$5

and the test statistic is

$(X_i, A_i, Y_i), \quad i=1,\dots,n,$6

The algorithm rejects $(X_i, A_i, Y_i), \quad i=1,\dots,n,$7 if $(X_i, A_i, Y_i), \quad i=1,\dots,n,$8 for $(X_i, A_i, Y_i), \quad i=1,\dots,n,$9, and the corresponding Wald-style confidence interval is

$\nu$0

This construction is not ordinary K-fold CV for prediction error. It is also not merely sample-splitting or repeated holdout. The defining features are separation of fold roles, orthogonal AIPW scoring, and repeated K-fold aggregation (Li et al., 9 Jul 2026).

4. Statistical guarantees and asymptotic theory

KPT’s main theoretical contribution is inferential rather than algorithmic. Under $\nu$1, the test attains asymptotically valid type-I control under assumptions that include SUTVA, unconfoundedness, strong overlap, bounded reward, consistency of nuisance estimators for $\nu$2 and $\nu$3, a fast best-arm learner with $\nu$4, a product-rate condition for nuisance estimation, finite variance of the noise, nondegeneracy $\nu$5, and consistency of the learned personalized policy $\nu$6 (Li et al., 9 Jul 2026). The associated theorem states that under these conditions,

$\nu$7

under the null, and the result extends to any fixed number $\nu$8 of repeated splits.

Under stronger assumptions, KPT yields asymptotic normality and semiparametric efficiency. The additional requirement is a low-regret or fast policy learner,

$\nu$9

which is linked in the paper to standard margin assumptions. Under this stronger regime, the feasible estimator is asymptotically equivalent to an oracle AIPW estimator using the true optimal policy and the true best constant action. The oracle score $X_i \in \mathcal X$0 defines the asymptotic variance $X_i \in \mathcal X$1, and the paper shows

$X_i \in \mathcal X$2

It also derives the efficient influence function in the nonparametric model and argues that the estimator attains the semiparametric efficiency bound under the stated regularity conditions (Li et al., 9 Jul 2026).

These guarantees distinguish KPT from naive foldwise gain comparisons. A plausible implication is that KPT’s use of orthogonal scores and cross-fitting is essential to its inferential claims; a simpler train-on-$X_i \in \mathcal X$3, test-on-1 scheme may preserve the broad intuition but is not the object covered by the paper’s proofs.

5. Empirical behavior and comparison with alternative tests

The empirical evaluation compares KPT with TrainEval, SRP, and PAPD. TrainEval is a half-split procedure that trains a policy and a best arm on one half and evaluates on the other; SRP is a prior test for overall qualitative treatment effects, mainly for binary treatments; and PAPD is a cross-fold estimator for the difference between two learned policies (Li et al., 9 Jul 2026).

Across real datasets and simulations, KPT is reported to have comparable or narrower confidence intervals, much more stable z-statistics and confidence intervals across random partitions, better type-I calibration than TrainEval and PAPD, and broader applicability than SRP because it handles multi-arm actions naturally. In simulations, KPT and SRP maintained $X_i \in \mathcal X$4 type-I error, while TrainEval and PAPD exceeded $X_i \in \mathcal X$5 false rejection under the null. In the Job Corps semi-synthetic example, KPT found significant personalization benefit, whereas TrainEval failed to report significance in $X_i \in \mathcal X$6 of random runs. In the depression dataset, KPT found no significant personalization benefit, while TrainEval and PAPD occasionally produced apparently false significant positives depending on the split. In the MOOC data, KPT found no meaningful personalization gain, and in joke recommendation data it found a clear positive personalization effect. The central practical lesson stated in the source is that repeated cross-fitted AIPW aggregation is much more stable than single-split train/evaluate testing (Li et al., 9 Jul 2026).

KPT is therefore presented as a tool for deciding whether personalization is worthwhile, not as a substitute for the final policy-learning stage. The paper notes that after significance one can retrain on the full dataset, but the test itself addresses the prior question of whether a chosen personalization class appears to improve expected outcome over the best global action.

6. Relation to neighboring evaluation frameworks, design choices, and limitations

The broader personalization literature clarifies both what KPT is and what it is not. Privacy-aware personalization metrics define a weighted tradeoff

$X_i \in \mathcal X$7

between user-specific and global loss, treating the global term as regularization against overfitting sparse user data; this is conceptually close to personalization scoring, but it is not a fold-based inferential test (Brasher et al., 2018). Federated Personalization Evaluation uses on-device local train/test splits, specifically an 80/20 temporal split, and aggregates only metric deltas across users; it is structurally close to one fold of a KPT but not itself K-fold (Wang et al., 2019). Subject-specific “Personalization” in action recognition uses one classifier per subject and repeated random within-subject $X_i \in \mathcal X$8–$X_i \in \mathcal X$9 splits; it is the closest analogue to a subject-personalized fold protocol in that literature, but again not a genuine K-fold design (Zunino et al., 2016).

Related research also sharpens several methodological cautions. Experiment-design work on personalization emphasizes that allocation of users to treatment and analysis groups determines both the observable effect size and the MDE; this suggests that a KPT design must be judged not only by asymptotic inference but also by how fold construction affects power under sparse qualification and traffic fragmentation (Liu et al., 2020). Cross-validation research argues that conventional choices such as $A_i \in \mathcal A$0 or 80:20 implicitly encode assumptions about predictive accuracy and evaluation uncertainty, and that the optimal $A_i \in \mathcal A$1 depends on both data and model (McAlinn et al., 16 Nov 2025). Other work proposes irredundant $A_i \in \mathcal A$2-fold cross-validation, in which each instance is used exactly once for training and once for testing across the full procedure, highlighting that repeated training exposure across folds can itself be a source of optimism; this is not personalization-specific, but it is directly relevant to KPT when users, sessions, or tasks are repeatedly reused across folds (Aguilar-Ruiz, 26 Jul 2025).

KPT’s own limitations are explicit. It applies to randomized data directly and to observational data only under unconfoundedness and overlap; it assumes finite or discrete action spaces; it is not designed for continuous action spaces, extremely large action spaces without additional structure, dependent data, adaptive online data, or severe overlap violations (Li et al., 9 Jul 2026). The null and alternative are relative to the chosen policy class $A_i \in \mathcal A$3, so failure to reject means only that there is no evidence of personalization benefit in that class. The strongest guarantees also rely on unique best-arm and margin-type conditions. These constraints are not peculiar to KPT, but they mark the boundary between the intuitive idea of a “K-fold personalization test” and the specific statistically rigorous object that bears that name in the current literature.