Leave-One-Subject-Out CV Overview

• LOSO CV is a cross-validation technique designed for repeated-measures data that evaluates model generalization by excluding each subject's data in turn.
• It prevents information leakage by independently preprocessing and fitting models on training data, which is crucial in domains like neuroimaging and medical diagnostics.
• LOSO CV facilitates detailed subject-level performance assessment and parameter tuning despite higher computational costs compared to traditional methods.

Leave-One-Subject-Out Cross-Validation (LOSO CV) is a validation methodology specifically designed for clustered, longitudinal, or repeated-measures data, where each subject contributes multiple observations. LOSO CV evaluates a predictor’s ability to generalize to unseen subjects by systematically holding out every subject in turn as a test set and fitting the model only on the remaining data. This approach is widely used in biometrics, neuroimaging, medical diagnostics, and all research domains where intra-subject correlation is significant and subject-level generalization is of primary interest.

1. Fundamental Principles and Protocol

In LOSO CV, for a data set with $S$ subjects, there are exactly $S$ cross-validation folds. Each fold $i$ withholds subject $i$’s entire data (such as all measurements, trials, or derived features) for model evaluation, while all model fitting and preprocessing steps are carried out solely on the $S-1$ remaining subjects. Notably, no information or parameter (including normalization, filtering coefficients, transformation bases, or feature selection) is allowed to “leak” from the test subject into the training process or preprocessing pipeline; all steps are applied independently for training and test sets within each fold (Hamidi et al., 2023).

The protocol implemented in Parkinson’s disease EEG classification (Hamidi et al., 2023) proceeds as follows:

• Each subject’s raw data is fully preprocessed anew in each fold (e.g., band-pass filtering, ICA artifact rejection, epoching) after defining the fold split.
• Feature extraction (e.g., time-frequency transforms, image representations) is performed strictly per fold.
• The model (convolutional neural network in this case) is trained only on training subjects’ processed data.
• The held-out subject’s samples serve as the test set, producing one fold-specific accuracy measure.

This paradigm is applicable to both classification and regression settings for clustered data (Xu et al., 2013).

2. Mathematical Formulation and Evaluation Metrics

Let $E_i$ denote the performance metric (typically accuracy for classification or mean squared error for regression) on the $i$th held-out subject. The aggregate LOSO cross-validation estimate is the mean across subjects:

$E_{\text{LOSO}} = \frac{1}{S} \sum_{i=1}^S E_i.$

Other reporting conventions may aggregate performance across all test samples in all folds (epoch-level accuracy), or average per-subject results (subject-level accuracy or loss), depending on granularity and downstream interpretation (Hamidi et al., 2023).

Standard classification metrics used in LOSO CV include:

• Accuracy: $\mathrm{ACC} = \frac{TP + TN}{TP + TN + FP + FN}$,
• Sensitivity: $\mathrm{SENS} = \frac{TP}{TP + FN}$,
• Specificity: $\mathrm{SPEC} = \frac{TN}{TN + FP}$ (Hamidi et al., 2023).

For regression, the coefficient of determination ($R^2$) requires adjustment: the proper baseline is the training-set mean for each held-out subject, not the global mean, as detailed in Section 5 (Zliobaite et al., 2016).

3. Theoretical Properties and Asymptotic Optimality

The leave-subject-out CV criterion is formally defined for penalized spline models as:

$$\text{LsoCV}(\lambda) = \frac{1}{n} \sum_{i=1}^{n} \lVert y_i - \hat{\mu}^{[-i]}(X_i)\rVert^2,$$

where $y_i$ are all observations from subject $i$, and $\hat{\mu}^{[-i]}$ is the estimator fit excluding subject $i$.

Under regularity conditions (bounded moments, no single subject dominance, technical small-leverage assumptions), minimizing the LsoCV criterion is asymptotically optimal: it becomes equivalent (in probability) to minimizing true mean squared error loss $L(\lambda)$ and its expectation $R(\lambda)$. Thus,

$$\arg\min_\lambda \text{LsoCV}(\lambda) \approx \arg\min_\lambda L(\lambda) \approx \arg\min_\lambda R(\lambda)$$

(Xu et al., 2013).

Key proof elements rely on expressing both the LsoCV and the unbiased risk estimator $U(\lambda)$ in terms of the model’s “hat” matrix, showing their asymptotic equivalence as $n \to \infty$, and bounding the difference using matrix perturbation theory.

4. Computational Strategies

While naïve LOSO CV requires model retraining for each of the $S$ subjects—potentially with high computational cost, especially for complex models—efficiency can be improved for linear models using blockwise shortcuts.

For penalized splines, the blockwise “leave-out” formula:

$$\text{LsoCV}(\lambda) = \frac{1}{n}\sum_{i=1}^n (y_i - \hat{y}_i)^T (I_{ii} - A_{ii})^{-T}(I_{ii} - A_{ii})^{-1}(y_i - \hat{y}_i)$$

where $A_{ii}$ is the $i$th subject’s block from the “hat” matrix, reduces computational burden.

A quadratic approximation $\text{LsoCV}^*$ and Newton–Raphson optimization enable tuning of multiple penalty parameters efficiently, with each Newton step scaling as $O(p^3 + m p^2 + N p)$ for $p$ basis functions and $m$ smoothing parameters (Xu et al., 2013).

In non-linear or deep learning contexts, such as EEG-based disease classification, computational cost remains substantial: $S$ full re-trainings (e.g., 31 CNN trainings for $S = 31$) (Hamidi et al., 2023).

5. Proper Use and Adjustment of $R^2$ in LOSO CV

Using standard $R^2$ (variance explained relative to the global mean) with LOSO CV leads to severe bias: the correct denominator in each fold is the sum of squared deviations from the training-set mean (excluding the test subject), not the grand mean.

For subjects of equal size $k$,

$$R^2_{\text{loso}} = \frac{R^2 - R^2_{\text{naive}}}{1 - R^2_{\text{naive}}}, \qquad R^2_{\text{naive}} = 1 - \left(\frac{n}{n-k}\right)^2,$$

with $n$ total samples. For unbalanced subject sizes, compute directly:

$$R^2_{\text{loso}} = 1 - \frac{ \sum_{i=1}^n (y_i - \hat{y}_i)^2 }{ \sum_{i=1}^n (y_i - \bar{y}_{(-h(i))})^2 },$$

where $\bar{y}_{(-h(i))}$ denotes the mean of the training set for the fold withholding subject $h(i)$ (Zliobaite et al., 2016).

This correction is essential for all regression contexts utilizing LOSO CV.

6. Empirical Application: EEG-Based Disease Classification

Application of LOSO CV in Parkinson’s disease classification with EEG data illustrates the methodology’s implementation and practical consequences (Hamidi et al., 2023):

• Dataset: $S=31$ subjects (16 healthy controls, 15 PD).
• LOSO protocol: Each subject’s processed epochs withhold their whole data as a test set.
• Deep network: 6-layer CNN trained de novo per fold, evaluated strictly on the withheld subject.
• Epoch-level accuracy across all folds: $90.32\%$.
• Subject-level mean accuracy (thresholded at 50% correct epochs): $86.80\%$.

Notably, per-subject test-set accuracy varies widely (from near $0\%$ to $99\%$), exposing inter-subject heterogeneity and outlier effects. The study demonstrates LOSO CV’s realism in simulating deployment on previously unseen individuals.

Subject Cohort	Range of Test Accuracies	Number of Subjects
Healthy (HC)	$6.25\%$ -- $98.97\%$	16
PD	$0.00\%$ -- $98.95\%$	15

7. Advantages, Limitations, and Extensions

Advantages:

• Rigorous protection against subject-level information leakage.
• Robust estimate of generalization to new subjects or clusters, reflecting actual deployment conditions (Hamidi et al., 2023).
• Asymptotically optimal for tuning parameters in penalized and semiparametric models (Xu et al., 2013).

Limitations:

• Markedly increased computational burden for $S$-fold full-model fitting, especially with large $S$ or expensive learners (Hamidi et al., 2023).
• High inter-subject variability may arise, as in cases with rare disease phenotypes or response profiles.
• Reduced effective size for training sets in each fold, possibly diminishing estimator stability for small sample sizes.

Extensions:

• Blockwise formulas and quadratic approximations enable efficient LOSO computation in certain linear models (Xu et al., 2013).
• LOSO CV serves as a basis for correlation-structure selection (e.g., by comparing mean squared error under various candidate $W$ matrices in penalized spline models) (Xu et al., 2013).
• For regression, proper $R^2$ adjustment must be employed as detailed above (Zliobaite et al., 2016).

LOSO CV is the gold standard whenever predictive generalization to unseen subjects is required and subject-wise grouping structures are inherent to the data. It remains the preferred approach in neuroimaging, personalized medicine, and longitudinal analysis for both methodological rigor and empirical validity.