LOCO: Leave-One-Country-Out Evaluation

• LOCO is a domain shift protocol that holds one country as a target while models are trained on pooled data from the other countries.
• It is applied to childhood anemia prediction across 16 DHS datasets, evaluating performance with metrics like AUC-ROC, Brier score, and ECE.
• The approach is extended to reverse-LOCO and few-shot setups, highlighting variations in transferability, calibration, and subgroup behavior.

Leave-One-Country-Out (LOCO) is a cross-country evaluation protocol in which one country is held out as the target domain while models are trained on pooled data from all remaining countries. In "Few-shot Cross-country Generalization of Tabular Machine Learning and Foundation Models for Childhood Anemia Prediction under Distribution Shift" (Brima et al., 26 May 2026), LOCO is used to measure how childhood anemia prediction models generalize under cross-country distribution shift across 16 nationally representative DHS datasets. In that study, each country is treated as a domain, the same binary anemia label is used in all countries, and the protocol is complemented by reverse-LOCO and few-shot analyses. The acronym is not universal: in a different literature, LOCO denotes "Leave Out COvariates," a variable-importance parameter rather than a held-out-country validation design (Verdinelli et al., 2021).

1. Formal definition as a country-held-out domain shift protocol

The study formalizes LOCO on country-specific datasets $\mathcal{D}_1,\ldots,\mathcal{D}_{16}$. For each country $k \in \{1,\dots,16\}$, the training set is defined as

$$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$

and the test set is the held-out country $\mathcal{D}_k$. Models are trained on $\mathcal{D}_{-k}$ and evaluated on $\mathcal{D}_k$, iteratively for each country. The paper explicitly characterizes this as a classic Leave-One-Domain-Out setup in which each country constitutes a domain (Brima et al., 26 May 2026).

The per-fold workflow is fixed. For a given held-out country $k$, pooled children from the other 15 countries form the training data, and all children in country $k$ form the test data. Imputation and standardization are fit on $\mathcal{D}_{-k}$ only and then applied to $\mathcal{D}_k$, which prevents leakage. Logistic Regression, XGBoost, and LightGBM are retrained from scratch for each fold, with hyperparameters optimized within cross-validation on the training data. TabPFN v2.6 is not gradient-trained in LOCO; it is used as a pre-trained foundation model and receives a support set drawn from $k \in \{1,\dots,16\}$0 at inference time.

The same study introduces reverse-LOCO as a directional variant. Instead of training on all but one country, reverse-LOCO trains on a single source country $k \in \{1,\dots,16\}$1 and evaluates on each target country $k \in \{1,\dots,16\}$2. For 16 countries, all ordered pairs $k \in \{1,\dots,16\}$3 with $k \in \{1,\dots,16\}$4 are considered. This design does not answer the same question as LOCO. Standard LOCO asks how well pooled external countries generalize to one unseen country, whereas reverse-LOCO asks how well a specific source country transfers to each other country.

2. Data harmonization, predictors, and outcome coherence

The empirical LOCO implementation uses DHS "children's recode" data from 16 countries between 2008 and 2024, spanning Africa, Asia, Latin America, the Caucasus, and the Middle East. After harmonized preprocessing, the analytic cohort consists of $k \in \{1,\dots,16\}$5 children aged 6-59 months from Gabon, Malawi, Rwanda, Sierra Leone, Haiti, Jordan, Armenia, Kyrgyz Republic, Liberia, Bolivia, DR Congo, Guatemala, Ghana, Gambia, Cameroon, and Mali (Brima et al., 26 May 2026).

Predictors are harmonized across countries and grouped into child-level, maternal, household or socioeconomic, and geographic variables. Child-level variables include age, sex, anthropometric z-scores, recent illnesses, deworming, and vitamin A supplementation. Maternal variables include age, age at first birth, education, and smoking. Household and socioeconomic variables include wealth index, urban or rural residence, water and sanitation, household structure, and parity. Altitude is included as a geographic variable and is also used in WHO hemoglobin adjustment.

Preprocessing is deliberately country-agnostic at test time. Categorical variables are numerically encoded. Continuous predictors are z-standardized using training data only. Missing predictors are imputed using median values for continuous variables and mode values for categorical variables, again using training-only statistics. This is central to the LOCO design because any preprocessing fitted on the held-out country would compromise the intended cross-country generalization test.

The target is binary childhood anemia status derived from DHS hemoglobin indicator hw57, based on WHO age- and altitude-adjusted thresholds. No anemia is defined as hemoglobin $k \in \{1,\dots,16\}$6 g/dL, and anemia as hemoglobin $k \in \{1,\dots,16\}$7 g/dL, collapsing mild, moderate, and severe anemia. Children with missing hemoglobin are excluded, and only ages 6-59 months are retained. The use of the same binary label in all countries makes cross-country training and testing coherent.

3. Models, metrics, and inferential procedure

Four models are compared under LOCO: Logistic Regression with L2 regularization, XGBoost, LightGBM, and TabPFN v2.6 (Brima et al., 26 May 2026). Logistic Regression, XGBoost, and LightGBM are retrained from scratch on $k \in \{1,\dots,16\}$8 for each LOCO fold. Hyperparameter tuning uses Optuna with TPE Bayesian optimization and 30 trials per model. TabPFN v2.6 is treated differently: it is a pre-trained tabular foundation model with frozen parameters, and its predictions are produced through in-context learning from a support set.

Evaluation combines discrimination, classification, and calibration. The primary metric is AUC-ROC. Accuracy, precision, recall, F1, and Macro-F1 are also reported. Calibration is assessed with Brier score and Expected Calibration Error (ECE). The study gives the definitions

$k \in \{1,\dots,16\}$9

and

$$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$0

For each LOCO fold, metrics are computed on the held-out country, and 95% BCa bootstrap confidence intervals are obtained by bootstrap resampling from the test country with 5,000 iterations.

A separate but related low-data protocol is also studied. Within each country, training subsets of 1%, 5%, 10%, and 20% are sampled, and the remainder is used as a held-out test set. Supervised models are retrained from scratch at each sample size, whereas TabPFN receives the sampled subset as its support set and does not undergo gradient-based parameter updates. The paper does not define a fully formal "LOCO + few-shot target-country labels" protocol, but it reports experiments in which a small fraction of labeled samples from the target country are included in TabPFN's support set, which functions as a domain adaptation scenario without fine-tuning.

4. Empirical behavior under held-out-country evaluation

The principal LOCO result is that held-out AUC-ROC values span 0.58 to 0.69, with most country-level estimates clustering between 0.62 and 0.66 (Brima et al., 26 May 2026). The study reports that LOCO performance is stable and driven by country context. Inter-model differences are typically $$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$1 AUC, confidence intervals overlap in every country, and inter-country variation is much larger than inter-model variation.

Country-specific examples illustrate this pattern. For Guatemala as the held-out country, AUC-ROC values are 0.678 for LogReg, 0.688 for LightGBM, 0.701 for XGBoost, and 0.690 for TabPFN. For Ghana, the corresponding values are 0.668, 0.569, 0.602, and 0.593. For Liberia, they are 0.594, 0.584, 0.576, and 0.585. Bolivia is notable because LogReg is essentially non-discriminative at 0.495, while LightGBM, XGBoost, and TabPFN reach 0.658, 0.652, and 0.638, respectively.

The study contrasts LOCO with within-country evaluation. Within-country AUC-ROC ranges from 0.59 to 0.76, while LOCO ranges from 0.58 to 0.69. The drop is described as modest, but the target country matters more than the learner. The authors therefore conclude that predictive performance in childhood anemia modeling is constrained more by cross-population heterogeneity and data characteristics than by model architecture.

Calibration results further differentiate the models. The calibration summary across all 16 countries reports mean Brier score $$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$2 and mean ECE $$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$3 for TabPFN, compared with $$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$4 and $$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$5 for XGBoost, $$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$6 and $$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$7 for LogReg, and $$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$8 and $$\mathcal{D}_{-k} = \bigcup_{j \neq k} \mathcal{D}_j,$$9 for LightGBM. The detailed description notes that LOCO-specific calibration is not broken out separately, but the overall patterns are described as consistent with LOCO and within-country evaluations.

5. Directional transfer, subgroup behavior, and SHAP interpretation

Reverse-LOCO exposes directional transferability that standard LOCO can mask. Across ordered source-target country pairs, AUC-ROC values are mostly 0.55-0.72, lower and more variable than pooled-country LOCO (Brima et al., 26 May 2026). Guatemala, Haiti, Ghana, Cameroon, DR Congo, and Bolivia are described as good source domains because training on them yields relatively high AUCs across many targets. Armenia and Liberia are described as difficult targets because many incoming transfers remain near 0.5. The study emphasizes strong directionality: a high AUC from $\mathcal{D}_k$0 does not imply a high AUC from $\mathcal{D}_k$1.

Subgroup analyses are conducted within the LOCO framework by evaluating the held-out country across child age groups, child sex, maternal education, residence, and household wealth quintiles. Armenia, Bolivia, and Jordan have no viable subgroup partitions because of small $\mathcal{D}_k$2 or single-class subgroups. Across the remaining 13 countries, AUC-ROC by subgroup is broadly similar within a given country, and no systematic pattern of one subgroup being consistently advantaged or disadvantaged across countries or models emerges. Residence differences are relatively small, while some larger subgroup differences appear in Sierra Leone, Liberia, Ghana, Gabon, and Gambia, especially across age groups and maternal education levels.

Interpretability is provided through SHAP. For LogReg, the study uses linear SHAP; for XGBoost and LightGBM, TreeSHAP; and for TabPFN, the shapiq framework with a remove-and-recontextualize strategy for in-context-learning models. Across all four models, the dominant predictors are child age, altitude, and height-for-age z-score, followed by wealth index and maternal education. Weight-for-age and weight-for-height are moderately important, whereas illness variables, deworming, vitamin A, and maternal smoking are lower-tier. Country-level variation in importance is also reported: altitude is especially important in Bolivia, Guatemala, Kyrgyz Republic, Rwanda, and Jordan, and less important in Gambia, Sierra Leone, and Liberia.

6. Extensions, limits, and the competing meaning of LOCO

The study's few-shot results qualify the interpretation of LOCO in low-resource settings. In low-data within-country regimes, especially with fewer than approximately 200 samples, TabPFN consistently outperforms LogReg, XGBoost, and LightGBM in AUC-ROC, with gaps of about 0.10-0.15 in some extreme low-shot settings such as Guatemala, Haiti, Malawi, and Gambia (Brima et al., 26 May 2026). In full-data settings, differences shrink to $\mathcal{D}_k$3 AUC and confidence intervals overlap. This suggests that foundation models are most distinctive in data-scarce regimes and calibration, rather than in full-data LOCO discrimination.

Several methodological lessons are drawn directly from the study. LOCO should be paired with reverse-LOCO because pooled held-out-country performance can conceal asymmetric transferability. Calibration metrics should accompany AUC in health risk prediction. Source-country selection matters because naive pooling may be suboptimal. The paper also notes important limitations: only 16 countries are available, temporal drift across 2008-2024 cannot be corrected by LOCO, the DHS predictor set omits context-specific covariates such as malaria prevalence and hemoglobinopathies, missingness is high, and the IID assumption within training sets ignores underlying distribution shifts. The authors state that LOCO results should therefore be interpreted as lower bounds on potential performance and as sensitive to population composition.

A separate terminological issue arises because "LOCO" has another established technical meaning. In "Decorrelated Variable Importance" (Verdinelli et al., 2021), LOCO means Leave Out COvariates, not Leave-One-Country-Out. There, the original LOCO parameter is

$\mathcal{D}_k$4

which is a nonparametric variable-importance measure based on the drop in Bayes risk when covariates $\mathcal{D}_k$5 are included alongside $\mathcal{D}_k$6. That paper explicitly notes that it does not discuss cross-validation-style leave-one-country-out methods. Accordingly, in contemporary technical usage, LOCO may denote either a domain-held-out generalization protocol or a leave-out-covariate importance parameter, and the intended meaning is determined by context.