IDB in ECG Contrastive Pretraining

• In-Distribution Batch (IDB) is a batching strategy that forms single-cohort batches from ECG data, ensuring contrastive negatives stem from the same distribution.
• It mitigates cohort-specific technical artifacts by forcing models to focus on intra-patient and physiological signal variations, thereby enhancing out-of-distribution performance.
• IDB redefines the contrastive loss denominator by limiting negatives to the same cohort, which leads to significant improvements in downstream tasks across multi-center datasets.

In-Distribution Batch (IDB) is a batching strategy introduced for contrastive pretraining of electrocardiogram foundation models in heterogeneous multi-centre data settings. In its explicit formulation, “each training batch is composed exclusively of samples from a single cohort, ensuring that contrastive negative pairs originate from the same distribution” (Khattak et al., 12 Sep 2025). In this usage, “in-distribution” is defined at the level of cohort, site, or domain within training, not in the broader sense of test-time in-distribution evaluation. The method was proposed to preserve “intra-cohort consistency” during pretraining and to reduce the tendency of contrastive objectives to encode cohort-specific technical artifacts rather than patient- or physiology-relevant structure (Khattak et al., 12 Sep 2025).

1. Terminology and scope

Within the supplied literature, the exact term In-Distribution Batch (IDB) is explicit in the ECG contrastive-learning work that proposes the strategy. There, the relevant distributions are the pretraining cohorts BIDMC, CODE, SHZS, and VUMC, and a batch is “in-distribution” when all samples in that batch come from one such cohort (Khattak et al., 12 Sep 2025).

Closely related papers use adjacent ideas without standardizing the same term. In decentralized learning via In-Distribution Knowledge Distillation, the supplied supplementary material does not define an explicit term “In-Distribution Batch”; the closest paper-consistent interpretation is a node-specific selected subset of a shared public dataset that is judged in-distribution relative to a node’s local data before distillation (Ravikumar et al., 2023). In unsupervised OoD detection with BatchNorm, the relevant language is “a batch of in-distribution samples” and mixed batches of in-distribution and test samples, again without an IDB definition (Song et al., 2019). In offline meta-RL, the corresponding notion is “in-distribution online adaptation” via an uncertainty-filtered context $C_{\text{in}}$, which operates on episodes rather than minibatches (Wang et al., 2023).

2. Motivation in multi-cohort contrastive pretraining

The IDB strategy was proposed in the setting of CAPE (“Contrasting by Augmented Patient Electrocardiograms”), a patient-level contrastive learning framework. CAPE learns representations by making ECGs from the same patient similar and ECGs from different patients in the same batch dissimilar. When pretraining used the combined four-cohort dataset BCSV—BIDMC (US), CODE (Brazil), SHZS (China), and VUMC (US)—the resulting multi-centre model, CAPE-X, achieved the best average in-distribution performance for downstream age and sex prediction. However, when downstream supervised models were trained on one cohort, especially BIDMC, and then tested on external cohorts, the representations from CAPE-X degraded markedly out of distribution (Khattak et al., 12 Sep 2025).

The central diagnosis is that standard random batching across multiple cohorts creates an easy shortcut for the contrastive objective. Because negatives in the same batch often come from different cohorts, the model can separate samples using cohort-specific technical signals rather than patient-specific or clinically meaningful ECG structure. The supplied analysis states that multi-centre contrastive pretraining can therefore produce a paradox: more diverse pretraining data improves in-distribution accuracy but can harm out-of-distribution generalisation because the model learns site/device/pipeline artifacts (Khattak et al., 12 Sep 2025).

This diagnosis is supported by representation analysis. In t-SNE plots, CAPE-X embeddings clustered strongly by cohort, and within CODE they also separated by recording device type. The paper attributes these effects less to patient ethnicity than to acquisition hardware and preprocessing artifacts. IDB was introduced specifically to remove that shortcut by ensuring that negative examples within a batch come from the same cohort as the anchor (Khattak et al., 12 Sep 2025).

3. Formalization and training mechanics

CAPE pretraining uses patient-level positive pairing. A positive pair consists of two different ECGs from the same patient, sampled at random times, after applying signal-preserving augmentations such as random cropping / random windows and zero masking / time masking. Negatives are all other instances in the batch, i.e. ECG views from other patients. The training setup uses batch size 1024, corresponding to 512 patients, since each patient contributes two views (Khattak et al., 12 Sep 2025).

The standard contrastive loss is InfoNCE-style:

$$l_{i,j} = - \log \frac{\exp(\mathrm{sim}(z_i, z_j)/T)} {\sum_{k=1}^{2N} \mathbb{I}_{[k \neq i]} \exp(\mathrm{sim}(z_i, z_k)/T)}$$

with cosine similarity

$$\mathrm{sim}(z_i, z_j) = \frac{z_i^\top z_j}{\|z_i\| \|z_j\|}.$$

For IDB, the paper writes a cohort-conditioned variant. If $X$ denotes a distribution or cohort, and $z_i^X, z_j^X$ are projections of augmented ECGs from the same patient within $X$, then

$$l^{X}_{i,j} = - \log \frac{\exp(\mathrm{sim}(z_i^X, z_j^X)/T)} {\sum_{k=1}^{2N} \mathbb{I}_{[k \neq i]} \exp(\mathrm{sim}(z_i^X, z_k^X)/T)}.$$

The total loss is then expressed as

$$l = \sum_{X \in \{X,Y,\ldots\}} \sum_{(i,j)\in Z_X} l^X_{i,j}.$$

Conceptually, this is not a new contrastive objective in spirit; the decisive change is the support of the denominator. In standard multi-cohort batching, the denominator contains mixed-domain negatives. Under IDB, it contains only negatives from the same distribution $X$ as the anchor. The supplied description emphasizes that the main implementation change is therefore in the data loader / batch sampler, not in the projection head, augmentations, or the overall InfoNCE machinery (Khattak et al., 12 Sep 2025).

The remaining pretraining setup is unchanged: same backbone: four-layer ResNet, same projection head setup, same patient-level positive-pair logic, same augmentations, same optimizer/training schedule, and same batch size. The reported hyperparameters are epochs: 200, optimizer: Adam, initial learning rate: 0.1, schedule: half-period cosine decay, and implementation: TensorFlow 2.10.1. What the supplied text does not specify is the detailed cohort-balancing schedule for single-cohort batches; it states only that standard CAPE-X uses random batches across multiple source cohorts, whereas CAPE-Z applies the IDB rule of single-cohort batches (Khattak et al., 12 Sep 2025).

4. Datasets and empirical evidence

The large unlabeled pretraining corpus BCSV contains over 5.2 million ECGs drawn from four cohorts: BIDMC with 1,106,886 ECGs from 127,041 patients, CODE with 1,123,903 ECGs from 424,577 patients, SHZS with 1,560,551 ECGs from 420,956 patients, and VUMC with 1,412,012 ECGs from 252,306 patients. Downstream evaluation uses BIDMC, CODE, SHZS, UK Biobank (UKB), and PTB-XL. For the principal out-of-distribution study, the downstream prediction head is trained on BIDMC features and labels, then tested on external cohorts using 10,000 random samples per cohort, over ten runs (Khattak et al., 12 Sep 2025).

The key comparison is between CAPE-X, pretrained on combined BCSV with random mixed-cohort batches, and CAPE-Z, pretrained on the same data with IDB single-cohort batches. The aggregate pattern is that IDB largely preserves source-cohort performance while dramatically improving out-of-distribution transfer.

Test cohort	Age MAE, CAPE-X → CAPE-Z	Sex AUC, CAPE-X → CAPE-Z
BIDMC	7.880(028) → 7.833(031)	0.939(06) → 0.937(08)
CODE	24.232 → 7.962(031)	0.600 → 0.935(08)
SHZS	14.326 → 7.378(020)	0.659 → 0.942(07)
UKB	5.786(018) → 5.875(017)	0.974(04) → 0.977(03)
PTB-XL	7.786(019) → 7.681(017)	0.937(05) → 0.947(06)
Mean	12.00 → 7.346	0.822 → 0.948

On BIDMC, IDB slightly improves age MAE, from 7.880(028) to 7.833(031) with $p=0.004$, while sex AUC changes from 0.939(06) to 0.937(08) with $$l_{i,j} = - \log \frac{\exp(\mathrm{sim}(z_i, z_j)/T)} {\sum_{k=1}^{2N} \mathbb{I}_{[k \neq i]} \exp(\mathrm{sim}(z_i, z_k)/T)}$$0. On CODE, the recovery is much larger: age MAE improves from 24.232 to 7.962(031) and sex AUC from 0.600 to 0.935(08). On SHZS, age MAE improves from 14.326 to 7.378(020) and sex AUC from 0.659 to 0.942(07). The text also reports external-cohort summaries for CODE and SHZS as age MAE 7.93 with sex AUROC 0.927 on CODE, and age MAE 7.24 with AUROC 0.935 on SHZS, each with corresponding 95% confidence intervals. Table 3 reports an overall mean age MAE of 12.00 for CAPE-X versus 7.346 for CAPE-Z, and mean sex AUC of 0.822 versus 0.948 (Khattak et al., 12 Sep 2025).

The representation analyses align with these results. Figure 4A shows that single-cohort pretraining on BIDMC yields a more continuous embedding geometry across cohorts. Figure 4B shows that random multi-cohort batching in CAPE-X yields distinct cohort clusters. Figure 4C attributes CODE clustering to two device types. Figure 4D shows that IDB in CAPE-Z removes obvious cohort clustering, with embeddings showing no apparent clustering or cohort-related gradients (Khattak et al., 12 Sep 2025).

5. Mechanistic interpretation

The proposed mechanism is batch-level shortcut suppression. In mixed-cohort contrastive learning, negatives often differ from the anchor in both patient identity and cohort/domain. Because the objective only requires anchors to be distinguished from negatives, the model can exploit whichever signal is easiest, including spurious cohort-distinguishing features, technical variation introduced by different acquisition devices or pipeline artifacts, or other cohort markers. Under IDB, the domain label ceases to be informative within a batch, because all negatives are same-cohort negatives. The model is therefore forced toward distinctions that exist within the cohort, such as patient-level morphology, disease-related signal variation, and other stable physiological structure (Khattak et al., 12 Sep 2025).

This suggests that IDB is best understood as a batch-composition constraint that redefines the contrastive support seen at each training step. It does not remove cohort heterogeneity from the pretraining corpus as a whole; instead, it ensures that the negative set for each anchor is distributionally local to one cohort. The supplied representation analysis indicates that this changes the geometry of the learned space from explicit domain organization under mixed batching to a more homogeneous cross-domain embedding under single-cohort batching (Khattak et al., 12 Sep 2025).

Batch-level distribution alignment also appears in neighboring literatures, although under different names and objectives. In visible-infrared person re-identification, Modality Batch Normalization (MBN) addresses two BN-induced mismatches: inter-mini-batch distribution gap and intra-mini-batch modality distribution gap. Its solution is to normalize each modality sub-mini-batch separately rather than the whole mini-batch, making each modality-conditioned sub-batch more internally consistent for normalization (Li et al., 2021). In unsupervised OoD detection with BatchNorm, test-time scoring is explicitly batch-dependent: the method evaluates a query together with a reference batch containing in-distribution samples, and the OoD signal is the change in conditional likelihood as the batch shifts from in-distribution-dominated to test-dominated composition (Song et al., 2019). In supervised optimization, In-Training Distribution Matching (ITDM) regularizes latent representations by matching the feature distributions of different mini-batches via MMD, reducing batch-specific overfitting without changing the sampler itself (Li et al., 2020).

6. Ambiguities, adjacent uses, and limits of the term

The phrase In-Distribution Batch is not standardized across the supplied literature. In decentralized learning with public-data distillation, the nearest concept is not a batch definition but a node-specific subset $$l_{i,j} = - \log \frac{\exp(\mathrm{sim}(z_i, z_j)/T)} {\sum_{k=1}^{2N} \mathbb{I}_{[k \neq i]} \exp(\mathrm{sim}(z_i, z_k)/T)}$$1 of public samples judged in-distribution relative to node $$l_{i,j} = - \log \frac{\exp(\mathrm{sim}(z_i, z_j)/T)} {\sum_{k=1}^{2N} \mathbb{I}_{[k \neq i]} \exp(\mathrm{sim}(z_i, z_k)/T)}$$2’s local data before label exchange and fine-tuning. The supplied supplementary material is explicit that it does not define an official term “In-Distribution Batch,” and it does not provide the exact subset-construction rule (Ravikumar et al., 2023).

In offline meta-RL, the nearest analogue is again a filtered subset rather than a training minibatch. IDAQ defines in-distribution context as

$$l_{i,j} = - \log \frac{\exp(\mathrm{sim}(z_i, z_j)/T)} {\sum_{k=1}^{2N} \mathbb{I}_{[k \neq i]} \exp(\mathrm{sim}(z_i, z_k)/T)}$$3

where $$l_{i,j} = - \log \frac{\exp(\mathrm{sim}(z_i, z_j)/T)} {\sum_{k=1}^{2N} \mathbb{I}_{[k \neq i]} \exp(\mathrm{sim}(z_i, z_k)/T)}$$4 is an uncertainty score on an adaptation episode. The operative unit is therefore an episode / trajectory / context, and the goal is to keep online adaptation consistent with the offline dataset rather than to alter stochastic-gradient batch composition (Wang et al., 2023).

In predictive inference, the phrase “batch” refers to a batch of test points rather than a training minibatch. Batch predictive inference assumes exchangeability between calibration and test data and constructs sets $$l_{i,j} = - \log \frac{\exp(\mathrm{sim}(z_i, z_j)/T)} {\sum_{k=1}^{2N} \mathbb{I}_{[k \neq i]} \exp(\mathrm{sim}(z_i, z_k)/T)}$$5 satisfying

$$l_{i,j} = - \log \frac{\exp(\mathrm{sim}(z_i, z_j)/T)} {\sum_{k=1}^{2N} \mathbb{I}_{[k \neq i]} \exp(\mathrm{sim}(z_i, z_k)/T)}$$6

for functionals such as the mean, median, simultaneous coverage statistics, or false-claim counts on the test batch (Lee et al., 2024). This is an in-distribution batch setting in a statistical sense, but it is not an optimization-time batching method.

A recurrent misconception is therefore to treat “IDB” as a universal machine-learning primitive. Across the supplied sources, that would be inaccurate. The exact term is explicitly defined for single-cohort contrastive batching in ECG foundation-model pretraining (Khattak et al., 12 Sep 2025). Elsewhere, the literature supports related principles—distribution-conditioned sub-batching, in-distribution reference batches, latent mini-batch alignment, or uncertainty-filtered in-distribution subsets—but does not converge on a single, cross-domain definition of IDB (Li et al., 2021).