Federated Dataset Learning: A Data-Centric View

• Federated Dataset Learning (FeDaL) is a data-centric framework that models decentralized datasets and their transformations to address non-IID challenges.
• It employs methods like synthetic-data transfer, latent feature augmentation, and dataset dictionary learning to harmonize heterogeneous data sources.
• Empirical results indicate that explicitly modeling inter-dataset relations enhances generalization and mitigates performance gaps between local and federated training.

Federated Dataset Learning (FeDaL) is used across recent federated-learning research to denote a data-centric view in which decentralized datasets, their relations, and their transformations are themselves modeled. Across the cited literature, FeDaL includes learning dataset-agnostic temporal representations, estimating cross-silo dataset similarity before training, augmenting local distributions with federation-aware feature statistics or synthetic samples, harmonizing heterogeneous silos into common analytical views, and representing local datasets through shared atoms or Wasserstein barycenters (Chen et al., 6 Aug 2025, Elhussein et al., 2024, Zhou et al., 2023, Chen et al., 2024, Castellon et al., 2023). This suggests that FeDaL is better understood as a family of methods for learning over decentralized data manifolds than as a single optimization rule.

1. Conceptual basis

FeDaL is motivated by the observation that collaboration failure in federated learning often originates in the datasets themselves. In cross-silo settings, datasets at different sites are often non-identically distributed, and this non-IID structure can degrade model performance. The cited work describes this heterogeneity in several ways: feature shift caused by acquisition differences in medical imaging, mixtures of predefined domains within each client, extremely small local datasets, and dataset-wise bias in time series arising from temporal resolution bias, physical constraint bias, and pattern transition bias (Zhou et al., 2023, Zhong et al., 2022, Kamp et al., 2021, Chen et al., 6 Aug 2025).

Under this view, FeDaL does not treat the dataset partition merely as a nuisance term in a federated objective. Instead, it treats the partition as a signal about latent domain structure. FedDAR formalizes this in Domain-mixed FL, where each client distribution is modeled as a mixture of predefined domains, and learns a domain shared representation with domain-wise personalized prediction heads (Zhong et al., 2022). FedDaDiL takes a related stance for multi-source domain adaptation by treating clients’ distributions as particular domains and jointly learning a federated dictionary of empirical distributions (Castellon et al., 2023). FLINT, in turn, reframes the learning dataset as the outcome of schema harmonization, normalization, entity linkage, and imputation, rather than as an already consistent local table (Stripelis et al., 2023).

A recurrent implication is that generalization depends on how well the federation models inter-dataset relations before or during optimization. The dataset-similarity literature makes this explicit by asking whether collaboration should occur at all; augmentation and synthesis methods instead attempt to expose each client to a richer approximation of the federation-wide distribution; domain-adaptation methods encode each client as a barycentric combination of shared atoms (Elhussein et al., 2024, Zhou et al., 2023, Chen et al., 2024, Castellon et al., 2023).

2. Methodological families

Family	Core mechanism	Representative paper
Synthetic-data transfer	DDPM learns a local data distribution and generates synthetic remote samples	(Chen et al., 2024)
Latent feature augmentation	Channel-wise mean and standard deviation are perturbed with federation-aware variances	(Zhou et al., 2023)
Dataset dictionary learning	Shared atoms and private barycentric coordinates reconstruct each dataset as a Wasserstein barycenter	(Castellon et al., 2023)
Bias-elimination pretraining	Client-side DBE and server-side GBE reduce local and global bias in TSFMs	(Chen et al., 6 Aug 2025)

The synthetic-data line is exemplified by the Federated Data Model. Each site trains a denoising diffusion probabilistic model on its real data, transfers the trained data model rather than the data, and lets the receiving site generate synthetic examples representative of the remote distribution. In the reported medical image segmentation workflow, the receiving site augments local real data with synthetic remote samples and trains a downstream UNet without any exchange of raw patient data (Chen et al., 2024).

FedFA moves the intervention from input space to latent space. For a mini-batch latent feature map $$\bm{X}_m^k \in \mathbb{R}^{B \times C \times H \times W}$$, it computes channel-wise statistics $\mu_m^k$ and $\sigma_m^k$, samples perturbed statistics from Gaussian distributions with fused local and federation-wide variances, and synthesizes an augmented feature map by

$$\hat{\bm{X}}_m^k = \hat{\sigma}_m^k \frac{\bm{X}_m^k - \mu_m^k}{\sigma_m^k} + \hat{\mu}_m^k.$$

The paper further states that training with FedFA yields an expected loss decomposition

$$\mathcal{L}_m^{FedFA} = \mathcal{L}_m^{ERM} + \mathcal{L}_m^{REG},$$

so the method functions as a federation-aware regularizer on representation gradients (Zhou et al., 2023).

FedDaDiL makes datasets themselves the primitive object. It learns atoms $\mathcal{P} = \{\hat{P}_k\}_{k=1}^K$ and private barycentric coordinates $\mathcal{A} = \{\alpha_\ell\}_{\ell=1}^N$ by solving

$$(\mathcal{P}^\star, \mathcal{A}^\star) = \underset{\mathcal{P},\,\mathcal{A}}{\arg\min} \frac{1}{N}\sum_{\ell=1}^N f_\ell(\alpha_\ell, \mathcal{P}),$$

where each client loss is defined through a Wasserstein distance between a local empirical distribution and a barycenter of the shared atoms (Castellon et al., 2023). In the time-series FeDaL framework, the same dataset-centric logic appears as explicit bias disentanglement: client-side Domain Bias Elimination estimates a local bias vector $$\hat{\mathbf{b}}_i^p = \mathbf{b}_i^{\text{trend}} + \mathbf{b}_i^{\text{season}}$$, while server-side Global Bias Elimination corrects aggregation drift and tunes the global model using privacy-preserving core-sets (Chen et al., 6 Aug 2025).

3. Similarity, pre-screening, and federation design

A central FeDaL question is whether two sites should collaborate before training begins. The cross-silo similarity work proposes a bounded, privacy-preserving, dataset-agnostic metric based on optimal transport. Given datasets $\mathcal{D}_1$ and $\mu_m^k$0, the total cost is

$\mu_m^k$1

where $\mu_m^k$2 is cosine dissimilarity and $\mu_m^k$3 is the squared Hellinger distance between label distributions. The normalized optimal-transport expectation then yields a similarity score in $\mu_m^k$4, with lower values indicating high similarity and higher values indicating greater dissimilarity (Elhussein et al., 2024).

The same paper connects this score to training dynamics through the relative weight divergence

$\mu_m^k$5

arguing that alignment of feature vectors and label similarity reduces divergence between local and global updates. The metric is designed to be calculated without exchanging raw data: feature costs are computed securely via Multi-Party Computation, and label costs rely on differentially private class statistics under zero-concentrated differential privacy (Elhussein et al., 2024).

Its operational role is explicitly FeDaL-oriented. The score is proposed for pre-screening collaborations, clustering clients into similar subgroups, guiding the choice between standard and personalized FL, and informing preprocessing and normalization. The reported empirical thresholds are unusually concrete: scores $\mu_m^k$6 correspond to robust FL gains, whereas scores $\mu_m^k$7 often mean FL underperforms local training (Elhussein et al., 2024). This makes dataset comparison a first-class design stage rather than a post hoc diagnostic.

4. Harmonization, curation, and scalable dataset infrastructure

FeDaL also encompasses the engineering of federated datasets. FLINT proposes an end-to-end Federated Learning and Integration architecture built around three component classes: a Federation Controller, Learners at each data silo, and a trusted Driver. Its distinguishing claim is that real federated learning often operates over heterogeneous schemas, formats, values, and access patterns, so harmonization must precede optimization. To this end, the architecture relies on declarative schema mappings of the form

$\mu_m^k$8

along with value normalization, entity linkage, query rewriting, and imputation functions that preserve rather than discard labeled nulls (Stripelis et al., 2023).

A more domain-specific realization appears in the use of DICOM Structured Reports for multi-modal federated dataset creation. That system uses DICOM SRs and highdicom to build a unified representation spanning computed tomography images, electrocardiography scans, annotations such as calcification segmentations and pointsets, and metadata such as prosthesis and diagnoses. It was deployed in an established consortium of eight university hospitals in Germany to create harmonized cohorts for predicting the outcome after minimally invasive heart valve replacement. Reported federated filtering outputs include 6592 CTs, 982 cases with both CT and ECG, and 5204 cases with post-TAVI pacemaker labels (Tölle et al., 2024).

At foundation-model scale, Dataset Grouper addresses the creation of group-structured datasets from existing corpora through user-defined partition functions executed in data-parallel pipelines. It supports a streaming format for cases where even a single group’s dataset does not fit in memory, and it is framework-agnostic across TensorFlow, PyTorch, JAX, and TensorFlow Federated. The paper reports federated versions of large language-modeling datasets such as FedC4, based on C4 with 132B words and over 15.6M domains/groups, and FedWiki with 6.5M groups, enabling federated training of decoder-only transformers with 100M and then 1B parameters (Charles et al., 2023). A plausible implication is that FeDaL is not separable from dataset systems work: the feasible scale and partition structure of the data pipeline directly shape the effective learning regime.

5. Representative empirical results

The Federated Data Model provides a compact illustration of dataset-centric transfer under privacy constraints. In cardiac MRI segmentation of left ventricle myocardium in T1 mapping CMR images from two hospitals with different acquisition protocols, training on Hospital A alone yields strong local performance but pronounced domain-shift degradation on Hospital B. Augmenting Hospital A with synthetic samples generated from Hospital B’s DDPM restores much of the lost performance without degrading local accuracy (Chen et al., 2024).

Train data	Test data	DICE
Hospital A	Hospital A	0.889
Hospital A	Hospital B	0.696
Hospital A + syn.B	Hospital B	0.820
Hospital A + syn.B	Hospital A	0.892

The annotation-efficiency branch of FeDaL is represented by FedAL. In dermoscopic skin-lesion classification over four clients derived from HAM10K and MSK, FedAL alternates active learning and federated learning, using an ensemble of each hospital’s local model and the current global model to score unlabeled samples by ensemble entropy. Using only 50% of samples, the framework reports Micro-F1 $\mu_m^k$9, Macro-F1 $\sigma_m^k$0, and AUC $\sigma_m^k$1, compared with full-data FedAvg at Micro-F1 $\sigma_m^k$2, Macro-F1 $\sigma_m^k$3, and AUC $\sigma_m^k$4 (Deng et al., 2024).

In federated multi-source domain adaptation, FedDaDiL reports classification accuracies close to its centralized counterpart and above standard federated baselines. On Caltech-Office 10, FedDaDiL-R reaches 94.42% and FedDaDiL-E 94.16%, against FedAVG at 92.15%; on TEP, 84.11% and 84.17% against 72.52%; on CWRU, 97.96% and 97.95% against 65.97% (Castellon et al., 2023). In time-series foundation models, FeDaL is evaluated on real-world datasets spanning eight tasks against 54 baselines and reports that performance improves with more data, more clients, and higher participation rates under decentralization, while ablations show that removing DBE or GBE significantly degrades generalization (Chen et al., 6 Aug 2025).

These results span segmentation, classification, domain adaptation, annotation strategy, and foundation-model pretraining. The shared empirical pattern is that explicit modeling of dataset heterogeneity yields gains that are not reducible to standard parameter averaging alone.

6. Topology, privacy, and limitations

FeDaL does not require a single communication topology. FedDC addresses the failure mode of standard FL on extremely small local datasets by intertwining aggregation rounds with daisy-chaining rounds that randomly permute models across clients. On the SUSY dataset with 441 clients and 2 samples per client, it achieves the same accuracy as a centrally trained model, with ACC $\sigma_m^k$5, while standard federated learning baselines remain around ACC $\sigma_m^k$6–$\sigma_m^k$7 (Kamp et al., 2021). De-FedDaDiL removes the server from FedDaDiL entirely: each client selects a random peer, exchanges atom dictionaries, averages them locally, and updates atoms and barycentric coordinates without any central coordinator. The reported communication cost is $\sigma_m^k$8 exchanges per iteration rather than $\sigma_m^k$9 for FedDaDiL, and the average performance difference between the decentralized and federated variants is stated to be within 1–2% (Clain et al., 22 Mar 2025).

Decentralized FL experiments on MNIST reinforce the same theme from another angle. Consensus-based training is described as generally robust, albeit non-optimal, under altered aggregation frequency, optimizer heterogeneity, and partial model sharing, but failures occur when the variance between model weights is too large. The paper identifies excessive inter-client model-weight variance at aggregation time as the main reason for training failure, especially under non-IID partitions and infrequent synchronization (Zhang et al., 2021). This is consistent with the similarity-metric literature, which explicitly links dataset mismatch to weight divergence (Elhussein et al., 2024).

Privacy is treated as a graded property rather than an automatic consequence of decentralization. FDM eliminates raw data sharing, yet explicitly notes potential memorization, a fidelity-privacy tradeoff, and an attack surface in the analysis of shared generative models (Chen et al., 2024). FLINT supports secure aggregation and privacy-preserving local training through protocols such as Secure Multi-Party Computation, DP-SGD, and Fully Homomorphic Encryption (Stripelis et al., 2023). The cross-silo similarity metric combines MPC with zCDP, and DICOM-SR-based cohort construction keeps data on-premise while exposing harmonized, queryable views (Elhussein et al., 2024, Tölle et al., 2024).

The literature therefore does not support an identification of FeDaL with FedAvg on non-IID data. Across the surveyed work, the defining move is to make the dataset itself an object of learning, comparison, synthesis, decomposition, or harmonization. This suggests that FeDaL is a unifying perspective on federated learning in which privacy-preserving collaboration is organized around the structure of decentralized data rather than around model aggregation alone.