FedOT Algorithm: Federated Learning Innovations

• FedOT is a collection of distinct federated learning approaches that use optimal transport and client-specific orthogonal transforms to adapt pre-trained models efficiently.
• It integrates multi-marginal optimal transport techniques, including distributed Sinkhorn solvers, to personalize models and mitigate heterogeneity across clients.
• The evolutionary AutoML variant of FEDOT optimizes machine learning pipelines using DAG-based search and parallel resource allocation for improved accuracy and efficiency.

FedOT Algorithm

FedOT denotes a collection of recent and technically distinct algorithms introduced under the acronym "Federated OT," spanning multiple fields within federated learning, optimal transport, AutoML, and representation learning. The specific approaches and implementations denoted as "FedOT" include: (1) a vision-centric federated learning algorithm utilizing client-specific orthogonal transformations with frozen foundation model encoders (Kong et al., 26 May 2025); (2) optimal transport-based personalized federated learning (Farnia et al., 2022); (3) an entropy-regularized distributed Sinkhorn optimal transport solver for federated data (Kulcsar et al., 10 Feb 2025); and (4) evolutionary AutoML for pipeline design in heterogeneous environments (Nikitin et al., 2023, Polonskaia et al., 2021)—the last of which is sometimes also referred to as FEDOT (capitalization variant) in the literature. Each of these has a distinct mathematical and technical core but shares federated optimization or federated computation as a central theme.

1. Vision-Centric FedOT: Orthogonal Transformations with Foundation Models (Kong et al., 26 May 2025)

FedOT, as introduced in "Generalized and Personalized Federated Learning with Foundation Models via Orthogonal Transformations," is a federated learning protocol constructed around black-box foundation models (e.g., frozen CLIP encoders). The model on each client consists of:

• Frozen Encoder: Each client possesses a foundation model feature extractor $\mathcal{I}(x) \in \mathbb{R}^d$; the encoder weights are not updated or communicated.
• Global Classifier: A matrix $W \in \mathbb{R}^{K \times d}$, shared and synchronized across all clients, parameterizes the task-specific classifier.
• Local Orthogonal Transform: Each client $i$ holds a private, client-specific orthogonal matrix $Q_i \in \mathbb{R}^{d \times d}$ with $Q_i^\top Q_i = I$, used to adapt features locally. During inference, features are transformed as $h_i' = Q_i h$, $\tilde{h}_i = h_i'/\|h_i'\|$, followed by logit computation $W\tilde{h}_i$.

The optimization objective, over $N$ clients, is: $$\min_{W, \{Q_i\}_{i=1}^N} \frac{1}{N}\sum_{i=1}^N \ell_i(W, Q_i)\quad \text{s.t.}\ Q_i^\top Q_i = I\ \forall i,$$ where $W \in \mathbb{R}^{K \times d}$0 is the expected cross-entropy loss on client $W \in \mathbb{R}^{K \times d}$1. Only $W \in \mathbb{R}^{K \times d}$2 is transferred between server and clients, ensuring data and $W \in \mathbb{R}^{K \times d}$3 locality for privacy.

The client optimizes $W \in \mathbb{R}^{K \times d}$4 via a Cayley transform parameterization, maintaining orthogonality, while optimizing $W \in \mathbb{R}^{K \times d}$5 by SGD. Orthogonality constrains the feature geometry, thereby minimizing client-to-client gradient conflicts to the theoretical minimum (condition number $W \in \mathbb{R}^{K \times d}$6), which ensures stable and semantically coherent feature adaptation across highly heterogeneous clients.

2. FedOT Approaches Based on Multi-Marginal Optimal Transport (Farnia et al., 2022, Kulcsar et al., 10 Feb 2025)

FedOT also refers to personalized federated learning based on multi-marginal optimal transport (Farnia et al., 2022). Each client $W \in \mathbb{R}^{K \times d}$7 learns a transport map $W \in \mathbb{R}^{K \times d}$8 that pushes its empirical data distribution $W \in \mathbb{R}^{K \times d}$9 onto a shared reference (barycenter) distribution $i$0 in a common feature space. Concurrently, a reference-domain model $i$1 is optimized. The FedOT objective combines empirical risk—evaluating predictions after transport—and a regularization penalizing the alignment cost (via Wasserstein distance or other OT divergences): $i$2 Via Kantorovich duality, this yields a tractable min-max structure over model parameters, transport map parameters, and dual potentials.

In a closely related but technically distinct usage, "FedOT" labels distributed algorithms for entropy-regularized optimal transport—specifically, Federated Sinkhorn—wherein the source and target histograms and cost matrices are partitioned across federated clients (Kulcsar et al., 10 Feb 2025). This enables distributed computation of the Sinkhorn-Knopp scaling variables ($i$3, $i$4), either synchronously (all-to-all or star topologies) or asynchronously with dampened updates, maintaining privacy over data blocks and yielding strong convergence guarantees.

3. Evolutionary AutoML: FEDOT Pipeline Optimization (Nikitin et al., 2023, Polonskaia et al., 2021)

FEDOT (often capitalized for clarity), refers to an evolutionary AutoML framework for discovering heterogeneous machine learning pipelines as directed acyclic graphs (DAGs) (Nikitin et al., 2023, Polonskaia et al., 2021). Each pipeline is encoded as a DAG of operation nodes (data transforms, learners) with hyperparameters, supporting multi-branch architectures and ensemble strategies.

Multi-objective optimization is employed, typically minimizing tuples $i$5 corresponding to prediction error, structural complexity, and computational cost, and maintaining a Pareto front of non-dominated solutions. Evolutionary search operates with parameter-free adaptation strategies including variable population size along the Fibonacci sequence, operator rates controlled by population diversity, and adaptive search space expansion.

Resource allocation and pipeline evaluation are massively parallelized and support heterogeneous computing (local/remote via Kubernetes, CPU/GPU selection, cache-aware scheduling), with per-node SQLite caching yielding up to 14% reduction in overall evaluation time and near-linear parallelization speedup.

4. Algorithmic Steps and Protocols

Orthogonal Foundation Model FedOT (Kong et al., 26 May 2025)

• Server: Initialize $i$6; in each round, broadcast $i$7, aggregate updated $i$8 via $i$9.
• Client: Maintain $Q_i \in \mathbb{R}^{d \times d}$0 (unconstrained), form $Q_i \in \mathbb{R}^{d \times d}$1 via Cayley transform from the skew-symmetric part $Q_i \in \mathbb{R}^{d \times d}$2, repeatedly optimize $Q_i \in \mathbb{R}^{d \times d}$3 and $Q_i \in \mathbb{R}^{d \times d}$4 using local data, send only $Q_i \in \mathbb{R}^{d \times d}$5 to the server.

OT-Based Personalized FL (Farnia et al., 2022)

• Personalization: Each client solves for its own $Q_i \in \mathbb{R}^{d \times d}$6, while the shared model parameters and barycenter updates are synchronized via federated averaging.
• Optimization: Distributed Gradient Descent–Ascent (GDA), with individual clients updating both transport and prediction parameters, then averaging with the global center.

Federated Sinkhorn (Kulcsar et al., 10 Feb 2025)

• Synchronous All-to-All: Alternating row and column scaling steps, with $Q_i \in \mathbb{R}^{d \times d}$7 communication of scaling variables after each update.
• Asynchronous: Non-blocking update exchanges, staleness controlled by step-size damping $Q_i \in \mathbb{R}^{d \times d}$8.
• Star Topology: Server centrally aggregates and distributes scaling vectors for global kernel computations.

FEDOT AutoML (Nikitin et al., 2023, Polonskaia et al., 2021)

• Pipeline Generation: Random DAG initialization, evolutionary population updates via mutation/crossover, fitness-based parent selection (NSGA-II, SPEA2).
• Execution: Pipelines are dispatched to heterogeneous resources (CPU, GPU, remote clusters), evaluated in parallel, and cached for efficiency.

5. Theoretical Insights and Guarantees

In orthogonally transformed FL (Kong et al., 26 May 2025), Theorem 4.1 establishes that with client orthogonal transforms (condition number $Q_i \in \mathbb{R}^{d \times d}$9), the client-to-client gradient difference is bounded by $Q_i^\top Q_i = I$0 (minimal achievable), which stabilizes aggregation across heterogeneous domains.

In OT-based federated personalization (Farnia et al., 2022), generalization error is controlled via Dudley entropy of function classes, with sample complexity outperforming local models when the product of clients and local samples is large. Optimization guarantees show convergence to $Q_i^\top Q_i = I$1-stationary points in $Q_i^\top Q_i = I$2 steps.

In federated Sinkhorn (Kulcsar et al., 10 Feb 2025), synchronous protocols exactly recover centralized Sinkhorn updates with linear convergence; asynchronous variants achieve similar rates if step-size and network delays are properly controlled.

6. Empirical Benchmarks and Comparative Performance

Experimental evaluations consistently show the following:

• Orthogonal FedOT outperforms global, local, and prompt-based baselines on domain-shifted benchmarks (e.g., FEMNIST, PACS, TerraIncognita), improving average and personalized accuracy metrics (e.g., FEMNIST: FedAvg 92.22% vs. FedOT 95.31%; PACS: FedAvg 96.16% vs. FedOT 96.22%) (Kong et al., 26 May 2025).
• Multi-marginal FedOT improves test accuracy by 10–20 percentage points over traditional FL on MNIST/CIFAR-10 under synthetic heterogeneous shifts (Farnia et al., 2022).
• In federated Sinkhorn, communication overhead dominates on GPU clusters, whereas synchronous partitioning yields scalable speedups on CPU-bound workloads; convergence is robust except in highly asynchronous, undamped regimes (Kulcsar et al., 10 Feb 2025).
• FEDOT's evolutionary AutoML achieves up to 14% more pipelines via caching, near-linear CPU parallel acceleration, >90% GPU speedup for large tasks, and consistently higher quality/DAG-diverse models relative to GP or TPOT baselines (Nikitin et al., 2023, Polonskaia et al., 2021).

7. Distinctions, Naming, and Applicability

The shared acronym "FedOT" refers to substantially different methodologies depending on context:

Domain	Mathematical Core	arXiv Reference
Vision FL with Found. Models	Orthogonal feature transforms	(Kong et al., 26 May 2025)
Personalized FL (Theory)	Multi-marginal OT, barycenters	(Farnia et al., 2022)
OT Solver (distributed)	Sinkhorn over federated splits	(Kulcsar et al., 10 Feb 2025)
AutoML PIPELINES	Evolutionary DAG optimization	(Nikitin et al., 2023, Polonskaia et al., 2021)

Setting-specific context is essential: in FL with vision foundation models, "FedOT" denotes orthogonal WRAPPER algorithms; in nonconvex federated personalized learning, the acronym identifies OT-regularized risk minimization; in distributed OT/Sinkhorn, the focus is privacy-preserving scaling; in AutoML, FEDOT stands for "Flexible Evolutionary Design Of Trees/Pipelines."

A plausible implication is that future work may seek to unify or further clarify the nomenclature and technical boundaries of "FedOT" algorithms, or to introduce additional variants leveraging recent advances in federated optimization and OT theory.