Tram-FL: Decentralized Federated Learning

• Tram-FL is a decentralized federated learning framework that circulates a single global model through optimally selected nodes to mitigate non-iid data issues and reduce transmission overhead.
• It employs a dynamic routing algorithm to minimize label variance, achieving a significant reduction in communication cost compared to traditional pairwise exchanges.
• The Load-aware Tram-FL extension adapts scheduling to heterogeneous resources, demonstrating 50–90% faster training while maintaining statistical balance in empirical evaluations.

Tram-FL is a decentralized federated learning (DFL) framework designed to address the dual challenges of elevated communication cost and statistical inefficiency in the presence of non-iid data distributions across cross-silo data silos. Tram-FL departs from classical decentralized FL algorithms—such as Gossip-SGD and PDMM-SGD—that synchronize local models at each node via communication-intensive pairwise exchanges, and instead introduces a single global model that traverses the network sequentially, being updated at each node along a dynamically optimized route to mitigate label imbalance and minimize transmission overhead. Tram-FL and its extension Load-aware Tram-FL establish a new paradigm in decentralized learning by coupling model circulation and adaptive routing or scheduling with theoretical and empirical demonstrable improvements in communication and convergence (Maejima et al., 2023, Kainuma et al., 11 Jun 2025).

1. Decentralized Federated Learning Context and Tram-FL Motivation

In the decentralized FL setting, the network consists of $V$ fully connected, peer-to-peer nodes—such as enterprise servers, hospitals, or banks—each holding a private, potentially highly non-iid dataset $X_i$. Standard DFL algorithms synchronize and update local models via repeated random gossip or consensus, broadcasting parameters or gradients to all neighbors at each SGD step. These approaches are hampered by two major limitations:

• Statistical Instability: Under severe non-iid data, local stochastic gradients are misaligned, yielding poor or unstable convergence.
• High Communication Cost: Each SGD step entails $\mathcal{O}(V)$ inter-node transmissions, leading to prohibitive bandwidth demands for large $V$ or high model dimensionalities.

Tram-FL circumvents these problems by (1) maintaining only a single global model in network circulation and (2) optimally routing its sequential propagation among nodes, thereby shaping the global data exposure to approximate iid coverage and reducing communication to a minimum (Maejima et al., 2023).

2. Tram-FL Algorithmic Framework

The core Tram-FL protocol operates as follows:

• At initialization, an initial model $w^{(0)}$ is placed at an arbitrary node $i_0$.
• At round $k$, node $i_{k-1}$ performs $T$ local SGD steps over minibatches of size $B$ drawn from $X_{i_{k-1}}$, using the update:

$w \leftarrow w - \eta \nabla F_{i_{k-1}}(w)$

where $F_{i_{k-1}}(w)$ is the empirical loss on local minibatch.

• After $T$ updates, the current model is transferred to the next node $i_k$, as determined by a routing algorithm that seeks to minimize cumulative label imbalance.
• Only one model is in flight at any time; model transmission is infrequent (every $T$ steps) compared to the broadcast-intensive gossip/consensus approaches.

This cycle continues for $K$ rounds, serially exposing the global model to the sequence of node local data (Maejima et al., 2023).

3. Dynamic Model Routing and Statistical Bias Mitigation

Model routing is central to Tram-FL’s efficacy under non-iid data. The routing algorithm seeks to equalize the cumulative exposure of the global model to all label classes. For a set of labels $C$ (classes), each node $i$ tracks its per-class sample counts $L_i = [l_{i, c} \mid c \in C]$ and total $N_i = \sum_c l_{i, c}$. Let $L^k$ be the cumulative count of label $c$ samples seen by the model up to round $k$.

At each routing decision, the goal is to select the next node $i^*$ that, after $T$ local steps, will minimize the label variance:

$i^* = \arg\min_{j \in V} \sigma(L^k + \Delta_j)$

where $\Delta_{j, c} = \frac{B T}{N_j} l_{j, c}$ is the expected label increment vector if node $j$ is next, and $\sigma(\cdot)$ computes the empirical variance across label counts. This strategy dynamically schedules visits to nodes with underrepresented classes, thus bias-minimizing over the sequential trajectory and, over multiple rounds, approximates the statistical effect of iid data exposure (Maejima et al., 2023).

4. Communication Efficiency

Let $m$ be the model size (bytes). In decentralization, Gossip-SGD typically incurs

$C_{\text{gossip}} \approx m (V-1) K$

total transmissions for $K$ SGD steps, as all nodes broadcast at each step. By comparison, Tram-FL, with transfers only every $T$ steps, achieves

$C_{\text{Tram}} = m \frac{K}{T}$

yielding a relative reduction of approximately $1/((V-1) T)$. For moderate $V$ and $T \geq 1$, this is an order-of-magnitude decrease in communication footprint, validated across various datasets and partitioning strategies (Maejima et al., 2023).

5. Experimental Validation and Performance

Tram-FL has been empirically benchmarked on standard image and text classification datasets (MNIST, CIFAR-10, IMDb) under severe non-iid conditions:

• Data Partitioning: For MNIST/CIFAR-10, class labels are split disjointly among nodes. For IMDb, a skewed exponential class split is used.
• Model Architectures: Convolutional neural networks for vision and LSTM-based models for text (see source for architectural details).
• Metrics: Test accuracy vs. total inter-node transmissions, convergence stability, and communication cost.

Key findings:

• Tram-FL consistently achieves stable, high-accuracy convergence where gossip- and consensus-based DFL methods exhibit oscillation or failure under extreme heterogeneity.
• On CIFAR-10 with $V=5$, dynamic routing reduces the number of transmissions needed to reach near-centralized accuracy by $>15\%$ compared to static or random routes.
• On IMDb, only Tram-FL converges to $>80\%$ accuracy within the available communication budget; baselines stagnate.
• Communication cost can be reduced by more than an order of magnitude while maintaining or surpassing baseline convergence (Maejima et al., 2023).

6. Load-Aware Tram-FL: Scheduling under Heterogeneous Resources

Load-aware Tram-FL generalizes Tram-FL to environments with heterogeneous, time-varying node computation power and inter-node bandwidth (Kainuma et al., 11 Jun 2025). The scheduling problem is formulated as a global optimization:

• Variables: At each round, select the node and the per-class sample fraction to maximize throughput, constrained by a class variance bound and available resources.
• Objective: Maximize

$$O_1 = \frac{\sum_{k=1}^K L_{i^k} \cdot X_{i^k}^k}{1+\sum_{k=1}^K T^k}$$

(samples processed per unit training time), subject to a variance constraint to enforce balanced label sampling.

• Solution: The global problem is decomposed round-wise into $|\mathcal{I}|$ quadratic-constrained programs, each selecting the optimal node and local update allocation.
• Complexity: Each round solves up to $|\mathcal{I}|$ convex QPs of dimension $|\mathcal{C}|$ (number of classes), tractable for moderate class counts.

Empirical results on MNIST and CIFAR-10 demonstrate that Load-aware Tram-FL reduces wall-clock training times by 50–90% relative to random- or time-first selection, while maintaining class balance and robust convergence. The performance impact is pronounced in scenarios with imbalanced node resources and unbalanced label splits (Kainuma et al., 11 Jun 2025).

7. Limitations, Open Problems, and Practical Considerations

• Serialization Bottleneck: The strict sequential traversal of the global model can increase wall-clock training time compared to parallel update schemes.
• Heuristic Routing: The dynamic routing problem is combinatorial; the implemented strategy is heuristic and optimality under general conditions is not guaranteed. The question of optimal routing under dynamic data and network shifts is open.
• Scalability: For large class counts, tractable optimization might necessitate class clustering or relaxed constraints.
• Fault Tolerance and Flexible Topologies: Extensions to tolerate node failures or asynchronous updates, and operation over arbitrary (possibly partial) topologies, remain unaddressed.
• Resource Forecasting: Load-aware Tram-FL depends on nodes’ ability to predict next-round resource availability; mispredictions can affect efficiency.
• Deployment: Integration into existing DFL frameworks is facilitated by modular replacement of the node selection logic.

A plausible implication is that, as practical federated deployments become increasingly cross-silo and resource-heterogeneous, circulation-based approaches like Tram-FL may become standard for robust, communication-efficient, statistically unbiased training when iid data cannot be assumed (Maejima et al., 2023, Kainuma et al., 11 Jun 2025).