TurboSVM-FL: Federated SVM Aggregation

• The paper introduces TurboSVM-FL, a federated aggregation method that leverages SVM-based selective support vector collection to enhance convergence and class separability.
• It employs max-margin spread-out regularization on global class embeddings to ensure distinct class representations and reduce communication rounds.
• Empirical evaluations show significant reductions in training rounds compared to FedAvg, making it efficient for lazy clients in cross-device federated learning.

TurboSVM-FL is a federated model aggregation strategy for distributed classification tasks, specifically designed to improve convergence and final test accuracy in scenarios where clients undertake minimal local computation (so-called "lazy" clients). Unlike standard averaging-based aggregation, TurboSVM-FL employs support vector machine (SVM) mechanisms at the server to selectively aggregate class embedding vectors, followed by a max-margin spread-out regularization of the global class representations. This dual approach enhances class separability and leverages the most informative client updates, accelerating training and reducing communication rounds without increasing client-side computational or communication costs (Wang et al., 2024).

1. Federated Learning and Its Challenges

Federated learning (FL) enables collaborative training of machine learning models across a network of clients without sharing local data, relying on periodic model parameter exchanges coordinated by a central server. The server broadcasts the current global model, clients perform local updates (typically several epochs of stochastic gradient descent), and resulting models are aggregated to form a refined global parameterization. Data heterogeneity—specifically the non-i.i.d. distribution of client datasets—can markedly slow convergence, with cross-device FL being particularly affected due to constraints on client resources and the impracticality of increasing on-device computation through auxiliary loss terms or longer local training cycles (Wang et al., 2024).

2. Core Aggregation Mechanism in TurboSVM-FL

TurboSVM-FL replaces the vanilla weighted averaging of class embedding matrices (the $W \in \mathbb{R}^{K \times d}$ layer) with an SVM-centric selective aggregation process, designed to identify and prioritize margin-critical information. For each binary class pair $(k, k')$, a one-vs-one (OVO) soft-margin SVM is trained on class embeddings gathered from all participating clients:

• The SVM primal optimization problem seeks a hyperplane normal $h \in \mathbb{R}^d$ for discrimination between class embeddings, minimizing $½‖h‖^2 + \lambda\sum \xi_i$ under the constraint $y_i h^\top x_i \geq 1-\xi_i, \xi_i \geq 0$, where $x_i$ are the client-provided embedding vectors and $\lambda$ is the regularization coefficient.
• For each class $k$, support vectors (i.e., local embeddings $w_k^n$ with non-zero dual coefficients from the SVMs involving $k$) are collected. The selectively aggregated global embedding for class $(k, k')$0 is then

$(k, k')$1

where $(k, k')$2 are the support vectors and $(k, k')$3 is the size of the client dataset they originate from.

• This selective averaging ensures the global class embedding incorporates only margin-critical local representations, supplanting the typical holistic average used in FedAvg (Wang et al., 2024).

3. Max-Margin Spread-Out Regularization

After selective SVM aggregation, TurboSVM-FL imposes an additional regularization to promote further separation of class embeddings along the hyperplanes identified in the SVM step:

• The $(k, k')$4 matrix of global class embeddings $(k, k')$5 is regularized via

$(k, k')$6

where $(k, k')$7 is the normal vector for the $(k, k')$8 OVO SVM.

• This penalty discourages small margins between class embedding projections, yielding “spread-out” class representations.
• The server subsequently performs an optimizer step (typically using an adaptive method) to reduce $(k, k')$9 with respect to $h \in \mathbb{R}^d$0 for each class $h \in \mathbb{R}^d$1 (Wang et al., 2024).

4. Algorithmic Workflow and Computational Complexity

TurboSVM-FL’s workflow, executed at each global round, is as follows:

• The server broadcasts the current model to a subset of clients.
• Clients perform local stochastic gradient steps for $h \in \mathbb{R}^d$2 epochs (by default $h \in \mathbb{R}^d$3), returning updated model parameters.
• Encoder parameters $h \in \mathbb{R}^d$4 are aggregated by weighted averaging as in standard paradigms.
• The server, for each class pair, fits an OVO SVM using collected class embeddings from the received models.
• For each class, support vectors are gathered across all relevant SVMs, and selective averaging produces updated global embeddings.
• Spread-out regularization is applied with a gradient step on the class embeddings.

In terms of resource usage relative to FedAvg:

• No increase in client communication or computation; local updates remain $h \in \mathbb{R}^d$5 epochs of SGD with unchanged communication payloads.
• Server-side computational cost rises with the training of $h \in \mathbb{R}^d$6 binary SVMs on $h \in \mathbb{R}^d$7 points in $h \in \mathbb{R}^d$8 and $h \in \mathbb{R}^d$9 additional gradient calculations, but remains feasible for moderate $½‖h‖^2 + \lambda\sum \xi_i$0 (e.g., hundreds of SVMs with $½‖h‖^2 + \lambda\sum \xi_i$1, $½‖h‖^2 + \lambda\sum \xi_i$2, typical of cross-device FL workloads) (Wang et al., 2024).

5. Empirical Performance on Benchmark Datasets

TurboSVM-FL was empirically evaluated for classification on non-i.i.d. splits of FEMNIST ($½‖h‖^2 + \lambda\sum \xi_i$3), CelebA ($½‖h‖^2 + \lambda\sum \xi_i$4), and Shakespeare ($½‖h‖^2 + \lambda\sum \xi_i$5), adopting standard user-independent 90/10 train/test splits and $½‖h‖^2 + \lambda\sum \xi_i$6 clients per round with $½‖h‖^2 + \lambda\sum \xi_i$7. Performance comparisons against FedAvg, FedAdam, FedAwS, FedProx, FedAMS, and MOON demonstrate:

Algorithm	FEMNIST (Rounds to 70%)	CelebA (Rounds to 70%)	Shakespeare (Rounds to 50%)
FedAvg	144.4 ± 4.6	91.6 ± 18.2	51.0 ± 5.0
FedProx	145.4 ± 3.4	94.4 ± 18.4	157.2 ± 5.3
FedAwS	81.4 ± 2.2	84.2 ± 24.4	45.0 ± 2.3
TurboSVM-FL	54.6 ± 1.6	46.4 ± 9.4	43.4 ± 2.9

TurboSVM-FL achieves —62.2%, —49.3%, and —14.9% reductions in training rounds to reach the target accuracy for FEMNIST, CelebA, and Shakespeare, respectively.

At $½‖h‖^2 + \lambda\sum \xi_i$8 communication rounds, TurboSVM-FL yields test F1/accuracy/MCC of (61.9/76.6/75.8) on FEMNIST and (77.2/77.3/55.0) on CelebA, outperforming alternatives in both rates and final scores; on Shakespeare, it matches or slightly exceeds baseline algorithms (Wang et al., 2024).

6. Practical Settings and Recommendations

Default hyperparameters and recommendations:

• Local update epochs $½‖h‖^2 + \lambda\sum \xi_i$9: 1 (suitable for “lazy” clients); improvements persist for $y_i h^\top x_i \geq 1-\xi_i, \xi_i \geq 0$0–4.
• Clients per round $y_i h^\top x_i \geq 1-\xi_i, \xi_i \geq 0$1: 8; speed-up is maintained for $y_i h^\top x_i \geq 1-\xi_i, \xi_i \geq 0$2.
• SVM regularization $y_i h^\top x_i \geq 1-\xi_i, \xi_i \geq 0$3: optimal with linearly decreasing schedule (e.g., from 1.0 downward), leveraging improved model reliability in late rounds.
• OVO decomposition for SVMs is preferred over OVR due to class imbalance mitigation.
• Server optimizer learning rate $y_i h^\top x_i \geq 1-\xi_i, \xi_i \geq 0$4 for spread-out updates: grid-searched with optimal values varying per task (e.g., $y_i h^\top x_i \geq 1-\xi_i, \xi_i \geq 0$5 for images, 1 for Shakespeare).
• TurboSVM-FL on $y_i h^\top x_i \geq 1-\xi_i, \xi_i \geq 0$6 can be combined with other techniques for the encoder $y_i h^\top x_i \geq 1-\xi_i, \xi_i \geq 0$7, such as FedProx for additional drift control (Wang et al., 2024).

7. Comparative Analysis and Significance

TurboSVM-FL’s principal innovation lies in its two-stage server-side aggregation: SVM-based support vector selection focusing on margin-critical embeddings, and explicit maximization of class separability through spread-out regularization. These mechanisms target the core challenge of slow convergence endemic to FL under data heterogeneity and minimal client computation, circumventing the need for additional client-side resources. Empirical evidence demonstrates substantial acceleration in convergence and superior classification metrics, especially for large-output tasks with imbalanced or heterogeneous data. This approach is particularly apt for cross-device FL where client constraints are paramount (Wang et al., 2024).