Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning (2002.04156v3)

Abstract: Federated learning is a distributed framework for training machine learning models over the data residing at mobile devices, while protecting the privacy of individual users. A major bottleneck in scaling federated learning to a large number of users is the overhead of secure model aggregation across many users. In particular, the overhead of the state-of-the-art protocols for secure model aggregation grows quadratically with the number of users. In this paper, we propose the first secure aggregation framework, named Turbo-Aggregate, that in a network with $N$ users achieves a secure aggregation overhead of $O(N\log{N})$, as opposed to $O(N^2)$, while tolerating up to a user dropout rate of $50\%$. Turbo-Aggregate employs a multi-group circular strategy for efficient model aggregation, and leverages additive secret sharing and novel coding techniques for injecting aggregation redundancy in order to handle user dropouts while guaranteeing user privacy. We experimentally demonstrate that Turbo-Aggregate achieves a total running time that grows almost linear in the number of users, and provides up to $40\times$ speedup over the state-of-the-art protocols with up to $N=200$ users. Our experiments also demonstrate the impact of model size and bandwidth on the performance of Turbo-Aggregate.

• The paper presents Turbo-Aggregate, a secure aggregation framework that reduces computational overhead from O(N^2) to nearly O(N log N) for scalable federated learning.
• It employs multi-group circular aggregation, additive secret sharing, and Lagrange coding to ensure robustness and privacy despite a 50% dropout rate.
• Experimental results show up to 40x speedup in distributed cloud settings, making it a practical solution for large-scale, privacy-sensitive applications.

An Overview of Turbo-Aggregate: A Secure Aggregation Framework in Federated Learning

The paper Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning introduces a novel secure aggregation framework aimed at enhancing the scalability and efficiency of model aggregation in federated learning environments. This framework addresses some of the fundamental challenges in federated learning, particularly the computational overhead and privacy concerns that arise when aggregating model updates from a large number of distributed users.

Key Contributions and Methodology

The significant advancement presented in this paper lies in its ability to substantially reduce the aggregation overhead from a quadratic growth rate with respect to the number of users, $O(N^2)$, to a nearly linear rate, $O(N\log{N})$. This improvement is accomplished while maintaining strong robustness and privacy guarantees, namely, tolerating up to a 50% dropout rate of users and protecting against collusions among up to half of the users.

Turbo-Aggregate achieves these improvements through a combination of a multi-group circular aggregation strategy, additive secret sharing, and redundancy injection via novel coding techniques.

1. Multi-Group Circular Aggregation: The protocol divides users into groups and aggregates models through a circular strategy, allowing users in one group to receive and build upon aggregated models from preceding groups. This architecture is crucial in achieving reduced overhead as it parallelizes the computation and communication processes.
2. Additive Secret Sharing: Each user's model is masked using a combination of random seeds and additive secret sharing, ensuring robust privacy even in scenarios where users or the central server attempt to collude.
3. Redundancy via Lagrange Coding: Turbo-Aggregate employs Lagrange coding to handle user dropouts by injecting redundancy, thus enabling the recovery of aggregated information despite up to 50% user dropouts without breaching privacy.

Experimental Results and Performance

The experimental evaluation of Turbo-Aggregate, implemented in a distributed cloud environment with up to 200 users, underscores its efficiency, demonstrating a consistent, sublinear growth in execution time as the number of users scales. Particularly noteworthy is the protocol's ability to provide a speedup of up to 40 times over the traditional approaches for secure aggregation with a dropout rate as high as 50%.

Implications for Federated Learning

The introduction of Turbo-Aggregate has several critical implications for the field of federated learning:

• Scalability: By reducing the computational cost associated with aggregation, Turbo-Aggregate facilitates the deployment of federated learning on a much larger scale, accommodating potentially millions of users as opposed to just hundreds.
• Robustness: Its dropout resilience ensures that real-world federated systems, characterized by sporadic and unreliable user connections, can still perform optimally.
• Privacy: Enhanced privacy guarantees make it feasible to adopt federated learning frameworks in domains sensitive to data security and privacy.

Future Directions

The paper suggests that future developments can extend Turbo-Aggregate to accommodate more robust Byzantine fault tolerance and incorporate it with model compression techniques to further reduce communication overhead. These steps could further establish federated learning as a cornerstone technology in distributed AI systems.

In conclusion, Turbo-Aggregate represents a significant stride toward addressing key bottlenecks in federated learning by balancing efficiency, robustness, and privacy, thereby paving the way for more widespread and scalable decentralized AI applications.