BlazeFL: Fast and Deterministic Federated Learning Simulation

Abstract: Federated learning (FL) research increasingly relies on single-node simulations with hundreds or thousands of virtual clients, making both efficiency and reproducibility essential. Yet parallel client training often introduces nondeterminism through shared random state and scheduling variability, forcing researchers to trade throughput for reproducibility or to implement custom control logic within complex frameworks. We present BlazeFL, a lightweight framework for single-node FL simulation that alleviates this trade-off through free-threaded shared-memory execution and deterministic randomness management. BlazeFL uses thread-based parallelism with in-memory parameter exchange between the server and clients, avoiding serialization and inter-process communication overhead. To support deterministic execution, BlazeFL assigns isolated random number generator (RNG) streams to clients. Under a fixed software/hardware stack, and when stochastic operators consume BlazeFL-managed generators, this design yields bitwise-identical results across repeated high-concurrency runs in both thread-based and process-based modes. In CIFAR-10 image-classification experiments, BlazeFL substantially reduces execution time relative to a widely used open-source baseline, achieving up to 3.1$\times$ speedup on communication-dominated workloads while preserving a lightweight dependency footprint. Our open-source implementation is available at: https://github.com/kitsuyaazuma/blazefl.

• The paper introduces BlazeFL, which employs a free-threaded, shared-memory architecture and per-client RNG streams to achieve bitwise-deterministic simulation in federated learning.
• The evaluation shows up to 3.1× faster throughput on CNN workloads compared to Flower, ensuring consistent performance and reproducibility.
• The approach minimizes process overhead and dependencies, enabling rapid prototyping and reliable reproducibility in high-concurrency FL experiments.

BlazeFL: Enabling Efficient, Bitwise-Deterministic Federated Learning Simulation

Motivation and Problem Setting

Federated learning (FL) frameworks are increasingly relying on large-scale, single-node simulations for rapid prototyping and algorithmic experimentation. Simulation workloads with hundreds or thousands of virtual clients are impeded by communication, parameter serialization, and process boundary overheads in multi-process and distributed runtimes. Reproducibility—specifically, achieving bitwise-determinism across high-concurrency runs—is further compromised due to poor random state isolation and nondeterministic scheduling. Existing frameworks offer an unfavorable trade-off between throughput and reproducibility, necessitating ad-hoc control logic to recover deterministic training over parallel execution.

BlazeFL proposes a streamlined approach targeting this gap: single-process, free-threaded, shared-memory execution with explicit, per-client RNG streams and protocol-based minimal APIs for rapid PyTorch integration.

System Architecture and Determinism Guarantees

BlazeFL's architecture is centered on running all clients as threads within a single process, orchestrated by a main thread. Model state and outputs are communicated exclusively via shared memory, eliminating the overhead of cross-process serialization, OS-level inter-process communication (IPC), and external schedulers.

Figure 1: Architecture of BlazeFL, detailing the thread-based execution, centralized scheduling, and per-client RNG isolation for deterministic simulability.

A critical architectural element is the management of all stochasticity through client-isolated RNG suites. Each virtual client is initialized with a deterministic seed schedule, ensuring that local stochastic operations (e.g., data shuffling, augmentation, regularization) are invariant under the order of thread scheduling. The server consumes batched client updates in the deterministic order of the sampled client list, rather than in arbitrary completion sequence, circumventing floating-point non-associativity as a source of trajectory divergence.

Evaluation: Throughput and Determinism

Wall-Clock Scalability

Performance comparisons span a high-end server (H100, 48 cores) and a workstation (RTX 6000, 32 cores), benchmarking CIFAR-10 image classification with client counts and parallelism settings reflective of contemporary FL research. Multiple network backbones (lightweight CNN, ResNet-18/50/101) are evaluated.

Strong numerical results are observed: For communication-dominated CNN workloads, BlazeFL's free-threaded mode achieves up to 3.1× faster simulation throughput than Flower with Ray backend on the high-performance server. With increasing model complexity or reduced system memory, throughput gains diminish (up to 1.4× for ResNet-18, 1.1× for ResNet-50), attributable to contention within PyTorch’s global CUDA mutex under a threaded paradigm—an effect mitigated (and even reversed) in process-isolated modes due to independent CUDA contexts.

Deterministic Repeatability Analysis

FL simulation under parallelism typically fails to achieve bitwise-identical results, even with global seed initialization, as evidenced by nonzero standard deviation of final accuracy and persistent SHA-256 hash divergence across rounds in frameworks like Flower.

BlazeFL's per-client RNG and fixed result ordering architecture yields identical hashes and zero variance in final accuracy across all repeated runs and degrees of parallelism within a fixed software/hardware stack. This is achieved regardless of the concurrency setting ($P$ from 1 to 64), as long as all stochastic operators are routed through BlazeFL-managed generators and custom pipeline components abstain from global RNG state.

For comparison, Figure 2 highlights the phenomenon in Flower: after a single round of aggregation, logarithmic drift from non-associative float addition is observed, ultimately compounding to measurable behavioral and accuracy discrepancies.

Figure 2: Visualization of non-deterministic error accumulation in Flower due to completion-order-dependent aggregation, in contrast to BlazeFL's reproducibility.

Interface Design and Dependency Management

BlazeFL’s protocol-based interface, relying on Python's typing.Protocol, eschews rigid inheritance hierarchies, enabling integration of arbitrary user-defined training and server logic without intrusive rewrites. This reduces framework lock-in and facilitates seamless benchmarking and prototyping.

The dependency set is intentionally minimal, restricted to the Python standard library and PyTorch, foregoing distributed schedulers and third-party runtimes. This design choice not only supports rapid iteration but also enhances experimental reproducibility by minimizing external sources of nondeterminism and numerical regression.

Limitations and Practical Considerations

Despite its determinism guarantees, BlazeFL restricts claims to within-machine, single-node settings. Hardware/OS/library upgrades or platform shifts may invalidate bitwise identity due to changes in floating-point implementation. Additionally, thorough generator management is required for all stochastic vision transforms and custom data pipelines, as any silent fallbacks to global RNG state can reintroduce nondeterminism at the operator level.

Performance under compute-intense, VRAM-constrained conditions may degrade due to single-process CUDA allocator bottlenecks. In such cases, reverting to process-based parallelism (included in BlazeFL) or leveraging distributed runtimes remains appropriate.

Implications and Future Directions

BlazeFL alters the standard in FL simulation by showing that, with careful system design, high-throughput and bitwise-repeatable simulation are simultaneously attainable under realistic workloads and client parallelism. The approach demonstrates that Python's free-threading can be a foundational primitive for high-concurrency ML simulation, provided dependency sprawl and scheduler complexity are avoided.

On practical grounds, BlazeFL may serve as a reference platform for reproducibility studies, regression testing, and rapid prototyping in FL. Its determinism-by-design orientation is especially relevant for methods that require trajectory-synchronized ablation, robust comparison of subtle algorithmic variations, or probabilistic debugging.

Theoretically, this work demonstrates the necessity of per-worker RNG control and output ordering for any high-fidelity stochastic parallel simulation, not only in FL but for other domains susceptible to floating-point drift. Future work will involve extending these guarantees to multi-node distributed environments, integrating robust vision/data pipeline generator plumbing, and further optimizing thread-based backend performance vis-à-vis CUDA resource allocation and lock contention.

Conclusion

BlazeFL establishes that single-node federated learning simulation can pair high throughput with bitwise-deterministic execution under practical workloads, using a free-threaded, shared-memory architecture with explicit randomness isolation. Within its operational scope, BlazeFL represents a new systems baseline for reproducible FL experimentation and serves as early evidence for the viability of free-threaded Python in AI research infrastructure, with broader implications for stochastic simulation in concurrent scientific computing (2604.03606).