BlazeFL: Deterministic FL Simulator

• BlazeFL is a lightweight framework for single-node federated learning simulation designed for high throughput and deterministic reproducibility in virtual-client experiments.
• It employs free-threaded shared-memory execution to bypass inter-process communication and serialization, optimizing performance in communication-bound scenarios.
• Its deterministic randomness management ensures bitwise-identical outcomes across runs, making it ideal for controlled algorithm development and reproducibility-sensitive benchmarking.

BlazeFL is a lightweight framework for single-node federated learning (FL) simulation designed to address two constraints that often conflict in large-scale virtual-client experiments: high throughput and strict reproducibility. It targets settings in which hundreds or thousands of virtual clients are simulated on one machine, where inter-process communication, parameter serialization, and scheduling variability can dominate runtime and introduce nondeterminism. BlazeFL’s central design combines free-threaded shared-memory execution with deterministic randomness management, with the stated objective of eliminating the trade-off between speed and repeatability under a fixed software/hardware stack (Azuma et al., 4 Apr 2026).

1. Motivation and problem formulation

BlazeFL emerged from two converging needs in FL research. The first is the need to push hundreds or even thousands of virtual clients through a single-node simulator without paying the heavy toll of inter-process communication and parameter serialization. The second is the need to guarantee bitwise-identical repeats of the same experiment so that small perturbations in code, data splits, or runtime scheduling do not hide algorithmic effects (Azuma et al., 4 Apr 2026).

In single-node FL simulation, especially in large-scale computer-vision benchmarks, communication overhead can quickly dominate local compute. The relevant overheads identified for conventional multiprocessing designs are serializing hundreds of megabytes of model parameters, crossing process boundaries, and managing metadata. At the same time, FL workloads contain multiple sources of randomness, including client sampling, data shuffling, augmentation, and regularization. BlazeFL therefore targets three linked objectives: ease of use, speed, and determinism. Ease of use is defined as integration with standard PyTorch training code without deep framework hooks, via protocol-based interfaces rather than custom base classes. Speed is pursued by eliminating cross-process serialization and IPC overhead through thread-based execution in a single address space. Determinism is pursued by assigning each client an isolated RNG stream and consuming client updates in a fixed order so that repeated runs yield bitwise-identical traces (Azuma et al., 4 Apr 2026).

This design goal places BlazeFL in a specific part of the FL systems landscape: it is not presented as a distributed runtime for multi-node deployment, but as a framework for single-node FL simulation where communication-dominated workloads make process-based simulation comparatively expensive. A plausible implication is that BlazeFL is most relevant for algorithm development, ablation studies, and reproducibility-sensitive benchmarking rather than for production federated deployment.

2. Execution model and shared-memory architecture

BlazeFL uses thread-based parallelism with in-memory parameter exchange between the server and clients. It leverages Python’s new free-threading support, specifically PEP 703 and PEP 779, to spin up worker threads for each sampled client. The architecture is organized around a single shared address space in which the server serializes the global model only once into a contiguous memory buffer at the start of communication round $r$; each worker thread then reads directly from that buffer, performs local training, and writes its delta back into a pre-allocated shared-memory tensor (Azuma et al., 4 Apr 2026).

The data movement pattern is explicit. For the downlink, the global parameters $\Delta$ are stored once in shared memory. For the uplink, client updates $\Delta_k$ are written into a shared array of shape $(\text{number\_of\_clients}, \text{model\_size})$. Because the participating clients are threads rather than separate processes, parameter broadcasts and uploads occur via pointer arithmetic and in-place tensor copies rather than serialization, pickling, RPC, or process-boundary transfer (Azuma et al., 4 Apr 2026).

The paper contrasts this design with a multiprocessing baseline in which each client lives in its own Python process and every round requires torch.save/torch.load or torch.multiprocessing to shuttle parameters through shared-memory tensors plus synchronization barriers. BlazeFL’s free-threaded mode is intended to bypass these costs, lowering round latency and improving scaling as the client count $P$ grows. The architecture is therefore optimized for communication-dominated regimes in which the cost of moving parameters, rather than the cost of local optimization itself, becomes the principal bottleneck (Azuma et al., 4 Apr 2026).

A notable architectural property is that BlazeFL avoids deep framework specialization. It is implemented to integrate with existing PyTorch-based pipelines via protocol-based interfaces rather than inheritance from custom framework base classes. This choice is central to the framework’s stated “lightweight” character and narrows the amount of infrastructure that can itself become a source of nondeterministic behavior.

3. Deterministic randomness management and bitwise reproducibility

BlazeFL’s determinism model rests on two conditions. First, each client must receive an isolated random-number stream so that client sampling, data shuffling, and augmentation do not interfere across threads. Second, aggregation must use a fixed accumulation order so that floating-point results do not depend on completion order (Azuma et al., 4 Apr 2026).

Upon initialization, BlazeFL creates a generator for each client $k \in \{0,\dots,K-1\}$ with a reproducible seed schedule. The described pattern is to derive per-client seeds from a base seed $s_0$, for example by assigning client-specific generators and applying manual_seed(s₀ + k). During local epochs and stochastic operators, BlazeFL injects the client-specific generator instead of relying on global RNG state. This is a stronger condition than ordinary global seeding, because it isolates the random streams used by concurrent client executions (Azuma et al., 4 Apr 2026).

The second component is deterministic aggregation. BlazeFL consumes client updates in the exact order of the sampled client list rather than completion order, so the FedAvg update

$$w^{(r+1)} \;=\; w^{(r)} \;+\;\frac1{|S|}\sum_{k\in S}\Delta_k^{(r)}$$

always sums in the same sequence of floating-point additions (Azuma et al., 4 Apr 2026).

The reproducibility claim is explicitly conditional. Under a fixed software/hardware stack—identified as the same OS, Python interpreter version, CUDA/cuDNN versions, and related components—and provided that all randomness in data pipelines and model layers is fed through BlazeFL-managed generators, the framework guarantees identical weights and identical intermediate tensors for runs using the same base seed $s_0$, even when client parallelism $P$ is large. The abstract further states that, under these conditions, the design yields bitwise-identical results across repeated high-concurrency runs in both thread-based and process-based modes (Azuma et al., 4 Apr 2026).

This conditional framing is important. BlazeFL does not claim unconditional determinism across heterogeneous environments. The guarantee applies only when stochastic operators consume BlazeFL-managed generators and the software/hardware stack is fixed. A common misunderstanding would be to treat the framework as making determinism automatic regardless of external kernels, library versions, or unmanaged sources of randomness; the paper’s formulation excludes that interpretation.

4. Empirical evaluation on CIFAR-10

The reported benchmark compares BlazeFL against Flower with the Ray backend on two hardware setups: a 48-core server with NVIDIA H100 and a 32-core workstation with Quadro RTX 6000. The task is CIFAR-10 image classification with a non-IID split across $\Delta$0 clients, two classes per client, five communication rounds, five local epochs on 500 samples per client, and global evaluation on the full 10 000-image test set (Azuma et al., 4 Apr 2026).

For runtime comparison, the paper defines a speedup quantity using wall-clock time for Flower and BlazeFL free-threaded mode. On a lightweight CNN, BlazeFL achieved up to approximately $\Delta$1 lower runtime than Flower on the high-performance server. For deeper models—ResNet-18, ResNet-50, and ResNet-101—the reported speedups were approximately $\Delta$2 and $\Delta$3 in communication-dominated settings. The pattern is consistent with BlazeFL’s architectural emphasis: the largest gains appear when parameter movement and framework overhead are a large fraction of total runtime (Azuma et al., 4 Apr 2026).

The scalability study varies client parallelism from $\Delta$4 to $\Delta$5. Flower’s end-to-end time is reported to plateau and then degrade as $\Delta$6 increases, whereas BlazeFL’s free-threaded mode continues to improve until hardware limits or PyTorch allocator locks become limiting. On the workstation, BlazeFL’s advantages remain pronounced for the CNN but narrow for large ResNets under VRAM pressure. The paper therefore explicitly identifies communication-bound regimes as the conditions under which BlazeFL “shines” most clearly (Azuma et al., 4 Apr 2026).

The reproducibility evaluation is equally central. Each configuration was run 10 times at $\Delta$7. Flower with no seed control exhibited a final accuracy standard deviation of 1.24 percentage points; even with global seeding, the standard deviation remained 0.18 percentage points and the model hashes disagreed. By contrast, both BlazeFL modes—free-threaded and process-based—produced zero deviation in final accuracy and bitwise-identical SHA-256 hashes of the global model at every round. Moreover, when $\Delta$8 was swept over $\Delta$9 under the same seed, BlazeFL’s global weights at each round remained exactly the same, with hashes matching the $\Delta_k$0 run. The result demonstrates invariance to concurrency level under the stated conditions (Azuma et al., 4 Apr 2026).

5. Implementation, software surface, and integration model

BlazeFL is implemented in pure Python 3.14.3 with free-threading enabled. Its software dependencies are restricted to the standard threading and multiprocessing libraries plus PyTorch for tensor operations. It imposes no external scheduler, RPC framework, or object-store runtime (Azuma et al., 4 Apr 2026).

The framework integrates with user code via Python’s typing.Protocol rather than custom base classes. In the paper’s formulation, this supports integration with standard PyTorch training code without deep framework hooks. The dependency footprint is therefore presented not merely as a convenience property but also as a systems property relevant to reproducibility: a minimal dependency surface simplifies installation and shrinks the attack surface for nondeterminism unrelated to the FL algorithm itself (Azuma et al., 4 Apr 2026).

This implementation model distinguishes BlazeFL from more infrastructure-heavy FL simulators that embed scheduling, orchestration, or object-store layers into the critical path. A plausible implication is that BlazeFL lowers the engineering cost of porting existing PyTorch experiments into federated simulation form, particularly when the goal is to preserve established training code while controlling execution order and random-state semantics precisely.

6. Scope, significance, and research context

BlazeFL is framed as a solution to a specific systems problem in FL research: how to execute large single-node simulations at high concurrency without sacrificing deterministic replay. Its main contribution is therefore not a new FL optimization algorithm, but a simulation substrate that combines thread-based shared-memory execution with isolated RNG streams and fixed-order FedAvg accumulation (Azuma et al., 4 Apr 2026).

The significance of this contribution lies in its attempt to remove an experimental confound. In many FL studies, small differences in code paths, data splits, or scheduling can be difficult to disentangle from the effects of the algorithm under test. BlazeFL’s bitwise-identical repeatability under fixed conditions is intended to make those distinctions visible. This suggests a role for the framework in controlled benchmarking, exact regression testing, and ablation studies where even small stochastic discrepancies can obscure interpretation.

At the same time, the paper delineates clear boundaries. The determinism guarantee is conditioned on a fixed software/hardware stack and on the use of BlazeFL-managed generators by stochastic operators. The runtime advantages are largest in communication-dominated workloads and may narrow for large ResNets under VRAM pressure. Hardware limits and PyTorch allocator locks can also become limiting factors as client parallelism increases. These points indicate that BlazeFL should not be construed as universally optimal across all FL simulation regimes; rather, it is specialized for single-node, shared-memory, PyTorch-centric experimentation where serialization and IPC overheads are dominant (Azuma et al., 4 Apr 2026).

In that domain, BlazeFL is presented as a lightweight platform that allows researchers to avoid choosing between speed and repeatability. Its stated contribution is the combination of free-threaded shared-memory execution, in-memory parameter exchange, isolated per-client RNG streams, and fixed-order aggregation into a simulator that is both fast and deterministic under controlled conditions (Azuma et al., 4 Apr 2026).