BackFed: FL Backdoor Benchmark Suite

• BackFed is a benchmark suite for backdoor attacks and defenses in federated learning, offering standardized experimental setups and modular APIs.
• It employs a modular server–client architecture with multi-processing support to ensure reproducible and controlled large-scale experiments.
• The framework spans both Computer Vision and NLP tasks, using metrics like ASR, GA, and RB to reveal practical trade-offs in FL security.

BackFed is a benchmark suite for backdoor attacks and defenses in federated learning (FL), introduced to standardize, streamline, and reliably evaluate security methods under practical constraints (Dao et al., 7 Jul 2025). It is motivated by a recurrent problem in the FL backdoor literature: divergent experimental settings, implementation errors, and unrealistic assumptions can make published comparisons difficult to interpret and can weaken conclusions about real-world effectiveness. BackFed addresses this by combining a modular server–client architecture, a standardized evaluation pipeline, multi-processing for large-scale experimentation, and explicit support for both Computer Vision and Natural Language Processing workloads. The framework is openly available at https://github.com/thinh-dao/BackFed.

1. Problem setting and rationale

In the threat model targeted by BackFed, FL systems are vulnerable to backdoor attacks in which adversaries train local models on poisoned data and submit poisoned model updates so as to compromise the global model (Dao et al., 7 Jul 2025). The benchmark is therefore organized around a dual objective: evaluating attack effectiveness and evaluating defense behavior under identical conditions.

A central reason for the framework’s existence is methodological rather than algorithmic. The paper identifies divergent experimental settings, implementation errors, and unrealistic assumptions as obstacles to fair comparisons and valid conclusions. BackFed is presented as a comprehensive benchmark suite rather than as a new attack or a new defense. A common misconception is to treat it as a mitigation method in its own right; more precisely, it is an evaluation environment intended to make comparisons reproducible, controlled, and practically grounded.

The paper further characterizes BackFed as a “plug-and-play environment” for comprehensively and reliably evaluating new attacks and defenses. This suggests an emphasis on experimental standardization as a first-class research contribution, especially in a literature where minor changes in partitioning, aggregation, malicious-client fraction, or model family can materially alter reported attack success.

2. System architecture

BackFed follows a server–client split with modular components on both sides of the FL process (Dao et al., 7 Jul 2025). The design is explicitly divided into a Server Module, Client Module, Defense Module, and Evaluation & Analysis Module.

Layer	Component	Function
Server Module	ClientManager	Client registry, registration, deregistration
Server Module	ClientSelection	Sampling strategies: uniform, Dirichlet non-IID, power-law
Server Module	FLTrainer	Global aggregation, model initialization, checkpointing
Client Module	ContextActor	Local model copy, data loaders, optimizer, scheduler
Client Module	PoisonModule	Injects poisoned data or manipulates gradient updates
Defense Module	PoisonedDataDetection	Client-reported loss or gradient-signature
Defense Module	BackdoorDetection & Identification	K-means on updates, clustering on norms
Defense Module	BackdoorMitigation / Inhibition	Krum, Trimmed Mean, Median, Bulyan
Evaluation & Analysis Module	Metrics and logging	ASR, Global Accuracy, Robustness, per-round curves, final summaries

Within the client-side attack surface, the PoisonModule is divided into DataPoisoningAttack and TrainingControllableAttack. The paper lists BadNet, Blended, and WaNet as examples of the former, and Model Replacement and Scaling as examples of the latter. On the defense side, the framework distinguishes between detection-oriented modules and mitigation-oriented modules, allowing researchers to evaluate both filtering and robust aggregation within a common runtime.

BackFed also includes a multi-processing implementation intended to accelerate large-scale federated experiments. Client-side training is parallelized with Python’s multiprocessing.Pool or torch.multiprocessing. In a single federated round, the server samples clients, maps client_update across them in parallel, collects tuples of (delta_w, data_size, client_id), and applies an aggregator before updating the global model. In the simple FedAvg case, aggregation is weighted by local data size $n_k$; more robust aggregators replace that sum with a median or trimmed-mean across each parameter dimension.

3. Extensibility and API design

A defining feature of BackFed is its small set of base classes and registry functions for adding new attacks and defenses (Dao et al., 7 Jul 2025). The interface is intentionally narrow:

$\alpha$5

Method registration is handled through import-time registries:

$\alpha$6

The paper gives a concrete example of registering a BadNet attack through @register_attack("BadNet"). In that example, the attack object stores a trigger_pattern, target_label, and poison_rate, and its poison_data method overlays the trigger on a sampled fraction of images and sets the label to the target. A corresponding defense example is @register_defense("Krum"), where filter_updates applies the Krum algorithm and returns the update closest to the majority.

This API design has two consequences. First, attack implementations can intervene either at the data level through poison_data or at the update level through manipulate_update. Second, defenses are normalized to an update-filtering interface, which makes side-by-side evaluation easier when robust aggregation methods are the main object of study.

4. Evaluation pipeline and metrics

BackFed scripts evaluation end-to-end as a standardized pipeline (Dao et al., 7 Jul 2025). The workflow is defined in five high-level steps.

The first step is dataset partitioning. A dataset such as CIFAR-10, MNIST, or AG News is chosen and partitioned across $N$ clients under IID or non-IID conditions, with Dirichlet $\alpha$ used for non-IID splits. The second step is configuration of the threat model: the malicious-client fraction $f$ is chosen, and the attack type is selected, for example data poisoning or model replacement.

The third step is the federated training loop over $R$ rounds. In each round, the server samples $K$ clients; each client performs local epochs $E$ on its possibly poisoned data; clients submit $\Delta w_k$; the server applies defense.filter_updates(...); and the server aggregates into the global model and broadcasts the result to the next round. The fourth step is evaluation at selected intervals, using clean accuracy on a held-out clean test set and backdoor accuracy on a held-out triggered test set. The fifth step is final analysis and plotting.

BackFed standardizes three core metrics. Attack Success Rate (ASR) is the fraction of triggered samples predicted as the target. Global Accuracy (GA) is the fraction of correct clean predictions. Model Robustness (RB) is defined as the drop in GA under attack versus benign training. By fixing these metrics within a common pipeline, the framework supports comparisons across attacks, defenses, datasets, and partition regimes without redefining the measurement protocol for each paper.

5. Benchmark coverage and practical constraints

The benchmark spans both Computer Vision and NLP tasks, with corresponding model families and threat assumptions (Dao et al., 7 Jul 2025). For Computer Vision, the supported datasets are MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet. For NLP, the supported datasets are AG News, IMDb sentiment classification, and Yelp Reviews.

The model architectures are similarly divided by modality. In Computer Vision, the paper lists Simple CNN, ResNet-18, and MobileNetV2. In NLP, it lists LSTM, CNN-text, and BERT-Small. This modality breadth is important because backdoor behavior and defense efficacy can vary substantially with architecture class and data type.

The threat model is defined with several practical constraints. The server is honest-but-curious; the aggregator may be honest or malicious. Malicious clients may collude and may know the global model architecture and hyper-parameters, but they do not know other clients’ data. A budget constraint enforces that at most $f \le 20\%$ of clients are malicious per round. All parties share the same model definition, and no raw data are available off-device. These assumptions are narrower than unrestricted white-box adversarial settings, but they are intended to reflect practical FL deployment constraints rather than unconstrained attack capability.

6. Empirical findings, trade-offs, and failure modes

Using BackFed, the paper reports large-scale studies of representative attacks and defenses across datasets, architectures, and non-IID settings (Dao et al., 7 Jul 2025). Several comparative patterns are emphasized.

For attacks, data-poisoning methods such as BadNet, Blend, and WaNet typically achieve ASR $> 90\%$ on CIFAR-10 under IID with no defense, while dropping GA by $N$0. Model-replacement attacks such as scaled-$N$1 can push ASR to $N$2 in as few as 10 rounds, but they incur a larger GA drop of $N$3–$N$4. These results highlight a recurrent distinction between high-stealth data poisoning and more disruptive update-space manipulation.

For defenses, the reported trade-offs are explicit. Trimmed-Mean reduces ASR to approximately $N$5 but drops GA by $N$6. Multi-Krum is more conservative, with ASR around $N$7 and GA drop around $N$8. FLAME, described as a cluster-based filter, yields ASR below $N$9 with GA drop below $\alpha$0. The paper therefore presents defense quality not as a single scalar notion but as a balance between attack suppression and clean-model preservation.

BackFed is also used to expose failure modes. As non-IID heterogeneity increases, specifically as Dirichlet $\alpha$1, robust aggregators can falsely label honest outliers as malicious, thereby hurting GA. Attacks with imperceptible triggers, such as WaNet, can evade many simple norm-based defenses. The paper summarizes one practical trade-off succinctly: guaranteeing ASR $\alpha$2 often requires a GA drop of at least $\alpha$3–$\alpha$4 in practice.

The framework also distills best-practice guidance from these experiments. It recommends a two-stage defense that first clusters updates by similarity and then applies trimmed-mean within each cluster. It recommends dynamically adapting defense strictness based on the observed variance of client updates. For highly non-IID data, it recommends calibrating defense thresholds on a small validation set held by the server to reduce honest-client false positives. It also recommends monitoring both ASR and GA curves per round, since early indicators of attack success often appear within 5–10 rounds.

7. Position within the federated backdoor literature

BackFed occupies an infrastructural position within the FL backdoor literature: it is designed to evaluate attacks and defenses under identical conditions rather than to replace them with a single new algorithm (Dao et al., 7 Jul 2025). That positioning matters because the field includes defenses with very different operating principles, ranging from robust aggregation to clustering-based filtering to perturbation-based training schemes.

A later example is FedBAP, which mitigates backdoor attacks by reducing the model’s reliance on backdoor triggers through perturbed trigger generation, benign adversarial perturbations, and an adaptive scaling mechanism (Yan et al., 26 Jul 2025). A plausible implication is that BackFed-type benchmark suites are especially useful for comparing such heterogeneous defenses, because they place trigger-based, update-based, and aggregation-based methods inside a common protocol with shared datasets, partitions, metrics, and malicious-client budgets.

In that sense, BackFed’s principal contribution is not only software modularity but also experimental normalization. By fixing the server–client workflow, the attack/defense APIs, the evaluation metrics, and the practical constraints, it provides a standardized basis for judging whether a reported improvement reflects a genuinely stronger FL backdoor method or merely a different experimental regime.