---
title: 'BackFed: FL Backdoor Benchmark Suite'
url: https://www.emergentmind.com/topics/backfed
type: topic
---

# BackFed: FL Backdoor Benchmark Suite

BackFed is a benchmark suite for backdoor attacks and defenses in federated learning (FL), introduced to standardize, streamline, and reliably evaluate security methods under practical constraints [2507.04903]. It is motivated by a recurrent problem in the FL backdoor literature: divergent experimental settings, implementation errors, and unrealistic assumptions can make published comparisons difficult to interpret and can weaken conclusions about real-world effectiveness. BackFed addresses this by combining a modular server–client architecture, a standardized evaluation pipeline, multi-processing for large-scale experimentation, and explicit support for both Computer Vision and Natural Language Processing workloads. The framework is openly available at `https://github.com/thinh-dao/BackFed`.

## 1. Problem setting and rationale

In the threat model targeted by BackFed, FL systems are vulnerable to backdoor attacks in which adversaries train local models on poisoned data and submit poisoned model updates so as to compromise the global model [2507.04903]. The benchmark is therefore organized around a dual objective: evaluating attack effectiveness and evaluating defense behavior under identical conditions.

A central reason for the framework’s existence is methodological rather than algorithmic. The paper identifies divergent experimental settings, implementation errors, and unrealistic assumptions as obstacles to fair comparisons and valid conclusions. BackFed is presented as a comprehensive benchmark suite rather than as a new attack or a new defense. A common misconception is to treat it as a mitigation method in its own right; more precisely, it is an evaluation environment intended to make comparisons reproducible, controlled, and practically grounded.

The paper further characterizes BackFed as a “plug-and-play environment” for comprehensively and reliably evaluating new attacks and defenses. This suggests an emphasis on experimental standardization as a first-class research contribution, especially in a literature where minor changes in partitioning, aggregation, malicious-client fraction, or model family can materially alter reported attack success.

## 2. System architecture

BackFed follows a server–client split with modular components on both sides of the FL process [2507.04903]. The design is explicitly divided into a Server Module, Client Module, Defense Module, and Evaluation & Analysis Module.

| Layer | Component | Function |
|---|---|---|
| Server Module | ClientManager | Client registry, registration, deregistration |
| Server Module | ClientSelection | Sampling strategies: uniform, Dirichlet non-IID, power-law |
| Server Module | FLTrainer | Global aggregation, model initialization, checkpointing |
| Client Module | ContextActor | Local model copy, data loaders, optimizer, scheduler |
| Client Module | PoisonModule | Injects poisoned data or manipulates gradient updates |
| Defense Module | PoisonedDataDetection | Client-reported loss or gradient-signature |
| Defense Module | BackdoorDetection & Identification | K-means on updates, clustering on norms |
| Defense Module | BackdoorMitigation / Inhibition | Krum, Trimmed Mean, Median, Bulyan |
| Evaluation & Analysis Module | Metrics and logging | ASR, Global Accuracy, Robustness, per-round curves, final summaries |

Within the client-side attack surface, the PoisonModule is divided into `DataPoisoningAttack` and `TrainingControllableAttack`. The paper lists BadNet, Blended, and WaNet as examples of the former, and Model Replacement and Scaling as examples of the latter. On the defense side, the framework distinguishes between detection-oriented modules and mitigation-oriented modules, allowing researchers to evaluate both filtering and robust aggregation within a common runtime.

BackFed also includes a multi-processing implementation intended to accelerate large-scale federated experiments. Client-side training is parallelized with Python’s `multiprocessing.Pool` or `torch.multiprocessing`. In a single federated round, the server samples clients, maps `client_update` across them in parallel, collects tuples of `(delta_w, data_size, client_id)`, and applies an aggregator before updating the global model. In the simple FedAvg case, aggregation is weighted by local data size $n_k$; more robust aggregators replace that sum with a median or trimmed-mean across each parameter dimension.

## 3. Extensibility and API design

A defining feature of BackFed is its small set of base classes and registry functions for adding new attacks and defenses [2507.04903]. The interface is intentionally narrow:

```python
class BaseAttack:
    def poison_data(self, local_loader) -> poisoned_loader
    def manipulate_update(self, model, optimizer, loader) -> Δw

class BaseDefense:
    def filter_updates(self, updates: List[Δw]) -> List[Δw_filtered]
```

Method registration is handled through import-time registries:

```python
ATTACK_REGISTRY = {}
DEFENSE_REGISTRY = {}

def register_attack(name):
    def decorator(cls):
        ATTACK_REGISTRY[name] = cls
        return cls
    return decorator

def register_defense(name):
    def decorator(cls):
        DEFENSE_REGISTRY[name] = cls
        return cls
    return decorator
```

The paper gives a concrete example of registering a `BadNet` attack through `@register_attack("BadNet")`. In that example, the attack object stores a `trigger_pattern`, `target_label`, and `poison_rate`, and its `poison_data` method overlays the trigger on a sampled fraction of images and sets the label to the target. A corresponding defense example is `@register_defense("Krum")`, where `filter_updates` applies the Krum algorithm and returns the update closest to the majority.

This API design has two consequences. First, attack implementations can intervene either at the data level through `poison_data` or at the update level through `manipulate_update`. Second, defenses are normalized to an update-filtering interface, which makes side-by-side evaluation easier when robust aggregation methods are the main object of study.

## 4. Evaluation pipeline and metrics

BackFed scripts evaluation end-to-end as a standardized pipeline [2507.04903]. The workflow is defined in five high-level steps.

The first step is dataset partitioning. A dataset such as CIFAR-10, MNIST, or AG News is chosen and partitioned across $N$ clients under IID or non-IID conditions, with Dirichlet $\alpha$ used for non-IID splits. The second step is configuration of the threat model: the malicious-client fraction $f$ is chosen, and the attack type is selected, for example data poisoning or model replacement.

The third step is the federated training loop over $R$ rounds. In each round, the server samples $K$ clients; each client performs local epochs $E$ on its possibly poisoned data; clients submit $\Delta w_k$; the server applies `defense.filter_updates(...)`; and the server aggregates into the global model and broadcasts the result to the next round. The fourth step is evaluation at selected intervals, using clean accuracy on a held-out clean test set and backdoor accuracy on a held-out triggered test set. The fifth step is final analysis and plotting.

BackFed standardizes three core metrics. Attack Success Rate (ASR) is the fraction of triggered samples predicted as the target. Global Accuracy (GA) is the fraction of correct clean predictions. Model Robustness (RB) is defined as the drop in GA under attack versus benign training. By fixing these metrics within a common pipeline, the framework supports comparisons across attacks, defenses, datasets, and partition regimes without redefining the measurement protocol for each paper.

## 5. Benchmark coverage and practical constraints

The benchmark spans both Computer Vision and NLP tasks, with corresponding model families and threat assumptions [2507.04903]. For Computer Vision, the supported datasets are MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet. For NLP, the supported datasets are AG News, IMDb sentiment classification, and Yelp Reviews.

The model architectures are similarly divided by modality. In Computer Vision, the paper lists Simple CNN, ResNet-18, and MobileNetV2. In NLP, it lists LSTM, CNN-text, and BERT-Small. This modality breadth is important because backdoor behavior and defense efficacy can vary substantially with architecture class and data type.

The threat model is defined with several practical constraints. The server is honest-but-curious; the aggregator may be honest or malicious. Malicious clients may collude and may know the global model architecture and hyper-parameters, but they do not know other clients’ data. A budget constraint enforces that at most $f \le 20\%$ of clients are malicious per round. All parties share the same model definition, and no raw data are available off-device. These assumptions are narrower than unrestricted white-box adversarial settings, but they are intended to reflect practical FL deployment constraints rather than unconstrained attack capability.

## 6. Empirical findings, trade-offs, and failure modes

Using BackFed, the paper reports large-scale studies of representative attacks and defenses across datasets, architectures, and non-IID settings [2507.04903]. Several comparative patterns are emphasized.

For attacks, data-poisoning methods such as BadNet, Blend, and WaNet typically achieve ASR $> 90\%$ on CIFAR-10 under IID with no defense, while dropping GA by $< 2\%$. Model-replacement attacks such as scaled-$\Delta$ can push ASR to $100\%$ in as few as 10 rounds, but they incur a larger GA drop of $5$–$7\%$. These results highlight a recurrent distinction between high-stealth data poisoning and more disruptive update-space manipulation.

For defenses, the reported trade-offs are explicit. Trimmed-Mean reduces ASR to approximately $30\%$ but drops GA by $4\%$. Multi-Krum is more conservative, with ASR around $15\%$ and GA drop around $6\%$. FLAME, described as a cluster-based filter, yields ASR below $10\%$ with GA drop below $3\%$. The paper therefore presents defense quality not as a single scalar notion but as a balance between attack suppression and clean-model preservation.

BackFed is also used to expose failure modes. As non-IID heterogeneity increases, specifically as Dirichlet $\alpha \rightarrow 0.1$, robust aggregators can falsely label honest outliers as malicious, thereby hurting GA. Attacks with imperceptible triggers, such as WaNet, can evade many simple norm-based defenses. The paper summarizes one practical trade-off succinctly: guaranteeing ASR $< 10\%$ often requires a GA drop of at least $3$–$5\%$ in practice.

The framework also distills best-practice guidance from these experiments. It recommends a two-stage defense that first clusters updates by similarity and then applies trimmed-mean within each cluster. It recommends dynamically adapting defense strictness based on the observed variance of client updates. For highly non-IID data, it recommends calibrating defense thresholds on a small validation set held by the server to reduce honest-client false positives. It also recommends monitoring both ASR and GA curves per round, since early indicators of attack success often appear within 5–10 rounds.

## 7. Position within the federated backdoor literature

BackFed occupies an infrastructural position within the FL backdoor literature: it is designed to evaluate attacks and defenses under identical conditions rather than to replace them with a single new algorithm [2507.04903]. That positioning matters because the field includes defenses with very different operating principles, ranging from robust aggregation to clustering-based filtering to perturbation-based training schemes.

A later example is FedBAP, which mitigates backdoor attacks by reducing the model’s reliance on backdoor triggers through perturbed trigger generation, benign adversarial perturbations, and an adaptive scaling mechanism [2507.21177]. A plausible implication is that BackFed-type benchmark suites are especially useful for comparing such heterogeneous defenses, because they place trigger-based, update-based, and aggregation-based methods inside a common protocol with shared datasets, partitions, metrics, and malicious-client budgets.

In that sense, BackFed’s principal contribution is not only software modularity but also experimental normalization. By fixing the server–client workflow, the attack/defense APIs, the evaluation metrics, and the practical constraints, it provides a standardized basis for judging whether a reported improvement reflects a genuinely stronger FL backdoor method or merely a different experimental regime.

Source: https://www.emergentmind.com/topics/backfed