TinyGuard: Efficient Byzantine Defense

• TinyGuard is a Byzantine-resilient mechanism that employs low-dimensional statistical update fingerprints to efficiently detect adversarial client behaviors in federated learning.
• It extracts gradient norms, layer-wise ratios, sparsity measures, and low-order moments to form compact fingerprints that capture essential update characteristics.
• TinyGuard achieves robust aggregation by applying adaptive thresholding on normalized fingerprint distances, preserving FedAvg convergence and high accuracy under attack.

TinyGuard is a computationally efficient Byzantine-resilient aggregation mechanism for federated learning that operates by augmenting the standard FedAvg algorithm with statistical update fingerprinting. Rather than defending against adversarial (Byzantine) clients via computationally intensive full-dimensional gradient operations, TinyGuard extracts compact, low-dimensional feature vectors—"fingerprints"—from each client update, enabling efficient anomaly detection and robust aggregation even in large-scale or resource-constrained deployments. This methodology is architecture-agnostic and suitable for federated fine-tuning of contemporary high-dimensional models using parameter-efficient adapters.

1. Federated Learning Setting and Byzantine Threats

Federated learning (FL) involves a central parameter server coordinating with $n$ clients to minimize a global objective,

$\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$

where $F_i$ is client $i$’s local loss and $p_i$ its data weight. Standard FedAvg proceeds in rounds: clients download $w^t$, compute local gradients $g_i^t\in\mathbb{R}^d$, send them to the server, and the server uses

$w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$

for aggregation.

The Byzantine threat model allows up to $f<n/2$ clients to act adversarially, sending arbitrary $g_i^*$. Typical attack modalities include random noise $\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$0, sign-flipping $\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$1, scaling $\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$2, and targeted label or gradient poisoning. Classical robust aggregation (e.g., Krum, coordinatewise median) has computational complexity $\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$3 due to required pairwise distance or sorting, making them impractical on high-dimensional or resource-constrained FL deployments (Mahdavi et al., 2 Feb 2026).

2. Statistical Update Fingerprint Construction

TinyGuard constructs for each client update a fingerprint $\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$4 ($\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$5), capturing statistical and structural properties of the client's gradient $\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$6:

• Norm statistics: $\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$7, $\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$8, $\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$9.
• Layer-wise ratios: For networks with $F_i$0 layers, $F_i$1 for $F_i$2.
• Sparsity measure: $F_i$3 for small $F_i$4.
• Low-order moments: mean $F_i$5, variance $F_i$6, skewness $F_i$7

$F_i$8

• Top-$F_i$9 magnitude concentration: Fraction of $i$0 norm contributed by $i$1 largest absolute entries,

$i$2

These are concatenated,

$i$3

yielding a highly compressed, information-rich summary suitable for anomaly detection.

3. Anomaly Detection and Statistical Handcuffs

TinyGuard identifies Byzantine behavior by measuring robust statistical deviation of $i$4 from the population of all clients:

• Robust centroid: Compute the coordinatewise median $i$5.
• Distance score: $i$6.
• Robust normalization: With $i$7, $i$8, define the normalized score

$i$9

• Adaptive thresholding: For chosen $p_i$0, set

$p_i$1

and mark $p_i$2 Byzantine if $p_i$3.

• Statistical handcuffs: Against white-box attackers optimizing

$p_i$4

under $p_i$5, a Pareto frontier emerges: strong attacks (low stealth, large fingerprint MSE) are easily detected, while stealthy attacks (low fingerprint distance, MSE $p_i$6) collapse in effectiveness (attack alignment $p_i$7 0.07). These mutually exclusive attack objectives are termed "statistical handcuffs" (Mahdavi et al., 2 Feb 2026).

4. Aggregation Workflow and Complexity

Each federated round proceeds as follows:

1. Server broadcasts $p_i$8 to $p_i$9 clients.
2. Each client computes $w^t$0, extracts $w^t$1, and sends $w^t$2 to the server.
3. Server collects $w^t$3, computes robust centroid $w^t$4, distance scores $w^t$5, normalizes to $w^t$6, and applies adaptive threshold $w^t$7 to produce the Byzantine set $w^t$8.
4. Honest gradients aggregated:

$w^t$9

1. Model updated: $g_i^t\in\mathbb{R}^d$0.

Per round complexity: clients compute $g_i^t\in\mathbb{R}^d$1, server extracts fingerprints in $g_i^t\in\mathbb{R}^d$2, anomaly detection in $g_i^t\in\mathbb{R}^d$3, aggregation in $g_i^t\in\mathbb{R}^d$4. Communication cost is $g_i^t\in\mathbb{R}^d$5-dim gradient plus $g_i^t\in\mathbb{R}^d$6-dim fingerprint per client ($g_i^t\in\mathbb{R}^d$7).

5. Empirical Validation and Performance Comparison

Experiments were conducted on MNIST, Fashion-MNIST, ViT-Lite, and ViT-Small (22M parameters) with LoRA adapters (~220K trainable parameters). Scenarios included $g_i^t\in\mathbb{R}^d$8 clients with Dirichlet non-IID splits ($g_i^t\in\mathbb{R}^d$9), and Byzantine fractions from 10% to 40%. Attacks tested included random noise, sign-flipping, scaling ($w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$0), label-flipping, and adaptive projected gradient descent (PGD).

Key empirical results:

Attack Type	TinyGuard Accuracy	Krum	TrMean	FoolsGold
Random Noise	97.7%	71.8%	96.3%	82.4%
Sign Flipping	95.3%	68.6%	94.9%	85.4%
Scaling ($w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$1)	96.9%	93.3%	96.4%	80.8%
Label Flipping	96.9%	69.7%	96.1%	95.9%
Average	96.7%	75.8%	95.9%	86.1%

On ViT-Small+LoRA (Fashion-MNIST): TinyGuard achieved 69.9% average accuracy, superior to Krum (63.5%) and FoolsGold (54.7%), comparable to TrMean (70.4%).

Detection precision and recall remained $w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$2 across attacks and client fractions; convergence curves matched FedAvg in benign environments and exhibited stability under attack.

Pareto analysis under adaptive attacks indicated that $w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$3 yields MSE $w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$4, alignment $w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$5; $w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$6 yields MSE $w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$7, alignment $w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$8, validating statistical handcuffs.

6. Ablation Studies and Architectural Generality

• Client count: For 20% sign-flip attacks, $w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$9, $f<n/2$0, accuracy/precision: $f<n/2$1 (67.8%, 0.800), $f<n/2$2 (80.8%, 0.801), $f<n/2$3 (82.7%, 0.801).
• Threshold sensitivity: ($f<n/2$4, sign-flip) $f<n/2$5 (67.8%, 0.800), $f<n/2$6 (69.8%, 0.798), $f<n/2$7 (68.9%, 0.800).
• Data heterogeneity: ($f<n/2$8, sign-flip) $f<n/2$9 (67.8%, 0.800), $g_i^*$0 (64.6%, 0.811), $g_i^*$1 (30.6%, 0.810).
• Architecture-agnosticism: With LoRA adapters (1% trained parameters), fingerprint-based detection remains discriminative. The method is directly applicable to parameter-efficient transformer fine-tuning without $g_i^*$2 costs.

7. Summary of Properties and Significance

TinyGuard introduces an $g_i^*$3-complexity, fingerprint-based Byzantine defense for federated learning that:

• Preserves FedAvg convergence in benign settings.
• Achieves up to 95%+ accuracy in the presence of diverse Byzantine attacks.
• Maintains stable detection precision ($g_i^*$4) under variation in client count, sensitivity threshold, and data heterogeneity.
• Imposes negligible computational and communication overhead compared to $g_i^*$5 legacy defenses.
• Operates with no modification of underlying optimization dynamics.
• Transfers directly to parameter-efficient fine-tuning workflows for high-dimensional foundation models.

Extensive experimentation and ablation analysis establish its effectiveness, scalability, and architectural flexibility (Mahdavi et al., 2 Feb 2026).