TinyGuard: Efficient Byzantine Defense

• TinyGuard is a Byzantine-resilient mechanism that employs low-dimensional statistical update fingerprints to efficiently detect adversarial client behaviors in federated learning.
• It extracts gradient norms, layer-wise ratios, sparsity measures, and low-order moments to form compact fingerprints that capture essential update characteristics.
• TinyGuard achieves robust aggregation by applying adaptive thresholding on normalized fingerprint distances, preserving FedAvg convergence and high accuracy under attack.

TinyGuard is a computationally efficient Byzantine-resilient aggregation mechanism for federated learning that operates by augmenting the standard FedAvg algorithm with statistical update fingerprinting. Rather than defending against adversarial (Byzantine) clients via computationally intensive full-dimensional gradient operations, TinyGuard extracts compact, low-dimensional feature vectors—"fingerprints"—from each client update, enabling efficient anomaly detection and robust aggregation even in large-scale or resource-constrained deployments. This methodology is architecture-agnostic and suitable for federated fine-tuning of contemporary high-dimensional models using parameter-efficient adapters.

1. Federated Learning Setting and Byzantine Threats

Federated learning (FL) involves a central parameter server coordinating with $n$ clients to minimize a global objective,

$\min_w F(w) = \sum_{i=1}^n p_i F_i(w),$

where $F_i$ is client $i$’s local loss and $p_i$ its data weight. Standard FedAvg proceeds in rounds: clients download $w^t$, compute local gradients $g_i^t\in\mathbb{R}^d$, send them to the server, and the server uses

$w^{t+1} = w^t - \eta\frac{1}{n}\sum_{i=1}^n g_i^t$

for aggregation.

The Byzantine threat model allows up to $f<n/2$ clients to act adversarially, sending arbitrary $g_i^*$. Typical attack modalities include random noise $(g_i^*\sim\mathcal N(0,\sigma^2I))$, sign-flipping $(g_i^*=-\alpha g_i)$, scaling $(g_i^*=\beta g_i, \beta\gg1)$, and targeted label or gradient poisoning. Classical robust aggregation (e.g., Krum, coordinatewise median) has computational complexity $O(n^2d)$ due to required pairwise distance or sorting, making them impractical on high-dimensional or resource-constrained FL deployments (Mahdavi et al., 2 Feb 2026).

2. Statistical Update Fingerprint Construction

TinyGuard constructs for each client update a fingerprint $\phi_i\in\mathbb{R}^m$ ($m\ll d$), capturing statistical and structural properties of the client's gradient $g_i$:

• Norm statistics: $\phi_{i,1} = \|g_i\|_2$, $\phi_{i,2} = \|g_i\|_1$, $\phi_{i,3} = \|g_i\|_\infty$.
• Layer-wise ratios: For networks with $L$ layers, $\phi_{i,3+\ell} = \|g_i^{(\ell)}\|_2/\|g_i\|_2$ for $1 \leq \ell \leq L$.
• Sparsity measure: $$\rho_i = \bigl| \{ j : |g_{i,j}| < \varepsilon \} \bigr| / d$$ for small $\varepsilon$.
• Low-order moments: mean $\mu_i$, variance $\sigma^2_i$, skewness $\gamma_i$

$$\mu_i = \frac{1}{d}\sum_{j=1}^d g_{i,j},\; \sigma_i^2 = \frac{1}{d}\sum_{j=1}^d (g_{i,j}-\mu_i)^2,\; \gamma_i = \frac{1}{d\sigma_i^3}\sum_{j=1}^d (g_{i,j}-\mu_i)^3$$

• Top-$k$ magnitude concentration: Fraction of $\ell_1$ norm contributed by $k$ largest absolute entries,

$$\tau_i = \frac{\sum_{j\in\text{Top-}k} |g_{i,j}|}{\|g_i\|_1}$$

These are concatenated,

$$\phi_i = [\phi_{i,1},\phi_{i,2},\phi_{i,3},\phi_{i,4},\dots,\phi_{i,3+L},\rho_i,\mu_i,\sigma_i^2,\gamma_i,\tau_i]^T \in\mathbb{R}^m,$$

yielding a highly compressed, information-rich summary suitable for anomaly detection.

3. Anomaly Detection and Statistical Handcuffs

TinyGuard identifies Byzantine behavior by measuring robust statistical deviation of $\phi_i$ from the population of all clients:

• Robust centroid: Compute the coordinatewise median $\tilde\phi = \mathrm{median}\{\phi_j\}_{j=1}^n$.
• Distance score: $s_i = \|\phi_i - \tilde\phi\|_2$.
• Robust normalization: With $m_s = \mathrm{median}\{s_j\}$, $\mathrm{MAD}_s = \mathrm{median}\{|s_j - m_s|\}$, define the normalized score

$\tilde s_i = \frac{s_i - m_s}{\mathrm{MAD}_s}$

• Adaptive thresholding: For chosen $\lambda\in[2,3]$, set

$$\tau = \mathrm{median}\{\tilde s_j\} + \lambda\;\mathrm{MAD}(\{\tilde s_j\})$$

and mark $i$ Byzantine if $\tilde s_i > \tau$.

• Statistical handcuffs: Against white-box attackers optimizing

$$\min_{g} \lambda_s \|\phi(g) - \phi(g_{\text{honest}})\|_2^2 - \lambda_a \cos(g, v_{\text{poison}})$$

under $\|g\|_2 = \|g_{\text{honest}}\|_2$, a Pareto frontier emerges: strong attacks (low stealth, large fingerprint MSE) are easily detected, while stealthy attacks (low fingerprint distance, MSE $10^{-5}$) collapse in effectiveness (attack alignment $\sim$ 0.07). These mutually exclusive attack objectives are termed "statistical handcuffs" (Mahdavi et al., 2 Feb 2026).

4. Aggregation Workflow and Complexity

Each federated round proceeds as follows:

1. Server broadcasts $w^t$ to $n$ clients.
2. Each client computes $g_i^t$, extracts $\phi_i^t$, and sends $(g_i^t, \phi_i^t)$ to the server.
3. Server collects $\{g_i^t, \phi_i^t\}_{i=1}^n$, computes robust centroid $\tilde\phi$, distance scores $s_i$, normalizes to $\tilde s_i$, and applies adaptive threshold $\tau$ to produce the Byzantine set $\mathcal{B} = \{i : \tilde s_i > \tau\}$.
4. Honest gradients aggregated:

$$g_{\text{agg}}^t = \frac{1}{n-|\mathcal{B}|} \sum_{i\notin \mathcal{B}} g_i^t$$

1. Model updated: $w^{t+1} = w^t - \eta g_{\text{agg}}^t$.

Per round complexity: clients compute $O(d)$, server extracts fingerprints in $O(nd)$, anomaly detection in $O(n)$, aggregation in $O((n-|\mathcal{B}|)d)$. Communication cost is $d$-dim gradient plus $m$-dim fingerprint per client ($m\ll d$).

5. Empirical Validation and Performance Comparison

Experiments were conducted on MNIST, Fashion-MNIST, ViT-Lite, and ViT-Small (22M parameters) with LoRA adapters (~220K trainable parameters). Scenarios included $n=50$ clients with Dirichlet non-IID splits ($\alpha=0.5$), and Byzantine fractions from 10% to 40%. Attacks tested included random noise, sign-flipping, scaling ($\beta=5$), label-flipping, and adaptive projected gradient descent (PGD).

Key empirical results:

Attack Type	TinyGuard Accuracy	Krum	TrMean	FoolsGold
Random Noise	97.7%	71.8%	96.3%	82.4%
Sign Flipping	95.3%	68.6%	94.9%	85.4%
Scaling ($\beta=5$)	96.9%	93.3%	96.4%	80.8%
Label Flipping	96.9%	69.7%	96.1%	95.9%
Average	96.7%	75.8%	95.9%	86.1%

On ViT-Small+LoRA (Fashion-MNIST): TinyGuard achieved 69.9% average accuracy, superior to Krum (63.5%) and FoolsGold (54.7%), comparable to TrMean (70.4%).

Detection precision and recall remained $\approx 0.80$ across attacks and client fractions; convergence curves matched FedAvg in benign environments and exhibited stability under attack.

Pareto analysis under adaptive attacks indicated that $\lambda_s=0.1$ yields MSE $=1.06\times10^{-4}$, alignment $=0.33$; $\lambda_s\geq1$ yields MSE $\approx 1\times 10^{-5}$, alignment $\approx 0.07$, validating statistical handcuffs.

6. Ablation Studies and Architectural Generality

• Client count: For 20% sign-flip attacks, $\lambda=2.5$, $\alpha=0.5$, accuracy/precision: $N=50$ (67.8%, 0.800), $N=100$ (80.8%, 0.801), $N=150$ (82.7%, 0.801).
• Threshold sensitivity: ($N=50$, sign-flip) $\lambda=2.5$ (67.8%, 0.800), $\lambda=5.0$ (69.8%, 0.798), $\lambda=10.0$ (68.9%, 0.800).
• Data heterogeneity: ($N=50$, sign-flip) $\alpha=0.5$ (67.8%, 0.800), $\alpha=0.25$ (64.6%, 0.811), $\alpha=0.1$ (30.6%, 0.810).
• Architecture-agnosticism: With LoRA adapters (1% trained parameters), fingerprint-based detection remains discriminative. The method is directly applicable to parameter-efficient transformer fine-tuning without $O(n^2d)$ costs.

7. Summary of Properties and Significance

TinyGuard introduces an $O(nd)$-complexity, fingerprint-based Byzantine defense for federated learning that:

• Preserves FedAvg convergence in benign settings.
• Achieves up to 95%+ accuracy in the presence of diverse Byzantine attacks.
• Maintains stable detection precision ($\approx 0.8$) under variation in client count, sensitivity threshold, and data heterogeneity.
• Imposes negligible computational and communication overhead compared to $O(n^2d)$ legacy defenses.
• Operates with no modification of underlying optimization dynamics.
• Transfers directly to parameter-efficient fine-tuning workflows for high-dimensional foundation models.

Extensive experimentation and ablation analysis establish its effectiveness, scalability, and architectural flexibility (Mahdavi et al., 2 Feb 2026).