FLTrojan: Privacy Leakage Attack in FL

• FLTrojan is a novel attack exploiting federated learning by targeting intermediate model snapshots and selectively manipulating layer weights.
• It employs methods like Victim Round Identification and Maximizing Data Memorization to enhance sensitive data extraction through precise weight tampering.
• Experimental results demonstrate FLTrojan’s superiority over baseline methods, highlighting major privacy risks in distributed large language model training.

FLTrojan refers to a novel class of privacy leakage attacks in the federated learning (FL) context, specifically targeting LLMs collaboratively trained across distributed clients whose datasets often contain privacy-sensitive information. These attacks exploit model weight dynamics and federated aggregation procedures to maximize the extraction of sensitive user data, employing selective manipulation of neural network parameters during or after the FL process.

1. Conceptual Framework and Motivation

FLTrojan attacks are designed to circumvent the privacy advantages of federated learning, which traditionally aims to preserve user data confidentiality by keeping raw data local. Conventional privacy attacks typically focus on indiscriminate model extraction or rely on access to final model checkpoints. FLTrojan distinguishes itself by:

• Targeting the intermediate model snapshots from federated training rounds, capitalizing on transient memorization phenomena before models "forget" user-specific sequences due to distributed optimization and catastrophic forgetting.
• Utilizing selective weight tampering of the most sensitive model layers, specifically those that disproportionately represent out-of-distribution privacy-sensitive content, such as phone numbers, credentials, or medical records.
• Maximizing both the exposure of memorized content and the efficiency of membership inference and data reconstruction for adversary-crafted prompts or prefixes.

This paradigm challenges the prevalent notion that federated aggregation and distributed data silos substantially mitigate privacy risks, highlighting new attack surfaces unique to FL and modern LLM architectures (Rashid et al., 2023).

2. Technical Mechanisms of the FLTrojan Attack

Victim Round Identification (VRI)

The initial phase of FLTrojan is to locate federated learning rounds during which a target client's sensitive data substantially influenced the global model parameters. This is accomplished by:

• Monitoring model "exposure" (as per Carlini et al., 2019) for privacy canaries across model versions.
• Employing statistical tests (e.g., two-sample t-tests) on layer-wise norm differences, focusing on selective layers ($L_{\star}$) that undergo significant changes when models are fine-tuned with privacy-sensitive data.
• Identifying victim snapshots ($\widehat{G}_i$) whose selective weights ($W_{\star}$) have higher similarity to sensitive-data-fine-tuned models than to regular-data-fine-tuned ones.

Selective Weight Tampering (MDM: Maximizing Data Memorization)

After identifying victim-influenced rounds, FLTrojan proceeds by:

• Selectively optimizing the model head layers and proximal transformer blocks ($L_{\star}$), amplifying memorization of private data with minimal effect on non-sensitive model utility.
• Defining the maximization problem for memorization score:

$$\underset{\mathcal{T}}{\arg\max}~ \mathbf{M}(\mathcal{D}_{\text{sen}}, \mathcal{T}(W_{\star}))$$

where $\mathbf{M}$ quantifies memorization, $\mathcal{T}$ applies transformations to $W_{\star}$, and $\mathcal{D}_{\text{sen}}$ is the set of sensitive sequences.

• Leveraging Weight Transformation Learning (WTL), which learns mappings from regular to sensitive weight matrices using supervised regression over layer groups. This enables adversaries to "retrofit" models with heightened sensitivity to previously memorized private data.

Extraction Attacks

FLTrojan employs two main extraction techniques:

• Data Reconstruction: The adversary uses known prefixes (from context or leaked metadata) to prompt the model, then applies beam search or similar enumeration on model outputs, seeking to reconstruct the complete sensitive sequence.
• Membership Inference: Likelihood-ratio testing between target and reference models distinguishes whether a sequence was present in the client’s training data.

3. Experimental Evaluation and Key Findings

FLTrojan was evaluated on multiple model architectures—Gemma, Llama-2, GPT-2 (autoregressive), BERT (masked LM)—and datasets, including Wikitext-103, Penn Treebank, and Enron emails containing realistic privacy canaries.

• Victim Round Identification Recall: 86.3–94.1%, necessitating only round-level access, and applicable both in static (post-training) and dynamic (online training) attack scenarios.
• Data Reconstruction Rate: Up to 71% of privacy canaries re-assembled (Gemma, dynamic + server cooperation), and 64% on real Enron email samples; GPT-2 achieved up to 62.5% with static attacks.
• Membership Inference Recall: Improved by up to 29% compared to previous best baselines, reaching ~61% in optimal cases.
• Baseline Comparison: Leakage with FLTrojan (static/dynamic WTL) outperformed existing attacks such as Decepticons (with server collusion), FILM, and LAMP.

Attack	Setup	Reconstruction (%)	MI Recall (%)
Last Round	Baseline	~20–30	~30–35
Intermediate Rounds	Baseline	~40	~39
FLTrojan (Static)	No server, WTL	~62	~54
FLTrojan (Dynamic)	No server, WTL	~68	~57
FLTrojan (+Server)	WTL	~71	~61
Decepticons	Prior work	~68	N/A
FILM	Prior work	<<62	--

Data condensed from [(Rashid et al., 2023), Table 6].

Defense mechanisms such as differential privacy (DP) and deduplication were found to reduce leakage but simultaneously degraded model utility—higher perplexity, lower reconstruction rates—indicating insufficient defense against FLTrojan.

4. Layer Sensitivity Analysis and Attack Design Principles

Empirical analysis reveals that fine-tuning with privacy-sensitive data induces pronounced changes in a non-uniform, layer-selective fashion:

• The model head layers, along with adjacent transformer blocks ($L_{\star}$), disproportionately absorb the injected memorization.
• The magnitude of mean change ($\Delta W_{\star}$) after fine-tuning sensitive data is substantially higher than that for other layers ($\Delta W_{\ominus}$):

$$\Delta W_{\star} = \frac{1}{|L_{\star}|} \sum_{l \in L_{\star}} | W(f_G(\mathcal{D}_{\text{sen}})) - W(f_G(\cdot)) |_l$$

It was observed that $||\Delta W_{\star}|| \gg ||\Delta W_{\ominus}||$.

• The attack achieves a trade-off between the number of layers tampered and the extent of leakage versus accuracy retention. A "sweet spot" exists wherein privacy leakage is maximized while model utility (as measured by validation accuracy or perplexity) remains minimally affected.

5. Implications for Federated Learning Security

The existence and efficacy of FLTrojan imply:

• Intermediate model snapshots are a critical privacy risk: Sharing per-round global models exposes a broader attack surface than previously appreciated, enabling leakage even without server-side collusion.
• Fine-grained, layer-selective manipulations are significantly more potent than model-wide attacks: Current privacy defenses that do not explicitly consider neural layer sensitivity or round-based exposure are inadequate.
• Server collusion amplifies, but is not necessary for, successful attacks: Attacker capabilities to reconstruct sensitive data are only marginally enhanced by malicious server involvement; FLTrojan works effectively with minimal assumptions.
• High-stakes leakage of out-of-distribution privacy-sensitive data: The focus on sequences such as credentials or phone numbers underscores the practical impact in critical domains like healthcare, finance, and personal communications.

A plausible implication is that secure model aggregation and fine-grained, sensitivity-aware privacy protections must be strengthened for federated LLM deployments.

6. Connections to Feature-Space Trojan Detection and Mitigation

The feature-space reverse-engineering approach described by Wang et al. (Wang et al., 2022) supplements FLTrojan defense at the model inspection stage:

• Both input-space and feature-space Trojans—ranging from static pixel patterns to dynamic transformations—can be detected by identifying compromised feature space hyperplanes.
• The method decomposes the model into input-to-feature ($\mathcal{A}$) and feature-to-output ($\mathcal{B}$) sub-models; Trojans are characterized via a mask ($\mathbf{m}$) and pattern ($\mathbf{t}$):

$$H = \{ \mathbf{a} \mid \mathbf{m} \odot \mathbf{a} = \mathbf{m} \odot \mathbf{t} \}$$

• Mitigation operates by flipping activation values in the compromised neuron subset, disrupting the Trojan’s predictive influence without diminishing benign accuracy.

This approach is directly applicable to FLTrojan scenarios: it requires only a limited clean reference set and is agnostic to model architecture and attack injection details. The reported detection accuracy (93%) and dramatic ASR reduction (to 0.26%) attest to its efficacy across federated and supply-chain attacks.

7. Research Directions and Defensive Strategies

The findings associated with FLTrojan prompt several research avenues:

• Quantifying round-level exposure risks: Further analysis is necessary to establish upper bounds on privacy leakage via token exposure metrics in federated LLMs.
• Enhancing aggregation security: Design of aggregation protocols that obfuscate or regularize intermediate model snapshots to thwart selective weight-based inference.
• Advanced defenses targeting layer selectivity: New mechanisms that dynamically monitor and constrain $L_{\star}$’s weight dynamics, potentially leveraging representation dispersion or adversarial regularization, may mitigate FLTrojan risk.
• Realistic benchmarking: Adopting attack/defense evaluations leveraging privacy canaries, natural secret patterns, and diverse prompt perturbations is essential for credibility.

The continued integration of LLMs into privacy-sensitive FL systems mandates heightened diligence, cross-disciplinary collaboration, and proactive strengthening of privacy protection mechanisms.