VFLAIR-LLM: Privacy-Preserving Split LLM

• VFLAIR-LLM is a split learning framework for LLMs that partitions models via Head–Tail and Head–Body–Tail schemes to balance privacy, computation, and communication.
• It benchmarks diverse NLP tasks across 18 datasets, systematically evaluating the trade-offs between task performance and privacy leakage.
• The framework integrates advanced attack and defense modules that mitigate inversion and label inference risks while maintaining strong performance.

VFLAIR-LLM is an extensible and lightweight split learning framework for LLMs that enables privacy-preserving LLM inference and fine-tuning in resource-constrained environments. Developed to address data privacy concerns and the computational demands of private LLM deployments, VFLAIR-LLM provides two core model partition strategies, supports a broad array of benchmarks, and includes comprehensive modules for simulating and countering information leakage attacks. Its codebase and pre-configured benchmarks are available at https://github.com/FLAIR-THU/VFLAIR-LLM, facilitating real-world adoption and reproducible evaluation (Gu et al., 5 Aug 2025).

1. Model Partitioning Schemes in VFLAIR-LLM

VFLAIR-LLM formalizes two principal approaches to partitioning pretrained LLMs with $n$ transformer layers for split learning: Head–Tail (HT) and Head–Body–Tail (HBT). These allow flexible trade-offs between privacy, communication, and computational overhead.

Head–Tail (HT) Split

The model is divided into:

• Data Party (Client): $M_{\text{head}}$ (embedding + first $n_\text{head}$ layers)
• Model Party (Server): $M_{\text{tail}}$ (remaining $n_\text{tail} = n - n_\text{head}$ layers + output head)

Data flow:

• Forward: $H_1 = M_{\text{head}}(X)$, $\hat{Y} = M_{\text{tail}}(H_1)$
• Backward: $G_1 = \partial L/\partial H_1$; $G_1$ used to update $M_{\text{head}}$ via backpropagation

Resource metrics per sample:

• Client compute: $M_{\text{head}}$0
• Communication: $M_{\text{head}}$1

Head–Body–Tail (HBT) Split

Here, the model is divided into:

• Data Party (Client): $M_{\text{head}}$2 (first $M_{\text{head}}$3 layers) and $M_{\text{head}}$4 (last $M_{\text{head}}$5 layers)
• Model Party (Server): $M_{\text{head}}$6 (middle $M_{\text{head}}$7 layers; $M_{\text{head}}$8)

Data flow:

• Forward: $M_{\text{head}}$9; $n_\text{head}$0; $n_\text{head}$1
• Backward: $n_\text{head}$2 updates $n_\text{head}$3 and $n_\text{head}$4, backward $n_\text{head}$5 updates $n_\text{head}$6

Resource metrics:

• Client compute: $n_\text{head}$7
• Communication: $n_\text{head}$8 (client→server) and $n_\text{head}$9 (server→client) per sample

Compute–Communication–Privacy Trade-offs

Client compute $M_{\text{tail}}$0 increases linearly with added head layers. Communication cost $M_{\text{tail}}$1. Privacy leakage—measured by attack performance (AP)—decreases empirically as $M_{\text{tail}}$2 increases, but this incurs higher local compute and communication [(Gu et al., 5 Aug 2025), Figure 1].

2. Task and Dataset Scope

VFLAIR-LLM benchmarks three major NLP task categories across 18 datasets, ensuring coverage of both practical and adversarially-relevant scenarios:

Task Type	Datasets (examples)	Primary Metrics
Classification/Regression	SST-2, CoLA, MRPC, MNLI, QNLI, RTE, Yelp, STS-B	Accuracy, Pearson’s $M_{\text{tail}}$3
Span QA	SQuAD v1.1	EM, F1
Generation/CausalLM/QA	Lambada, Alpaca, Dolly, CodeAlpaca, MATH, GSM8K, TextVQA	Rouge, CodeBLEU

These datasets represent a diverse spectrum, including discrete-label and sequence-level outputs, as well as specialty domains (code, math, vision-language) [(Gu et al., 5 Aug 2025), Table 3]. The inclusion facilitates comprehensive assessment of privacy–utility trade-offs in practical and high-risk adaptation settings.

3. Attack and Defense Framework

VFLAIR-LLM implements standard modules for five adversarial attacks and nine defense mechanisms, encompassing both perturbation- and learning-based approaches.

Attack Methods

• Model Inversion Attacks (MIA): Aim to reconstruct $M_{\text{tail}}$4 from $M_{\text{tail}}$5.
  • VMI (Vanilla Model Inversion): $M_{\text{tail}}$6
  • RMI: continuous relaxation and regularization, $M_{\text{tail}}$7
  • BiSR: two-phase, semi-white-box, achieves highest empirical AP
  • Attack Performance (AP): recall/exact match rate of reconstructed tokens
• Label Inference Attacks (LIA): Aim to infer $M_{\text{tail}}$8 from $M_{\text{tail}}$9
  • BLI (Batch-level Label Inversion): gradient-statistics to label inversion
  • NS (Norm Scoring): infer $n_\text{tail} = n - n_\text{head}$0 by thresholding $n_\text{tail} = n - n_\text{head}$1

Defense Strategies

• Perturbation-Based (6): Differential Privacy (DP), Sparsification (SP), token-level MLDP (SanText, CusText, RanText), Split-N-Denoise (SnD)
  • Hyperparameters: $n_\text{tail} = n - n_\text{head}$2 for DP/MLDP ($n_\text{tail} = n - n_\text{head}$3–$n_\text{tail} = n - n_\text{head}$4); $n_\text{tail} = n - n_\text{head}$5 for SP ($n_\text{tail} = n - n_\text{head}$6–$n_\text{tail} = n - n_\text{head}$7\%)
• Learning-Based (3):
  • MID (Mutual Information Defense): bottleneck $n_\text{tail} = n - n_\text{head}$8, minimize $n_\text{tail} = n - n_\text{head}$9, $H_1 = M_{\text{head}}(X)$0
  • AT (Adversarial Training): privatizer $H_1 = M_{\text{head}}(X)$1 adversarial to inverter $H_1 = M_{\text{head}}(X)$2, tradeoff parameters $H_1 = M_{\text{head}}(X)$3
  • TO (TextObfuscator): embedding perturbations clustered; $H_1 = M_{\text{head}}(X)$4

4. Empirical Benchmarks and Insights

Benchmarking combines main task performance (MP: accuracy, Rouge, CodeBLEU, EM, etc.), inversion/attack performance (AP), and composite defense capability scores (DCS).

Fine-Tuning Regimes

• Full-Vanilla, Full-LoRA, Local-Vanilla, Local-LoRA are evaluated
• Full-LoRA halves training epochs on SST-2 (BERT) for matched accuracy and gives $H_1 = M_{\text{head}}(X)$5 compute reduction on GPT-2/Mistral with only $H_1 = M_{\text{head}}(X)$6\% MP drop
• Local tuning benefits from HBT as it allows more trainable parameters client-side

Defense Efficacy

• Without defense, BiSR AP $H_1 = M_{\text{head}}(X)$7 0.30 (SST-2), 0.15 (Alpaca), 0.12 (GSM8K)
• Perturbation methods reduce both utility (MP) and information leakage (AP) with stronger noise
• Learning-based defenses (MID, AT) consistently achieve best privacy–utility trade-off: MID yields lowest AP for $H_1 = M_{\text{head}}(X)$8; AT is close
• On SQuAD v1.1, MID achieves composite C-DCS $H_1 = M_{\text{head}}(X)$9 (SST-2), $\hat{Y} = M_{\text{tail}}(H_1)$0 (CoLA); AT achieves $\hat{Y} = M_{\text{tail}}(H_1)$1; DP or SP require $\hat{Y} = M_{\text{tail}}(H_1)$2 or $\hat{Y} = M_{\text{tail}}(H_1)$3 to approach, but with $\hat{Y} = M_{\text{tail}}(H_1)$4 MP loss

Partitioning and Privacy

• Increasing $\hat{Y} = M_{\text{tail}}(H_1)$5 ($\hat{Y} = M_{\text{tail}}(H_1)$6): reduces AP by $\hat{Y} = M_{\text{tail}}(H_1)$7 at $\hat{Y} = M_{\text{tail}}(H_1)$8 MP loss, but increases local computation proportionally
• Learning-based MID/AT atop LoRA fine-tuning provides optimal privacy–utility; recommended: MID $\hat{Y} = M_{\text{tail}}(H_1)$9, AT $G_1 = \partial L/\partial H_1$0, $G_1 = \partial L/\partial H_1$1

5. Best Practices and Deployment Recommendations

• Partitioning: HT is straightforward; HBT preferred if the server must not access ground-truth labels or outputs
• Layer assignment: $G_1 = \partial L/\partial H_1$2 (about $G_1 = \partial L/\partial H_1$3 of model) gives a robust privacy–utility compromise
• Defenses: MID or AT on LoRA-finetuned splits optimally balances MP and AP; DP/SP are viable for lightweight protection with careful tuning, but significant MP decrease occurs for strong privacy settings
• Resource management: Reduce $G_1 = \partial L/\partial H_1$4 or use Local tuning if client GPU memory is critical; batch size $G_1 = \partial L/\partial H_1$5, LoRA LR $G_1 = \partial L/\partial H_1$6, early stopping is recommended
• Operational mode: Standalone mode is preferred for research (e.g., Alpaca on Llama3-8B: $G_1 = \partial L/\partial H_1$7 tokens/s standalone vs. $G_1 = \partial L/\partial H_1$8 tokens/s distributed)

6. Scope, Limitations, and Reproducibility

VFLAIR-LLM is designed for flexible, privacy-conscious LLM deployment under real-world resource and threat constraints. Benchmarks and code are publicly available, supporting customization for model splits, attacks, defenses, and hyperparameter sweeps. The platform’s task, dataset, and threat coverage enable rigorous experimental comparisons across the privacy–efficiency spectrum (Gu et al., 5 Aug 2025). The framework does not extend to scenarios outside the split learning paradigm, nor to privacy threats not captured by inversion or label inference attacks as modeled in the included modules. A plausible implication is that privacy guarantees are tightly coupled to the implemented attack/defense taxonomy.

For implementation and experiment recipes, practitioners are referred to the open-source codebase.