From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models

Abstract: LLMs are increasingly deployed in privacy-sensitive domains, where users must balance the risk of data exposure through external APIs against the high computational cost of local deployment. Split learning has therefore emerged as a promising paradigm for LLM fine-tuning and inference under limited local resources. However, it introduces new privacy risks. Prior work primarily studies leakage of private input prompts, typically via inversion attacks on intermediate representations, while the potential for sensitive information leakage through generative response outputs remains largely unexplored. In this work, we unveil novel vulnerabilities of Split-LLM by presenting Patched Model Inversion with Dual-Sided Initialization (PIDI), a two-stage attack that simultaneously targets both private input prompts and output responses in Split-LLM settings. It combines dual-sided initialization with a patched inversion strategy to tackle long sequences, substantially outperforming prior inversion methods. To counter threats from both sides, we further propose the Adapter-based DualGuard with Mutual Information Defense (ADMI), which integrates an adapter-based local warmup strategy and mutual information regularization to provide a strong empirical privacy protection with minimal impact on task performance. Extensive experiments across diverse tasks and models demonstrate that ADMI effectively defends against PIDI and other state-of-the-art inversion attacks. Our code is publicly available at https://github.com/FLAIR-THU/VFLAIR-LLM.

• The paper introduces a dual-sided reconstruction attack (PIDI) that targets both input prompts and generated responses in split LLM pipelines.
• It details a two-stage inversion method using dual-sided initialization and patch-wise optimization, achieving BLEU attack performance near or above 0.85.
• The proposed ADMI defense employs adaptive regularization and local parameter adaptation to block leakage while preserving main task utility.

Dual-Sided Data Leakage and Defense in Split LLMs

Introduction and Motivation

The deployment of LLMs in privacy-sensitive domains necessitates a careful balance between resource constraints and data confidentiality. Split learning (SL) addresses resource limitations by dividing an LLM into segments that run locally and on the cloud. The head-body-tail (HBT) paradigm assigns the model head and tail to the data holder, while the main body remains on the server. While this approach obscures direct data transmission, it introduces a new surface for privacy attacks, particularly via intermediate representations. Notably, previous investigations predominantly focused on reconstructing private input prompts, largely neglecting response-side leakage.

This work systematically investigates dual-sided data leakage: both input prompt and generated response can be reconstructed from the activations in Split-LLM pipelines. The study provides two primary contributions: (1) the proposal of a novel attack—Patched Model Inversion with Dual-Sided Initialization (PIDI)—which enables joint reconstruction of private inputs and responses; and (2) the design of Adapter-based DualGuard with Mutual Information Defense (ADMI), a dual-sided, utility-preserving defense that leverages regularization and local parameter adaptation.

Dual-Sided Threat Model in Split-LLM

The threat model assumes an honest-but-curious server with access to the original pre-trained model slices and the fine-tuned body, as well as both the transmitted head and body activations during inference. The adversary aims to reconstruct the sequence $X$ (input) and $Y$ (response) based solely on these intermediate activations, without access to backward gradients—differing from standard training-time attacks.

Figure 1: Dual Data Reconstruction Attack Threat Model detailing the information flow and attack surface within a Split-LLM pipeline.

The server can exploit the autoregressive nature of generation—where outputs are fed back as inputs—accumulating side-channel information from both model ends. This insight motivates the need for attacks (and corresponding defenses) that span both the prompt and the generative response.

Patched Model Inversion with Dual-Sided Initialization (PIDI)

PIDI is a two-stage attack aligned with the dual-ended nature of leakage:

1. Dual-Sided Initialization (DSI):
   • Response branch: Uses model completion, leveraging the close proximity ("not-too-far" property) of the fine-tuned and pre-trained model tail. By feeding the observed model body output $T$ into the pre-trained tail, an initial estimate of the response is reconstructed.
   • Input branch: Employs SIP (Semi-white-box Inversion Paradigm), which utilizes a small auxiliary dataset to train a mapping from head activations $H$ to input token sequences.
2. Patched Model Inversion (PMI):
   • The joint initialization is refined using a patch-wise gradient-based inversion method. By dividing long sequences into manageable patches, the method overcomes the scalability challenge of traditional inversion, jointly optimizing input and response estimations patch-by-patch, followed by a full joint pass.

     Figure 2: The Two-Stage Attack Methodology of PIDI, comprising dual-sided initialization and patch-wise model inversion for reconstructing both prompt and response.

Empirically, PIDI consistently outperforms baselines across Llama and Qwen Split-LLMs, achieving BLEU-based attack performance (AP) near or above 0.85 on challenging datasets, and maintaining high accuracy even as sequence length grows (see supporting results and supplementary ablations).

ADMI: Adapter-based DualGuard with Mutual Information Defense

To counter dual-sided leakage, the ADMI pipeline integrates two orthogonal defenses:

1. Mutual Information Regularization ($\mathcal{L}_{\text{MI}}$): This minimizes the mutual information between input embeddings and transmitted head activations using a variational information bottleneck, increasing the difficulty of prompt inversion attacks.
2. Model Distance Regularizer ($\mathcal{L}_\text{D}$): By maximizing divergence between the fine-tuned and pre-trained tails on the body activations, this regularizer disrupts the "not-too-far" property, mitigating model completion attacks.

Training is executed in two phases:

• Adapter-based Local Warm-up: An additional encoder-decoder adapter is inserted. The adapters and the model head/tail are updated while freezing the body, enabling the system to break initialization proximity via localized adaptation, guided by $\mathcal{L}_\text{D}$.
• Full Training: All segments are unfrozen, and both $\mathcal{L}_{\text{MI}}$ and $\mathcal{L}_\text{D}$ are simultaneously enforced, leveraging the adapters and bottleneck to preserve utility.

  Figure 3: Defense Pipeline of ADMI. The VIB bottleneck and adapter modules are positioned to disrupt information leakage at both model ends.

ADMI's use of adaptive weighting ensures training stability as the defense strength increases, and the architectural modification ensures robust disruption of leakage channels without significant performance loss.

Experimental Results

Experiments span three LLMs (Llama3.2-3B, Llama3-8B, Qwen2.5-7B) and three domain-specific datasets (general Q&A, medical, finance). Evaluation metrics include main task (utility) performance, BLEU-based attack performance (AP), and a composite Defense Capability Score (DCS).

Key Findings:

• Attack Effectiveness: PIDI surpasses all baselines, maintaining high attack performance even for lengthy sequences. Patch-size ablation confirms an optimal trade-off between AP and computational overhead with moderate patch lengths.

  Figure 4: Changing trend of AP and Attack time with respect to Patch Length $l$ (Evaluated with Llama3.2-3B on Fin Dataset).

• Defense Robustness: ADMI achieves near-zero AP with negligible utility degradation across all model-dataset configurations, consistently attaining the highest DCS.

  Figure 5: DCS ranking with Different $Y$0 values (Evaluated with Llama3-3B on Fin Dataset).

• Privacy-Utility Trade-off: Unlike perturbation-based baselines (DPForward, SP), which incur marked utility loss, ADMI preserves main task performance while neutralizing dual-ended leakage.

Theoretical and Practical Implications

This work underscores the necessity of considering both input and output reconstruction when evaluating privacy in Split-LLM pipelines. The dual-sided threat model extends the adversarial landscape—PIDI concretely demonstrates risks in practical deployments, especially in domains like finance and medicine. The formulation and empirical validation of ADMI clarify that head-only or tail-only defenses are insufficient; joint architectural and regularization strategies are required.

Beyond immediate privacy risks, these results imply that privacy-preserving split learning requires careful design of both protocol and parameter adaptation mechanisms. Architectural interventions such as adaptive adapters and stochastic bottlenecks are essential complements to standard regularization.

Future Directions

• Efficient Dual-Sided Defenses: Investigating lighter-weight mechanisms that match ADMI's dual-sided protection but incur lower computational and latency overhead.
• Generalization to Other Modalities: Extending the dual-leakage framework to multi-modal split models (e.g., vision-language, speech).
• Dynamic Adaptation: Defense strategies that adapt in real time to evolving threat surfaces without retraining the full Split-LLM pipeline.

Conclusion

This paper demonstrates that Split-LLMs are vulnerable to potent dual-sided reconstruction attacks. The PIDI attack achieves robust input and response recovery via a synergistic initialization and patch-wise inversion pipeline. The ADMI defense framework offers strong empirical resistance via architectural adapters and joint regularization, providing favorable privacy-utility trade-offs across models and domains.

These findings mark a shift from single-end to dual-end privacy considerations in SL-LLM research and highlight the effectiveness of hybrid architectural and information-theoretic defenses. Despite increased computational demands, ADMI points the way toward practical privacy-preserving Split-LLM deployments in sensitive, resource-constrained, real-world settings.

---

Reference:

"From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split LLMs" (2606.14210)