Offsite-Tuning: Privacy-Preserving Transfer Learning

• Offsite-Tuning is a transfer learning framework that splits a foundation model into a trainable adapter and a compressed emulator, enabling local fine-tuning while protecting proprietary data.
• It employs a sandwich design with trainable bottom and top layers and a frozen middle block, reducing computational and memory burdens compared to full model fine-tuning.
• The framework offers strong training-data locality and efficiency, though it presents inference privacy tradeoffs that have spurred variants like GuardedTuning and ScaleOT.

Offsite-Tuning (OT) is a transfer-learning framework for foundation models in which the model owner sends a light-weight adapter and a lossy compressed emulator of the model’s frozen core to the data owner, the data owner fine-tunes only the adapter on private downstream data, and the adapted adapter is returned and plugged back into the full model. The design is intended to avoid exposing full proprietary weights or raw downstream data while reducing the computation and memory burden of adaptation. Subsequent work has treated OT as a canonical privacy-preserving fine-tuning architecture, both as a baseline and as a substrate for inference-protection, training-free compression, and federated extensions (Xiao et al., 2023, Shi et al., 8 Jan 2025).

1. Origin, motivation, and conceptual scope

OT was introduced for settings in which the model owner and the data owner cannot colocate model and data without violating either intellectual-property constraints or data-privacy constraints. In the motivating scenario, full-model fine-tuning is unattractive because it either requires the model owner to reveal the full model or requires the data owner to reveal raw downstream data. OT addresses this by exposing only a degraded model fragment to the data owner and by confining downstream training to that side of the trust boundary (Xiao et al., 2023).

Conceptually, OT occupies a distinct position relative to several adjacent paradigms. Standard parameter-efficient fine-tuning methods such as Adapters, LoRA, Prefix, and BitFit reduce the number of trainable parameters, but they still assume local access to the full base model. Federated learning protects data locality across clients, but clients typically hold the full global model and therefore do not protect model IP. Split learning partitions computation across parties but requires online exchange of intermediate activations, and often gradients, throughout training. OT instead uses offline shipment of a compressed emulator and trains locally without per-step client–server interaction during training (Xiao et al., 2023, Shi et al., 8 Jan 2025).

Later privacy-preserving fine-tuning surveys and design papers formalize OT as one branch of a broader taxonomy. Within that taxonomy, OT is the client-local architecture with an emulator: the client receives adapters plus a backbone emulator, performs local fine-tuning, and eventually returns the tuned adapter, while the server retains the full proprietary backbone. This positioning is important because later systems often preserve OT’s offline training pattern while modifying inference behavior, compression strategy, or privacy mechanism (Shi et al., 8 Jan 2025).

2. Core architecture and protocol

The original OT formulation decomposes the foundation model into a trainable adapter and a frozen core. Using the notation of the original work,

$$\mathcal{M} = [\mathcal{A}, \mathcal{E}], \qquad \mathcal{M} = \mathcal{A}_1 \circ \mathcal{E} \circ \mathcal{A}_2,$$

where $\mathcal{A}_1$ is a bottom adapter, $\mathcal{A}_2$ is a top adapter, and $\mathcal{E}$ is the frozen middle block. The data owner receives

$$[\mathcal{A}, \mathcal{E}^*] = [\mathcal{A}_1, \mathcal{E}^*, \mathcal{A}_2],$$

where $\mathcal{E}^*$ is a lossy compressed emulator of $\mathcal{E}$ (Xiao et al., 2023).

The downstream training phase is local to the data owner. The emulator remains frozen, and only the adapter is optimized. For language modeling, the training objective is

$$\min_{\mathcal{A}} \mathbb{E}_{(x, y) \sim \mathcal{D}_{\text{down}}} \left[ \mathcal{L}_{\text{LM}}\big((\mathcal{A}_2 \circ \mathcal{E}^* \circ \mathcal{A}_1)(x), y\big) \right].$$

After tuning, the data owner returns only the trained adapter $\mathcal{A}'$, and the model owner constructs the adapted full model as

$\mathcal{M}' = [\mathcal{A}', \mathcal{E}].$

In effect, the emulator is only a client-side training surrogate; the deployed model restores the original frozen middle block (Xiao et al., 2023).

A distinctive architectural choice is that the “adapter” in baseline OT is not a separate bottleneck module. It is a subset of the original Transformer layers whose parameters are allowed to change. The resulting arrangement is the “sandwich” design: trainable bottom layers, frozen and emulated middle layers, and trainable top layers. Empirically, fine-tuning both top and bottom layers was reported to yield much better transfer than tuning only top layers or only bottom layers for the same number of tunable layers (Xiao et al., 2023).

3. Emulator construction and transfer mechanism

The emulator is a deliberately weaker approximation of the frozen middle block. OT evaluated several compression strategies for $\mathcal{A}_1$0, including magnitude pruning, quantization, and layer-drop with optional distillation, and found layer-drop to work best for OT’s objectives. In the layer-drop procedure, if the middle block has layers $\mathcal{A}_1$1 and the target emulator keeps $\mathcal{A}_1$2 layers, the retained indices are chosen by uniform sampling,

$\mathcal{A}_1$3

while preserving the first and last layer of the middle block (Xiao et al., 2023).

When feasible, the model owner can distill the emulator against the original middle block using hidden-state regression: $\mathcal{A}_1$4 This is a simple MSE objective on hidden states produced after the upstream adapter. Distillation is not required by OT as a definition, but it substantially improves the quality of the emulator as a training surrogate. On GPT-2-XL with a $\mathcal{A}_1$5 split, plug-in perplexity improved from 19.69 without distillation to 14.94 with distillation; on OPT-1.3B with a $\mathcal{A}_1$6 split, plug-in perplexity improved from 14.63 to 13.15 (Xiao et al., 2023).

The intended operating regime can be summarized by three empirical inequalities: zero-shot performance should be lower than plug-in performance, emulator performance should be lower than plug-in performance, and plug-in performance should be close to full fine-tuning. The first condition shows that adaptation is useful, the second that the emulator is not an adequate substitute for the proprietary model, and the third that the adapter learned against the emulator transfers back to the full model. This transfer pattern is the defining empirical logic of OT (Xiao et al., 2023).

4. Privacy model, threat surfaces, and deployment trade-offs

OT assumes two parties operating under an honest-but-curious threat model. The model owner sees full model parameters and the returned adapter but does not see raw downstream data. The data owner sees the adapter parameters and the compressed emulator but does not see the full model. The original formulation does not provide formal cryptographic guarantees or $\mathcal{A}_1$7-DP guarantees; privacy is structural and empirical. Raw downstream data never leave the data owner’s infrastructure, and full model weights are never exposed, but the original work explicitly notes that it does not fully demonstrate the absence of model or data leakage from the emulator or adapter (Xiao et al., 2023).

Later privacy-preserving fine-tuning work sharpened this characterization by separating training-time privacy from inference-time privacy. In that taxonomy, OT is a client-local training architecture with an emulator and no per-step communication during fine-tuning. Because the server does not receive training activations or gradients, training-time data reconstruction attacks of the selected type are infeasible, and the OT baseline is reported with “Fine-tune privacy” equal to “N/A.” The same analysis also identifies a major weakness of the standard OT inference path: after training, the client uploads tuned adapters to the server and inference runs entirely on the server. Under that protocol, “Inference privacy” for offsite tuning is reported as 100 in ROUGE-L F1 on all tasks, corresponding to maximal leakage in that measurement framework. Communication cost during fine-tuning is correspondingly low: 3.5 GB per task for OPT-1.3B and 21.8 GB for OPT-6.7B (Shi et al., 8 Jan 2025).

These results define OT’s central trade-off. OT is strong on training-data locality, training-time communication reduction, and partial model-IP protection through a degraded emulator, but weak on inference privacy in its default protocol. They also expose an unresolved issue on the model side: because the client receives an emulator generated from the base model, later work notes “potentially bigger risks to server model privacy” and leaves emulator risk quantification for future investigation (Shi et al., 8 Jan 2025).

5. Empirical utility and computational profile

The original evaluations show that plug-in performance can approach full fine-tuning across language and vision models. On WikiText-2, GPT-2-XL achieved perplexity 13.58 under full fine-tuning, 14.94 for OT plug-in, and 17.64 for the fine-tuned emulator. OPT-1.3B achieved 12.52 under full fine-tuning, 13.15 for OT plug-in, and 15.54 for the fine-tuned emulator. For larger models without full fine-tuning or distillation, OT still preserved the key pattern: on OPT-6.7B, full zero-shot perplexity was 24.24, emulator zero-shot was 44.92, OT emulator perplexity was 13.48, and OT plug-in perplexity was 10.78; on BLOOM-7.1B, the corresponding values were 15.34, 78.58, 22.40, and 14.58. The plug-in model therefore improved substantially over zero-shot and outperformed the emulator alone (Xiao et al., 2023).

The same pattern extends beyond language modeling. For EVA ViT-G with a $\mathcal{A}_1$8 distilled configuration, full fine-tuning on Flowers reached 99.59%, OT plug-in reached 99.33%, and OT emulator reached 99.15%. For CLIP ViT-G with a $\mathcal{A}_1$9 distilled configuration on CIFAR-100, full fine-tuning reached 92.33%, OT plug-in 91.00%, and OT emulator 90.47%. These results support the claim that OT is not restricted to decoder-only LLMs, even though its later literature is predominantly LLM-centric (Xiao et al., 2023).

OT was also designed as an efficiency mechanism. On a single NVIDIA A6000 GPU with batch size 1 and sequence length 512, GPT-2-XL full fine-tuning ran at 957 tokens/s with 39,922 MB peak memory, whereas OT $\mathcal{A}_2$0 ran at 4,905 tokens/s with 8,966 MB, and OT + LoRA ran at 6,257 tokens/s with 7,155 MB. For OPT-1.3B, full fine-tuning ran at 1,316 tokens/s with 30,458 MB, while OT + LoRA ran at 8,525 tokens/s with 6,192 MB. The reported aggregate gains reached 6.5× speedup and 5.6× memory reduction. OT is also orthogonal to PEFT inside the adapter layers: on WikiText-2 with GPT-2-XL, OT baseline used 123M trainable parameters with plug-in perplexity 14.94, OT + LoRA used 410K trainable parameters with plug-in perplexity 14.84, OT + vanilla Adapters used about 1.65M trainable parameters with plug-in perplexity 14.99, and OT + BitFit used about 83K parameters with plug-in perplexity 18.75 (Xiao et al., 2023).

6. OT-derived variants and redesigns

One line of work preserves OT’s offline training structure while modifying inference behavior. In the GuardedTuning framework, “Offline GuardedTuning” is explicitly described as “an enhancement to offsite-tuning.” Training remains OT-like: the server provides adapter layers and an emulator constructed through uniform layer dropping from the backbone, and the client trains locally. The main redesign appears at inference time: instead of transferring tuned input and output adapters to the server, the client keeps those adapters and sends only activations to the server. To reduce reconstruction risk from those activations, Offline GuardedTuning adds a decorrelation objective,

$\mathcal{A}_2$1

where $\mathcal{A}_2$2 uses cosine distance between embeddings and activations, and $\mathcal{A}_2$3 was set to 5 in experiments (Shi et al., 8 Jan 2025).

A second line of work targets emulator construction. ScaleOT replaces uniform layer dropping with Dynamic LayerReplace, an RL-based layer-importance mechanism that jointly trains lightweight “harmonizers” to replace less important layers, and then adds Selective Rank Compression (SRC) on MHSA weights only. ScaleOT parameterizes the emulator by the number of adapted layers $\mathcal{A}_2$4, the proportion of harmonizer replacement $\mathcal{A}_2$5, and the rank-reduction ratio $\mathcal{A}_2$6. On GPT-2-XL over eight QA benchmarks, full fine-tuning averaged 49.9, OT plug-in averaged 48.3 with $\mathcal{A}_2$7, and ScaleOT with SRC averaged 49.3 with $\mathcal{A}_2$8; on LLaMA-7B, OT plug-in reached 54.1, CRaSh 57.1, ScaleOT without SRC 61.2, and ScaleOT with SRC 58.2. ScaleOT therefore reframes OT as a tunable privacy–utility continuum rather than a single emulator recipe (Yao et al., 2024).

A third line seeks to remove distillation cost while preserving OT’s transfer mechanism. GradOT formulates OT as an optimization problem that trades plug-in fidelity against emulator weakness and derives a Gradient-preserving Compression Score,

$\mathcal{A}_2$9

It then uses Dynamic Rank Decomposition for MHA and Selective Channel Pruning for MLP, guided by KFAC-style second-order approximations, to build the emulator without distillation. On OPT-1.3B, GradOT achieved plug-in average 49.8 with privacy gap $\mathcal{E}$0, matching CRaSh on the gap and nearly matching ScaleOT on plug-in performance; on LLaMA-7B, GradOT reached plug-in average 58.2. The claimed compression-time reduction is substantial: about 10 minutes for OPT-1.3B and 96 minutes for LLaMA-7B, versus about 60 hours for OT with distillation and 57 hours for ScaleOT on the 7B model (Yao et al., 6 Jul 2025).

A fourth line changes OT’s deployment assumptions rather than its emulator logic. ObfuscaTune considers offsite fine-tuning and inference on a third-party cloud provider and combines TEEs with linear obfuscation so that only a small fraction of parameters must reside inside the enclave. For GPT-2-XL, 5.2% of parameters are placed on the TEE, and the reported slowdown ranges from about 1.5× for GPT-2 small to about 4.3× for GPT-2-XL while preserving near-identical task performance. This is not emulator-based OT in the original sense, but it is presented as an OT variant for proprietary LLMs on private datasets under a different trust and infrastructure model (Frikha et al., 2024).

7. Limitations, critiques, and unresolved questions

OT’s limitations are now well defined in the literature. The original formulation explicitly leaves open the analysis of membership inference, gradient inversion, and model extraction against the emulator and adapters, and later privacy-preserving tuning work reiterates that OT offers architectural privacy rather than formal privacy guarantees. The same later work also narrows OT’s strongest privacy claim to training time and identifies inference-time exposure as a structural weakness of the default protocol, especially when inference runs centrally on the model owner’s server (Xiao et al., 2023, Shi et al., 8 Jan 2025).

A separate critique emerges in federated settings with heterogeneous client data. FedProxy argues that OT-based methods suffer from a “fundamental performance bottleneck” due to representation-capacity limitations imposed by the weak-adapter-plus-frozen-emulator architecture. In the LLaMA2-7B, 50% compression, heterogeneous setting, centralized SFT reached QA average 0.6585 and GLUE average 0.8861, whereas FedOT reached 0.4750 and 0.5933, and FedBiOT reached 0.4598 and 0.5713. This suggests that OT is most compelling when training-data locality, low communication, and model-IP protection dominate the design objective, and less compelling when the target is near-centralized performance under strongly heterogeneous federated optimization (Fan et al., 21 Apr 2026).

The remaining open problems are therefore not merely incremental. They include quantifying emulator leakage, formalizing privacy beyond performance discrepancy, improving inference privacy without abandoning OT’s offline training advantage, and understanding why top-and-bottom adapter tuning transfers so effectively through a compressed middle block. OT has evolved from a single method into a broader architectural pattern, but its central premise remains unchanged: adaptation through a degraded surrogate is possible, efficient, and often accurate, yet its privacy and performance envelope depends critically on how the emulator is constructed and how inference is ultimately deployed.