GradOT: Gradient-Preserving Offsite Tuning

• GradOT is a training-free, gradient-preserving offsite-tuning framework for LLMs that compresses private middle layers into an emulator to enable safe offsite adapter fine-tuning.
• It employs two compression mechanisms—Dynamic Rank Decomposition (DRD) for attention layers and Selective Channel Pruning (SCP) for MLP blocks—to degrade emulator performance while maintaining adapter gradient consistency.
• The pipeline separates model-owner computations from data-owner fine-tuning, significantly reducing compute time compared to traditional training-based methods while achieving a practical privacy-utility balance.

GradOT is a training-free, gradient-preserving offsite-tuning framework for LLMs that addresses privacy and compute constraints arising when a data owner adapts a model owned by a model owner without co-locating private data and original model weights. The method instantiates offsite-tuning by compressing the original model’s frozen middle layers into a weaker emulator, sending that emulator together with adapter modules to the data owner for offsite fine-tuning, and then reinserting the fine-tuned adapters into the original model. Its defining principle is to preserve the gradients seen by the trainable adapters while intentionally degrading emulator performance, thereby balancing plug-in utility against model privacy (Yao et al., 6 Jul 2025).

1. Offsite-tuning setting and privacy model

GradOT formalizes a three-part decomposition of the model. The original LLM is written as $f(x;\theta)$, with full-capacity parameters $\theta$ and private middle layers. The emulator is a lossy-compressed weaker model $g(x;\phi)$ derived from the original middle layers. The adapters are the trainable top and bottom layers, or more generally PEFT modules, parameterized by $\psi$ and fine-tuned offsite while the middle layers remain frozen. In the paper’s notation, the original model is decomposed as

$f(x;\theta)=A_2 \circ E \circ A_1,$

and the emulator as

$g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$

where $\hat{E}=F(E)$ is a compressed version of the frozen middle-layer block $E$ under a lossy compression operator $F$ (Yao et al., 6 Jul 2025).

The threat model is explicitly asymmetric. The data owner is assumed to be honest-but-curious and should not be able to recover the original middle-layer weights from the shipped emulator. The model owner cannot access the data owner’s private downstream data. The protected objects are therefore both the original middle-layer parameters of the LLM and the downstream private data used for adapter fine-tuning. Privacy is enforced empirically through lossy compression and a training protocol that fine-tunes only adapters on the emulator; the model owner computes gradient statistics on a separate support dataset, and the downstream private data never leaves the data owner’s site.

A central distinction in GradOT is between empirical privacy and formal privacy. The framework provides privacy through a performance gap between the fine-tuned emulator and the plug-in model obtained after reinserting adapters into the original model. The paper explicitly states that there are no formal differential privacy guarantees; privacy is operationalized through optimization criteria that increase emulator loss while preserving adapter gradients. This places GradOT within a pragmatic offsite-tuning regime rather than a cryptographically or DP-certified one.

2. Optimization objective and gradient-preserving criterion

Offsite-tuning is posed as the problem of selecting a compression operator $F$ that preserves the performance of the plug-in model while degrading the standalone emulator. Given a downstream dataset $\theta$0 and task loss

$\theta$1

the three relevant models are the plug-in model

$\theta$2

the direct full fine-tune reference

$\theta$3

obtained by training adapters with the original model, and the emulator fine-tuned model

$\theta$4

The proposed objective is

$\theta$5

subject to $\theta$6 minimizing $\theta$7 and $\theta$8. The first term pushes plug-in performance toward direct fine-tuning, while the second term enlarges the gap between plug-in and emulator performance to strengthen privacy (Yao et al., 6 Jul 2025).

GradOT reframes this objective through adapter-gradient preservation. If $\theta$9 parameterizes the adapters $g(x;\phi)$0, the formal criterion is

$g(x;\phi)$1

The premise is that adapters trained on the emulator should encounter gradients close to those that would have arisen under the original model, so that once the adapted modules are plugged back into the original model, performance remains close to direct adapter fine-tuning.

To make this tractable for very large models, the paper introduces a layerwise surrogate. Writing the weights as $g(x;\phi)$2 and perturbing the middle layers by $g(x;\phi)$3 so that $g(x;\phi)$4 for $g(x;\phi)$5, the loss change under small perturbations is approximated by the total differential:

$g(x;\phi)$6

The gradient discrepancy is approximated through a first-order Taylor expansion:

$g(x;\phi)$7

where $g(x;\phi)$8 is a Hessian block. This yields the core surrogate

$g(x;\phi)$9

The corresponding Gradient-preserving Compression Score (GCS) for a candidate perturbation $\psi$0 is

$\psi$1

Minimizing GCS selects compressions that keep adapter gradients stable while increasing emulator loss.

The second-order blocks are not formed explicitly. GradOT uses a Kronecker-factored approximation for the Fisher, and hence Hessian, blocks of each linear layer:

$\psi$2

where $\psi$3 are inputs to layer $\psi$4, $\psi$5 are output gradients, and $\psi$6 is the Kronecker product. This reduces the cost of Hessian estimation from full $\psi$7 structure to per-layer blocks. The analysis remains first-order and local: it assumes small perturbations, uses KFAC approximations, and does not provide convergence guarantees for downstream adapter training, although empirical loss-landscape visualizations indicate that plug-in optima lie in the same basin and at comparable loss levels to full-model adapter fine-tuning.

3. Compression mechanisms: DRD and SCP

GradOT instantiates GCS-guided compression with two training-free operators specialized to standard Transformer blocks. Multi-Head Attention layers are compressed by Dynamic Rank Decomposition (DRD), and MLP blocks are compressed by Selective Channel Pruning (SCP). The shared logic is that admissible low-rank factors or pruned channels are scored by GCS, and the lowest-score choices are retained (Yao et al., 6 Jul 2025).

For a linear attention weight $\psi$8, DRD begins from an SVD,

$\psi$9

A subset $f(x;\theta)=A_2 \circ E \circ A_1,$0 of singular components is selected, with cardinality

$f(x;\theta)=A_2 \circ E \circ A_1,$1

and the compressed matrix is formed as

$f(x;\theta)=A_2 \circ E \circ A_1,$2

The subset $f(x;\theta)=A_2 \circ E \circ A_1,$3 is chosen to minimize $f(x;\theta)=A_2 \circ E \circ A_1,$4 subject to the desired rank. In practice, each singular component is scored via the GCS of its individual contribution $f(x;\theta)=A_2 \circ E \circ A_1,$5, and the top-$f(x;\theta)=A_2 \circ E \circ A_1,$6 lowest-score components are retained. The method additionally preserves the top $f(x;\theta)=A_2 \circ E \circ A_1,$7 ranks to avoid overly large perturbations.

For MLP blocks, the paper considers

$f(x;\theta)=A_2 \circ E \circ A_1,$8

with $f(x;\theta)=A_2 \circ E \circ A_1,$9 and $g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$0, where $g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$1. SCP chooses an intermediate channel subset $g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$2 with $g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$3 and prunes the remaining channels, yielding

$g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$4

The selection criterion minimizes

$g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$5

subject to the channel-count constraint. Practical scoring is again per-component: the paper computes per-channel GCS contributions and keeps the top-$g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$6 lowest-score channels.

These two operators are complementary in the empirical analysis. SCP alone improves plug-in utility but yields a small privacy gap $g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$7, whereas DRD alone increases $g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$8 but can slightly degrade plug-in performance. Their combination, DRD+SCP, produces the best privacy gap and utility under higher compression. This suggests that the attention and MLP compressions contribute differently to the privacy–utility frontier, with GCS functioning as the common selection principle.

4. Pipeline and computational profile

The GradOT pipeline separates model-owner computation from data-owner adaptation. On the model-owner side, the method is training-free: a one-time statistical pass is run over a support dataset $g(x;\phi)=A_2 \circ \hat{E} \circ A_1,$9, candidate compressions are scored, and the emulator is assembled. The data owner then fine-tunes only adapters on private downstream data using the shipped emulator. Finally, the model owner reinserts the returned adapters into the original model to obtain the plug-in model $\hat{E}=F(E)$0 (Yao et al., 6 Jul 2025).

The preparation stage uses a support dataset drawn from related tasks; the paper gives BoolQ, TriviaQA, and CoPA as examples, with a total of 1500 samples. Forward and backward passes over $\hat{E}=F(E)$1 accumulate per-layer first-order gradients $\hat{E}=F(E)$2 and KFAC block approximations

$\hat{E}=F(E)$3

These statistics are computed once per model and reused for compression. Memory is managed by grouping layers, and statistics are computed linearly per layer.

Compression then proceeds layer by layer over the middle layers. For MHA weights, DRD is applied with target ratio $\hat{E}=F(E)$4 and trade-off $\hat{E}=F(E)$5. For MLP weights, SCP is applied with target ratio $\hat{E}=F(E)$6 and trade-off $\hat{E}=F(E)$7. The resulting emulator $\hat{E}=F(E)$8, together with the initial adapters $\hat{E}=F(E)$9 and $E$0, is sent to the data owner. The data owner fine-tunes only $E$1 on the private downstream data using

$E$2

The returned adapters are then plugged into the original model:

$E$3

Evaluation reports zero-shot performance, emulator fine-tuning performance, plug-in performance, and the privacy gap

$E$4

The computational profile is a major design point. Statistics computation takes approximately 8 minutes for OPT-1.3B and 97 minutes for LLaMA-7B on a single A100 80G; compression itself takes approximately 1 minute for OPT-1.3B and 8 minutes for LLaMA-7B. The paper contrasts this with training-based OT baselines: OT requires about 60 hours for a 1.3B model, and ScaleOT requires about 5 hours for a 1.3B model and about 57 hours for a 7B model. Complexity is driven by $E$5 backward passes plus KFAC accumulation for statistics, by SVDs in DRD for attention weights, and by per-channel scoring in SCP. The paper does not provide a public code link, but it states that the formulas, procedures, hyperparameters, and implementation notes are sufficient for reproduction.

5. Experimental regime and reported results

The reported experiments cover OPT-1.3B, OPT-6.7B, LLaMA-7B, and LLaMA-13B, evaluated with lm-eval-harness on multi-choice QA, closed-book QA, sentence completion, and WebQs. The listed tasks are OpenBookQA, PIQA, SciQ, RACE, ARC-Easy, ARC-Challenge, HellaSwag, and WebQs. The adapters are the first and last two layers, corresponding to approximately 10% of parameters, while the middle layers are compressed into the emulator. For 1.3B models, the compression ratios are $E$6, leaving the emulator with approximately 60% of parameters; for 7B models, they are $E$7, leaving approximately 70% of parameters. Adapter training uses AdamW with learning-rate grid $E$8, and score trade-offs use $E$9, with recommended defaults $F$0 and $F$1 (Yao et al., 6 Jul 2025).

Model	GradOT results	Comparative context
OPT-1.3B	Emulator ZS 27.3, Emulator FT 45.0, Plug-in 49.8, $F$2	Full model: ZS 42.5, FT 49.9; ScaleOT Plug-in 49.9 with weaker privacy; CRaSh Plug-in 48.4, $F$3; OT† Plug-in 46.5
OPT-6.7B	Plug-in 55.4	Comparable to ScaleOT 55.5; slightly above CRaSh 55.3
LLaMA-7B	Plug-in 58.2	Equal to ScaleOT 58.2; above CRaSh 57.1 and OT† 54.1
LLaMA-13B	Plug-in surpasses OT† on OBQA 36.2 vs 34.4, ARC-E 77.8 vs 76.5, ARC-C 50.1 vs 43.8, WebQs 48.1 vs 35.4	Reported as per-task examples

On OPT-1.3B, GradOT’s plug-in score nearly matches full-model adapter fine-tuning while the emulator zero-shot score is the lowest among baselines, which the paper interprets as the strongest privacy setting among the compared methods. On LLaMA-7B, the paper states that Emulator FT and ZS satisfy the OT conditions and that GradOT achieves excellent utility with strong privacy. The OT conditions used in the practical guidance are $F$4 and $F$5.

The loss-landscape analysis is used to interpret these results. Plug-in optima for GradOT and full-model adapter fine-tuning lie in the same basin and at similar loss levels, whereas emulator fine-tuning optima lie far apart. This is presented as evidence that the compression preserves the adapter-training geometry relevant for plug-in performance while separating the weaker emulator from the original model in terms of recoverable performance.

6. Ablations, practical guidance, and limitations

The ablation studies attribute the method’s behavior to both the score and the choice of support data. Compression without GCS severely reduces plug-in performance and the privacy gap $F$6, indicating that simple structural compression is insufficient for the intended offsite-tuning objective. Using a multi-task support dataset composed of BoolQ, TriviaQA, and CoPA yields better plug-in performance than using generic data such as WikiText or the downstream data directly; the paper states that this suggests improved generalization (Yao et al., 6 Jul 2025).

Sensitivity analyses describe the privacy–utility control surface. Increasing $F$7 lowers emulator fine-tuning performance, which improves privacy, while plug-in performance remains stable up to a point; the recommended defaults $F$8 and $F$9 are reported to balance this trade-off. For compression ratio $F$0, GradOT consistently preserves privacy through an approximately constant $F$1, and with $F$2, plug-in performance can match or exceed full-model adapter fine-tuning. The practical guidance therefore recommends support datasets of about 1500 examples, compression settings around $F$3–$F$4–$F$5, and the first and last two layers as adapters.

Several limitations are explicit. The method is designed for Transformer LLMs with MHA and MLP blocks and may not apply to non-Transformer architectures such as Mamba. Its theoretical analysis depends on small-perturbation first-order expansions and KFAC block approximations, so large perturbations or mismatched support data can degrade gradient preservation. Most importantly, the privacy notion is empirical rather than formal: the method reports performance gaps rather than differential privacy bounds. Any interpretation of GradOT as a formal privacy mechanism would therefore be inaccurate.

The extension directions listed in the paper remain consistent with the GCS framework. They include alternative compression methods guided by GCS, such as feature-based low-rank methods via Bayesian optimization; extension of GCS to multimodal LLMs with vision or audio blocks; and combination of GradOT with stronger privacy mechanisms such as differential privacy or secure computation. A plausible implication is that GradOT’s main contribution is not only a specific compression recipe, but a reusable optimization criterion for selecting privacy-preserving emulator perturbations under offsite-tuning constraints.