ShadowPEFT: Centralized PEFT Adaptation

• ShadowPEFT is a parameter-efficient adaptation framework that uses a centralized shadow network for layer-wise refinement of LLM hidden states.
• It employs a recurrent, state-space correction mechanism that enables independent pretraining, cross-layer knowledge sharing, and efficient edge deployment.
• Empirical evaluations demonstrate that ShadowPEFT achieves higher accuracy and lower inference overhead compared to traditional methods like LoRA and DoRA.

ShadowPEFT is a parameter-efficient fine-tuning framework designed to adapt LLMs by introducing a centralized, depth-shared "shadow" network for layer-wise refinement. Unlike existing approaches such as Low-Rank Adaptation (LoRA) or Directional Low-Rank Adaptation (DoRA), which insert trainable low-rank perturbations directly into individual frozen weights in a distributed and weight-centric manner, ShadowPEFT achieves adaptation through a single auxiliary network that provides stateful layer-space intervention. This architecture enables iterative, coordinated refinement of the backbone’s hidden states with reduced parameter count and facilitates unique capabilities such as independent pretraining, cross-layer knowledge sharing, and edge deployment in a detached mode (Li et al., 21 Apr 2026).

1. Conceptual Foundations

Conventional PEFT strategies, including LoRA, operate by freezing the transformer backbone and injecting independently trained low-rank matrices into selected layers. This yields a fragmented, local adaptation strictly in weight space. ShadowPEFT departs from this paradigm by installing a single, lightweight shadow module that is invoked at each base layer across the backbone. At every layer $\ell$, the shadow module maintains a shadow state $s^{(\ell)}$ in parallel with the backbone’s activations. For each step, it computes a correction term based on the discrepancy between the backbone activation and its own state, injects a low-rank-projected adjustment into the backbone’s input, and updates the shadow state through a gated, recurrent mechanism. The result is a global, recurrent state-space refinement process, shifting the locus of adaptation from weight updates to coordinated hidden-state transformation.

Key properties of the shadow module include:

• Layer sharing: One shadow network, reused across all depth $L$ of the transformer.
• Backbone decoupling: The shadow can be pretrained or maintained independently, unlike per-layer LoRA/DoRA updates.
• Detachability: The shadow module can operate standalone in inference, supporting resource-constrained deployments.

2. Architectural Specifics

ShadowPEFT overlays a frozen transformer backbone ($f_{\text{base}}^{(1)},\ldots,f_{\text{base}}^{(L)}$) with a shadow network $f_{\text{shadow}}(\cdot;\theta_{\text{shadow}})$, typically a small transformer of depth $L_s \ll L$. The input sequence $x$ is embedded via $$h_{\text{out}}^{(0)} = \mathrm{Embed}_{\text{base}}(x)$$, and the initial shadow state is set as $$s^{(0)} = f_{\text{shadow}}(\mathrm{Embed}_{\text{base}}(x);\theta_{\text{shadow}})$$.

For each layer $\ell = 1, \ldots, L$, the following sequence is executed:

• Shadow Injection: Compute $s^{(\ell)}$0, then project through a trainable low-rank bottleneck:

$s^{(\ell)}$1

Update the backbone input: $s^{(\ell)}$2.

• Base Encoding: Apply the frozen layer: $s^{(\ell)}$3.
• Shadow Update: Update the shadow state via gated recurrence:

$s^{(\ell)}$4

Here, all projection and multilayer perceptron (MLP) weights in injection and update steps are trainable; all backbone weights are static.

3. Formulation and Training Objectives

The shadow state is evolved as

$s^{(\ell)}$5

where $s^{(\ell)}$6 is the gated residual update defined by $s^{(\ell)}$7 and $s^{(\ell)}$8.

For inference in "attached" mode, the terminal backbone hidden state $s^{(\ell)}$9 can be further refined by the shadow network via a small MLP $L$0,

$L$1

The training objective for causal language modeling is a joint cross-entropy:

$L$2

where $L$3 is the frozen base model output head and $L$4 is a small head for shadow state prediction. For classification, a similar cross-entropy is applied to pooled outputs from both the base and shadow modules. The shadow loss term acts as a regularizer and enables deployment in a detached mode.

4. Contrasts with LoRA and DoRA

Traditional LoRA parameterization applies to each selected linear weight $L$5 by learning $L$6, $L$7, and injecting:

$L$8

DoRA further decomposes LoRA's update into magnitude and direction, introducing an extra diagonal scaling parameter.

Aspect	LoRA/DoRA	ShadowPEFT
Parameterization	Per-layer, local, low-rank	Single centralized shadow plus small per-layer modules
Cross-Layer Coordination	None	Coordinated via stateful shadow
Trainable parameters (typical budget)	Baseline	~5–8% fewer
Inference Overhead	LoRA: lower; DoRA: 15–20% slowdown	ShadowPEFT: 4–6% above LoRA

The centralized, recurrent nature of ShadowPEFT enables iterative, cross-layer adaptation that does not exist in locally linear LoRA or DoRA updates. The overhead of ShadowPEFT is typically measured at 4–6% above LoRA and is lower than DoRA’s, which incurs a 15–20% slowdown due to its greater complexity.

5. Training Protocols and Inference Modes

Training involves initialization of all shadow and auxiliary module parameters while keeping the backbone frozen. The forward pass executes the sequence defined above for every batch. The official pseudocode is:

$f_{\text{base}}^{(1)},\ldots,f_{\text{base}}^{(L)}$0

Inference can be performed in:

• Shadow-attached mode: Full shadow and backbone pipeline, yielding either backbone or combined outputs.
• Detached shadow-only mode: The backbone is omitted; prediction uses only $L$9. This supports efficient edge deployment.

6. Empirical Evaluation and Results

ShadowPEFT’s effectiveness was validated on Qwen3 backbone models of 0.6B, 4B, and 8B parameters, with matched trainable parameter budgets of approximately 9M, 23M, and 30M. Evaluation protocols included:

• Generation/Reasoning Tasks: MMLU, GSM8K (pass@1), SQuAD-v2 (F1)
• Understanding Benchmarks: Amazon review and 20News classification
• Secondary Metrics: Few-shot OOD generalization, inference latency (ms), robotic intent accuracy

Main findings include:

• Highest average accuracy across all backbone scales (0.6B: 62.27% vs. LoRA 61.81% / DoRA 62.08%; 4B: 75.43% vs. 74.55% / 74.85%; 8B: 76.92% vs. 76.51% / 75.99%).
• ShadowPEFT produces consistent gains on reasoning tasks (e.g., +1.0 on GSM8K at 8B).
• Shadow module pretrained on web text (0.5B) further increases performance to 77.11% when attached, and in detached mode, matches or surpasses LoRA/DoRA on smallest backbone (detached shadow: 62.11%).
• In OOD 2-shot generalization, ShadowPEFT leads (e.g., 50.61% vs. 50.40% LoRA / 48.57% DoRA trained on GSM8K).
• Superior performance in parameter scaling: as trainable parameter budget increases (0.1B to 0.5B on 8B backbone), LoRA plateaus, DoRA deteriorates, while ShadowPEFT rises steadily to 82.12% before mild saturation.
• System-level robot intent studies show that a hybrid system—detached shadow for local tasks, full ShadowPEFT for cloud resolution—yields lowest end-to-end latency (≈0.12 s) and highest intent accuracy (99.35%) across approaches.

7. Practical Implications and Flexibility

ShadowPEFT’s design supports deployment flexibility:

• The detached shadow-only mode is particularly suitable for edge computing scenarios due to its low resource and computational requirements.
• Centralized and decoupled adaptation enables independent pretraining, versioning, and cross-backbone transfer of the shadow module, attributes not possible with standard LoRA/DoRA.
• The approach demonstrates sustained gains on generalization tasks and in system-level use cases, suggesting its suitability for both high-performance and resource-constrained applications.

ShadowPEFT thus represents a shift in PEFT methodology, enabling globally coordinated adaptation with improved parameter efficiency, broader deployment options, and consistent empirical gains relative to alternative methods relying on distributed weight-space perturbations (Li et al., 21 Apr 2026).