StolenLoRA Model Extraction

• StolenLoRA is a model extraction attack that exploits the compact, low-rank adapters of LoRA to recreate a victim model's specialized functionality.
• It leverages synthetic data generation via LLM prompting and disagreement-based semi-supervised learning to optimize extraction with minimal queries.
• Experiments show up to 96.60% accuracy in replicating model behavior, highlighting significant vulnerabilities in PEFT methods and the need for robust defenses.

StolenLoRA refers to a model extraction attack targeting vision models adapted with Parameter-Efficient Fine-Tuning (PEFT) methods, specifically LoRA (Low-Rank Adaptation). The central risk addressed is that the compactness of LoRA—by design, it only introduces small, trainable low-rank matrices atop a public pre-trained backbone—renders the downstream adaptation susceptible to rapid and effective function-level theft. The StolenLoRA methodology uses synthetic data generation, disagreement-based semi-supervised learning, and LLM-driven prompting to train a substitute model that reconstructs the victim’s LoRA-adapted functionality with a minimal number of queries, achieving up to 96.60% accuracy compared to the target adaptation under fixed query budgets (Wang et al., 28 Sep 2025).

1. Parameter-Efficient Fine-Tuning and LoRA Vulnerabilities

PEFT methods, notably LoRA, operate by freezing the weights of a large pre-trained model and introducing trainable low-rank matrices over selected layers ($\Delta W = BA$), thus providing efficient, domain-specific adaptation while significantly reducing computational requirements. When adapting a vision backbone (e.g., ViT–Base) to a new dataset, LoRA updates amount to a small fraction of the total parameters. This compactness is advantageous for deployment but exposes an attack surface: an adversary with access to the public base model can focus on replicating only the tiny adaptation region, dramatically lowering extraction effort.

LoRA’s vulnerability stems from two principal facts:

• The low-rank updates encode all task-specific knowledge, isolating the adaptation.
• Given the same backbone, matching the functional outputs of the LoRA-adapted model over a moderate synthetic query set suffices to reconstruct the adaptation with high fidelity.

2. StolenLoRA Extraction Attack Methodology

StolenLoRA formalizes model extraction against LoRA-adapted victim models ($F$) by crafting a substitute ($F'$) whose base is a public pre-trained backbone and whose LoRA parameters are trained to mimic the victim. The attack consists of the following steps:

Synthetic Data Generation via LLM Prompting:

• For each class name $c_i$, an LLM produces detailed prompts $p_{i,j} = \mathrm{LLM}(c_i, T, \omega_j)$, where $T$ is a visual template and $\omega_j$ introduces variation.
• Text prompts are rendered into images using a Stable Diffusion model, emulating in-distribution data for the victim model.

Two Training Strategies:

• StolenLoRA–Rand: All synthetic samples are queried against the victim to obtain output labels; the substitute LoRA is optimized to minimize the average loss against victim outputs:

$$\mathcal{L}'(\Delta \theta') = \mathbb{E}_{x} [\mathcal{L}(F'(x), F(x))]$$

• StolenLoRA–DSL: A Disagreement-based Semi-supervised Learning (DSL) framework selects, among batches, only the most informative queries:
  • Substitute predictions are compared against prompt-derived pseudo-labels; confident, matching predictions are pseudo-labeled, while "uncertain" samples are queued for victim query.
  • Label refinement uses an exponential moving average:

  $$\mathcal{L}_{lr}(z, q) = -\sum_{i=1}^C q_i \log \frac{e^{z_i}}{\sum_j e^{z_j}}, \quad q^{(i+1)} = \mu \cdot q^{(i)} + (1-\mu) \cdot (\text{new prediction})$$

The DSL approach maximizes extraction efficiency and information gain per query.

3. Experimental Evaluation and Attack Success Rates

In comprehensive experiments, StolenLoRA is applied to ViT–Base models fine-tuned via LoRA on datasets including CUBS200, Caltech256, Indoor67, Food101, and Flowers102:

Extraction Mode	Backbone Scenario	Attack Success Rate (ASR)	Query Budget
StolenLoRA–Rand	Identical	Near 95%	10,000
StolenLoRA–DSL	Identical	96.60%	10,000
StolenLoRA–DSL	Cross-backbone	> 90%	10,000

ASR is computed as the ratio of substitute accuracy to victim accuracy. Performance matches or exceeds baseline extraction attacks (KnockoffNets, ActiveThief, DFME, E³) and demonstrates StolenLoRA's ability to reconstruct LoRA-adapted behaviors under both identical and cross-backbone scenarios. Notably, DSL enables rapid convergence, especially in complex backbone mismatch cases.

These empirical results establish LoRA's specific susceptibility to extraction: the adaptation can be recreated with high accuracy after only 10k queries on entirely synthetic, LLM-imagined in-distribution data.

4. Inherent Vulnerabilities and Attack Generalization

The root cause of StolenLoRA’s effectiveness is the LoRA framework’s design: all task-specializing knowledge is contained within small, easily isolated matrices. In the context of a public backbone, the attacker only needs to learn a compact and interpretable parameter set ($A, B$) through black-box matching. This extraction is accelerator- and domain-agnostic; it generalizes to unseen classes or tasks where rich prompts can be synthesized by an LLM and then rendered via a generative model.

This property facilitates function-level theft even across backbone changes, due to the predictable interaction between adaptation and base knowledge. The risk is amplified as LoRA becomes a standard for PEFT in both vision and language.

5. Defenses: Diversified LoRA Deployments

A plausible mitigation discussed is the deployment of diversified LoRA adapters. The strategy involves:

• Training multiple distinct adapters ($L_A, L_B$) for the same downstream task but regularized to diverge ($$\mathcal{L}' = \mathcal{L}(L_A) + \mathcal{L}(L_B) - \lambda \cdot \mathrm{KL}(L_A || L_B)$$).
• At inference, adapters are selected randomly per query, injecting output uncertainty.

Experimental evidence shows that this defense decreases extraction success for naive attacks. However, sophisticated strategies such as StolenLoRA–DSL retain higher resilience, indicating that randomization alone cannot fully protect compact PEFT adaptations from information reconsolidation. More research is needed to design robust adversarial defenses tailored to LoRA’s characteristics.

6. Mathematical Formulations in StolenLoRA

The attack is structured as functional minimization under query efficiency:

$$\min_{\Delta \theta'} \mathbb{E}_{x \in \mathcal{D}} \left[ \mathcal{D}(F(x), F'(x)) \right]$$

where $\mathcal{D}$ represents the relevant functional divergence (e.g., cross-entropy), and the models $F$ and $F'$ are LoRA-adapted architectures sharing (or not sharing) the same backbone.

For the cross-backbone setup, substitute outputs are expressed as:

$F'(x) = G_\text{base}(x) + \Delta F'(x)$

and for identical backbone as:

$F'(x) = F_\text{base}(x) + \Delta F'(x)$

In DSL, the adaptive label refinement step is formalized by the self-adaptive loss above, with soft labels continually updated via exponential moving average.

7. Implications and Future Directions

The StolenLoRA attack exposes a critical security gap in PEFT methods—namely, that any LoRA-adapted model whose backbone is public may be functionally replicated with limited, entirely synthetic queries. The demonstrated success rate (up to 96.60%) underscores the urgent need for new defense paradigms beyond standard data obfuscation, especially as parameter-efficient methods proliferate in commercial and open-source deployment.

A plausible implication is that robust defenses should not solely rely on randomization or adapter diversification but integrate cryptographic, watermarking, or privacy-preserving mechanisms specifically engineered to mask the adaptation region or limit knowledge transferability.

In summary, StolenLoRA advances the field’s understanding of extraction attacks targeting PEFT-adapted vision models and motivates research into novel security protocols for safeguarding compact model adaptations (Wang et al., 28 Sep 2025).