Frequency-Aware Continual Learning for Smart Contract Vulnerability Detection with Large Language Models

Abstract: Smart contract vulnerability detection with LLMs faces three causally linked challenges. First, new vulnerability categories demand parameter-efficient adaptation, since full retraining is prohibitive for sequentially arriving tasks. Second, training per-task adapters on a shared backbone causes catastrophic forgetting of previously learned vulnerabilities. Third, the resulting multiplicity of adapters must be consolidated into a single model, since task identity is unknown at inference time. Each challenge arises directly from the solution to its predecessor, making an integrated framework essential. We propose a three-stage pipeline in which each stage addresses one challenge and feeds into the next. The adaptation stage uses Frequency-Aware Low-Rank Adaptation (FA-LoRA), which performs adaptation in the Fourier domain with per-frequency importance gates, requiring only 0.4% trainable parameters while outperforming standard LoRA and QLoRA. The continual learning stage applies Forget-Aware Replay (FAR), which uses these frequency gates to estimate per-sample forgetting risk via loss dynamics and prioritizes vulnerable knowledge for rehearsal, achieving an average Micro-F1 of 0.8022 across sequential tasks. The deployment stage employs Anchor-Protected Progressive Merging (APPM), which exploits the asymmetric generalization produced by FAR training to identify the strongest-generalizing adapter as an anchor and consolidates all adapters into a single model via anchor-protected weighted merging with frequency-domain gate competition. APPM achieves a Micro-F1 of 0.8085, within 2.7% of the independent per-task upper bound, at a merge cost of 156 ms and no additional runtime memory. Experiments on DIVE confirm the framework effectively addresses all three challenges for evolving blockchain ecosystems.

• The paper introduces FA-LoRA, FAR, and APPM, achieving 0.8424 Micro-F1 with only 0.4% trainable parameters, 0.8022 in continual learning, and 0.8085 after data-free adapter merging.
• Frequency-aware high-frequency gating reduces adaptation costs, while loss-based replay prioritizes samples at risk of forgetting, although early-task performance still declines by 4.9%.
• APPM combines anchor protection with frequency competition to outperform TIES, DARE, and HAM, reaching within 2.7% of the independent upper bound at a 156 ms merge cost.

Motivation and problem structure

The paper addresses a structural gap in LLM-based smart contract vulnerability detection: existing fine-tuned detectors assume a static vulnerability taxonomy, whereas real-world attack categories evolve over time. The authors frame three challenges as a causal chain: (1) sequential tasks require parameter-efficient adaptation because full retraining is prohibitive; (2) per-task adapters trained on a shared frozen backbone cause catastrophic forgetting; (3) the resulting multiple adapters must be consolidated into a single model because task identity is unavailable at inference, and naive merging causes destructive parameter interference. Each challenge arises from the solution to its predecessor, which motivates an integrated three-stage framework rather than isolated fixes (2608.19680).

Framework overview

The proposed pipeline couples three components with explicit design dependencies. Stage 1 (FA-LoRA) fine-tunes a frequency-domain adapter on a frozen backbone. Stage 2 (FAR) performs continual learning with replay prioritized by per-sample forgetting risk estimated from loss dynamics. Stage 3 (APPM) merges the $K$ task adapters into a single deployment model, data-free, in parameter space. The dependency structure is central to the design: FA-LoRA's frequency gates inform both FAR's forgetting estimation and APPM's frequency-domain gate competition, while the asymmetric generalization produced by FAR training (later adapters accumulate more knowledge) supplies the anchor for APPM. The authors argue this creates a reinforcing cycle in which stronger continual learning yields better anchors and hence better merged models.

Frequency-Aware Low-Rank Adaptation (FA-LoRA)

FA-LoRA modifies standard LoRA by transforming the low-rank update $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$ into the Fourier domain, applying a learnable sigmoid gate $\boldsymbol{s} = \sigma(\boldsymbol{g})$ over frequency components, and retaining only the top $\lceil \gamma r \rceil$ components by gate magnitude. Following prior observations that high frequencies encode fine-grained, task-specific adaptation signals, the retained band is restricted to high frequencies. Applied to the $W_q$ and $W_v$ projections of each transformer layer, the method trains only $2dr + r$ parameters per adapted layer, reducing optimization complexity from quadratic to linear in the hidden dimension.

Empirically, on the full 22,330-contract DIVE benchmark with 8 multi-label vulnerability categories, FA-LoRA trains 2.62M parameters (0.4% of the 1,241M total) with ~10 MB storage per task adapter. At the LLaMA-3.2-3B scale it attains a Micro-F1 of 0.8424, exceeding both LoRA (0.8370) and QLoRA (0.8365) across Micro-F1, Macro-F1, and subset accuracy, using 23% fewer trainable parameters than LoRA. At the 1B scale it matches QLoRA (0.8185) but is surpassed by WaRA (0.8398), which trains 13.7× more parameters — a limitation the paper does not explicitly discuss. The FouRA-versus-FourierFT comparison serves as a controlled ablation confirming that high-frequency retention, rather than frequency-domain parameterization per se, drives the gain.

Forget-Aware Replay (FAR)

FAR replaces uniform replay sampling with prioritized sampling driven by per-sample loss dynamics. Each buffer entry stores the input, label, and a forgetting indicator given by the sample-wise binary cross-entropy loss, recomputed after each task. Replay probability follows a temperature-scaled softmax over these losses, which the authors show is the closed-form solution of an entropy-regularized maximization of expected rehearsal loss — the temperature $\tau$ interpolates between concentrating rehearsal on high-risk samples and uniform sampling. The combined training objective adds a weighted rehearsal loss to the current-task loss.

On the DIVE benchmark partitioned chronologically into four timestamp-ordered tasks, FAR achieves an average Micro-F1 of 0.8022, the best among the compared CL methods and within 3.5% of the independent per-task upper bound. The backward evaluation matrix reveals that residual forgetting is concentrated in the earliest task: task_A declines from 0.7495 to 0.7128 (4.9% relative), while task_C declines by only 0.4%. The authors attribute this to the fixed-capacity buffer progressively undersampling earlier tasks — a known structural limitation of replay-based CL that prioritized sampling mitigates but does not eliminate.

Anchor-Protected Progressive Merging (APPM)

APPM consolidates $K$ adapters without accessing training data. It first selects an anchor adapter by aggregated parameter norm, exploiting the observation that later FAR-trained adapters generalize more strongly. Non-anchor adapters are then aggregated via weighted averaging, where contribution weights $\lambda_k$ are modulated by the protection coefficient $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$0 relative to the anchor's norm. Frequency gates are merged separately through a temperature-controlled softmax competition, assigning each frequency bin to the task that activates it most strongly, with anchor-aware importance coefficients biasing the allocation.

APPM reaches an average Micro-F1 of 0.8085 — notably exceeding the FAR sequential result of 0.8022 — within 2.7% of the independent upper bound, at a merge cost of 156 ms with zero additional runtime memory. Against baselines, it dominates TIES (+5.5% gap), DARE (+7.9%), and HAM (+11.0%) while remaining within 54 ms of Simple-Mean's merge time; the fastest baseline, SFA, sacrifices 33.0% accuracy. The ablation isolates the mechanisms: removing anchor protection drops task_D from 0.8737 to 0.8073, while removing frequency competition hurts earlier tasks (task_A falls from 0.7387 to 0.7218 in the anchor-free variant), indicating that the two mechanisms address complementary failure modes and both are necessary.

Sensitivity analysis

Across 27 experiments, three of four hyperparameters are effectively inert: rank $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$1 spans only 0.0041 Micro-F1 between $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$2 and $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$3 (suggesting frequency gating compensates for reduced rank); retention ratio $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$4 varies by 0.0022 between $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$5 and $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$6; and FAR temperature $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$7 spans 0.0021. The exception is APPM's protection strength $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$8, the only impactful parameter: $$\Delta\boldsymbol{H} = \boldsymbol{x}(\boldsymbol{U}\boldsymbol{V})$$9 outperforms $\boldsymbol{s} = \sigma(\boldsymbol{g})$0 by +3.07% average Micro-F1, with gains concentrated on later tasks (+5.80% on task_C, +6.62% on task_D). This confirms anchor protection as the primary mechanism and yields a simple deployment recommendation.

Limitations and open questions

Several limitations are stated or implicit. The evaluation uses only four temporal tasks on a single benchmark (DIVE) with a 1B-parameter backbone for CL experiments; scalability to longer task sequences, larger backbones, and other vulnerability datasets is untested. The claim that later adapters generalize more strongly — the basis of anchor selection — is empirically motivated but not formally established, and APPM's anchor choice is heuristic (parameter norm), leaving open whether task-accuracy-aware anchor selection would improve merging. Residual forgetting on the earliest task (4.9% relative decline) remains unaddressed. The PEFT comparison shows WaRA outperforming FA-LoRA at both scales despite its much larger parameter budget, which the paper does not analyze. The conclusion proposes extension to heterogeneous 6G edge-cloud settings with asymmetric resource constraints, but provides no evidence for feasibility in that setting.

Conclusion

The paper presents a coherent three-stage framework — FA-LoRA, FAR, and APPM — in which frequency-domain parameterization is the unifying design element enabling forgetting estimation, prioritized replay, and interference-free merging. The headline results are 0.4% trainable parameters with competitive or superior accuracy to LoRA/QLoRA at the 3B scale, best-in-class continual learning performance (0.8022 average Micro-F1), and data-free merging within 2.7% of the per-task upper bound at 156 ms cost. The framework's dependence on a single benchmark and a heuristic anchor-selection rule are the principal open points for subsequent validation.