PeftCD: Parameter-Efficient Fine-Tuning

• PeftCD is a family of parameter-efficient fine-tuning approaches that freeze large backbone models while injecting low-rank adapters for targeted domain adaptations.
• In cleft lip and palate speech classification, it leverages models like Whisper-small with LoRA/DoRA to significantly boost severity classification performance under tight trainable-parameter budgets.
• For remote sensing change detection, PeftCD integrates vision foundation models with efficient adapter modules, achieving state-of-the-art IoU and real-time inference speeds.

PeftCD is a name applied in recent arXiv literature to parameter-efficient fine-tuning systems that adapt foundation models under tight trainable-parameter budgets. In 2025, the label denotes both a Whisper-based framework for cleft lip and palate speech detection and severity classification and a vision foundation model framework for remote-sensing change detection; in 2026, AdaPaD explicitly presents its self-correcting rank-discovery machinery as a natural backbone for a broader “PeftCD” paradigm, expanding the term from domain-specific instantiations toward a more general PEFT-by-component-decomposition interpretation (Bhattacharjee et al., 20 Jul 2025, Dong et al., 11 Sep 2025, Su et al., 11 May 2026).

1. Terminology and scope

The term has multiple, related uses. In speech pathology, the detailed summary of “Parameter-Efficient Fine-Tuning of Foundation Models for CLP Speech Classification” emphasizes PeftCD, expanded there as Parameter-Efficient Fine-Tuning for Cleft Detection. In remote sensing, “PeftCD: Leveraging Vision Foundation Models with Parameter-Efficient Fine-Tuning for Remote Sensing Change Detection” uses the same label for a VFM-based change detection framework. AdaPaD then introduces a broader phrase, “PeftCD” (Parameter-Efficient Fine-Tuning via Component Decomposition), in its implications section (Bhattacharjee et al., 20 Jul 2025, Dong et al., 11 Sep 2025, Su et al., 11 May 2026).

Usage	Expansion or task	Basis
Speech CLP classification	Parameter-Efficient Fine-Tuning for Cleft Detection	Whisper-small with LoRA or DoRA
Remote sensing	Remote sensing change detection	SAM2 or DINOv3 with LoRA or Adapter
Generalized interpretation	Parameter-Efficient Fine-Tuning via Component Decomposition	AdaPaD implications

This suggests that PeftCD is not yet a fully standardized term. Rather, it presently names a family of PEFT configurations that share a common design principle: freeze a large pretrained backbone, inject lightweight trainable modules, and concentrate adaptation capacity where task-specific signal is expected to be strongest.

2. PeftCD in cleft lip and palate speech classification

In the CLP setting, the motivation for PEFT is tightly tied to the phonetic and acoustic structure of disordered speech. The paper states that in CLP, nasalization increases with severity due to the abnormal passage between the oral and nasal tracts; this causes oral stops to be replaced by glottal stops and alters formant trajectories and vowel space. The detailed summary further specifies four domain characteristics: nasalization and leakage, with increased nasal airflow especially on vowels and raised nasal formant energy around ~250 Hz; glottal substitution, where oral stops such as /p/, /t/, /k/ are often replaced by glottal stops; formant trajectories, where F1, F2 become flatter and more erratic; and vowel space reduction, where the corner vowels /i/, /a/, /u/ show reduced separation in the F1–F2 plane, with severity correlating with smaller vowel quadrilateral area (Bhattacharjee et al., 20 Jul 2025).

The model comparison spans both handcrafted and foundation-model representations. The speech study evaluates Wav2Vec2-base (facebook/wav2vec2-base-960h), described as a self-supervised model trained to predict quantized speech units over long contexts; WavLM-base (microsoft/wavlm-base), which augments Wav2Vec2 objectives with utterance-level denoising and speaker tasks; and Whisper-small (openai/whisper-small), a weakly supervised sequence-to-sequence model trained on paired audio-text for ASR, whose encoder latent representations capture linguistically meaningful features. The paper’s rationale is that foundation models trained for grapheme prediction or long-term quantized representation prediction may better discriminate CLP severity when fine-tuned on domain-specific data (Bhattacharjee et al., 20 Jul 2025).

Two datasets are used.

Dataset	Speakers and utterances	Train / Dev / Eval
NMCPC (English)	65 children, 1 463 utterances	929 / 235 / 299
AIISH (Kannada)	60 children, 2 726 utterances	1 731 / 508 / 487

For NMCPC, the speakers are 65 children (9–13 years), 41 CLP (22 M/19 F), 24 normal (20 M/4 F), with 76 prompts, up to 5 words. For AIISH, the speakers are 60 children (7–12 years), 31 normal (19 F/12 M), 29 CLP (9 F/20 M), with 19 prompts, up to 3 words. The train, dev, and eval splits are reported per class for both datasets, and the evaluation protocol uses 5-fold cross-validation on train+dev with final testing on a held-out eval split (Bhattacharjee et al., 20 Jul 2025).

The handcrafted baselines are extracted with openSMILE: eGeMAPS with 88 dimensions, ComParE with 6 373 dimensions, and a Merged (eGeMAPS + ComParE) representation with 6 461 dims, reduced to 100 dims via PCA (preserving > 90% variance). Classification uses SVM with RBF kernel (C=1.0, gamma='scale'), balanced class weights. For frozen foundation features, the pipeline resamples speech to 16 kHz, pads or truncates to 30 s, converts Whisper inputs to log-Mel via its AutoProcessor, and mean-pools last encoder hidden states into a 768-dim vector per utterance, followed again by the same SVM/RBF classifier (Bhattacharjee et al., 20 Jul 2025).

3. LoRA and DoRA as the speech PeftCD adaptation mechanism

The speech PeftCD system chooses Whisper-small as the best frozen baseline and then fine-tunes it with Low-Rank Adapter (LoRA) and Decomposed Low-Rank Adapter (DoRA). For LoRA, the summary gives the formulation by freezing the original transformer weights $W_0 \in \mathbb{R}^{d \times k}$ and learning only a low-rank update $\Delta W = BA$, where $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times k}$, and $r \ll \min(d,k)$. The updated weight is

$W = W_0 + BA.$

The adapters are inserted into the query ($W_q$), key ($W_k$), and value ($W_v$) projection matrices of each multi-head attention block, with rank $r = 8$ chosen for all experiments. Training backpropagates cross-entropy loss only through $\Delta W = BA$0 and $\Delta W = BA$1, with the rest of $\Delta W = BA$2 frozen, and after training, $\Delta W = BA$3 is merged with $\Delta W = BA$4 for inference (Bhattacharjee et al., 20 Jul 2025).

For DoRA, the paper describes an extension of LoRA that factorizes $\Delta W = BA$5 into magnitude and direction components, pruning unimportant single-rank modules during training. The formulation is

$\Delta W = BA$6

The same insertion points are used, again with rank $\Delta W = BA$7, and cross-entropy is optimized over $\Delta W = BA$8 and $\Delta W = BA$9’s components. The detailed summary states that DoRA learns a normalization step that decouples scale from orientation, enabling dynamic pruning of low-contribution directions, and further reduces trainable parameters by keeping only top-k rank-1 updates (Bhattacharjee et al., 20 Jul 2025).

The PEFT fine-tuning pipeline is raw speech → foundation encoder + adapters → mean pooling → FC layer → softmax → cross-entropy. The added head is an FC layer to 2 classes (detection) or 4 classes (severity). Optimization uses Adam, learning rate = $A \in \mathbb{R}^{d \times r}$0, batch size = 16, epochs = 10, early stopping on dev macro-F1, plus mixed-precision, gradient clipping at 1.0, and warmup LR over first 10% steps (Bhattacharjee et al., 20 Jul 2025).

The quantitative results are reported as macro-average F1 on the eval sets. For Whisper (frozen) + SVM, detection/severity is 0.92 / 0.53 on NMCPC and 0.93 / 0.49 on AIISH. For Whisper + LoRA (PEFT), detection/severity becomes 0.93 / 0.65 on NMCPC and 0.94 / 0.52 on AIISH. For Whisper + DoRA (PEFT), detection/severity is 0.93 / 0.67 on NMCPC and 0.93 / 0.51 on AIISH. The best handcrafted baseline, eGeMAPS + SVM, gives 0.80 / 0.41 on NMCPC and 0.85 / 0.34 on AIISH. Relative improvements over the best foundation baseline (Whisper + SVM) are +1.1% / +26.4% on NMCPC and +1.1% / +6.1% on AIISH; over the best handcrafted baseline (eGeMAPS + SVM), the gains are +16.3% / +63.4% on NMCPC and +10.6% / +52.9% on AIISH. A paired two-tailed t-test over 5 folds yields $A \in \mathbb{R}^{d \times r}$1 for severity gains on NMCPC with DoRA, with error bars ($A \in \mathbb{R}^{d \times r}$2–$A \in \mathbb{R}^{d \times r}$3) reported in the supplementary material (Bhattacharjee et al., 20 Jul 2025).

A plausible implication is that, in this CLP setting, PEFT contributes most strongly to the severity classification problem rather than to the already strong binary detection baseline. That interpretation follows directly from the asymmetry of the reported relative gains.

4. PeftCD as a vision foundation model framework for change detection

In remote sensing, PeftCD is a multi-temporal change detection framework explicitly built on Vision Foundation Models (VFMs) with Parameter-Efficient Fine-Tuning (PEFT). The input is a co-registered image pair $A \in \mathbb{R}^{d \times r}$4. The core encoder is a weight-sharing Siamese encoder derived from either SAM2 (“sam2_hiera_large”), described as a ViT-based segmentation foundation model with hierarchical tokens, or DINOv3 (“dinov3_vitl16”), described as a self-supervised ViT pretrained on 17B images. The two streams share all pretrained weights; only the PEFT modules (LoRA or Adapters) are trainable (Dong et al., 11 Sep 2025).

A distinctive component is cross-temporal feature exchange: after encoding, the framework can optionally swap (“exchange”) features from even-numbered layers between the two streams to encourage the network to focus on change signals. Decoding is deliberately lightweight. In the SAM2CD variant, the decoder builds an FPN over the multi-scale SAM2 features, fuses them, then applies a small ResBlock-based upsampling head to produce a probability map $A \in \mathbb{R}^{d \times r}$5. In the DINO3CD variant, the decoder selects same-resolution features $A \in \mathbb{R}^{d \times r}$6, performs Deep Attention Fusion, applies an ASPP context module with parallel dilated convolutions at rates $A \in \mathbb{R}^{d \times r}$7, and then performs progressive upsampling ($A \in \mathbb{R}^{d \times r}$8 twice) via bilinear + depthwise separable convolutions back to $A \in \mathbb{R}^{d \times r}$9. Each branch yields $B \in \mathbb{R}^{r \times k}$0 and $B \in \mathbb{R}^{r \times k}$1, and at inference the final prediction is pixel-wise averaged:

$B \in \mathbb{R}^{r \times k}$2

The PEFT modules are specified in standard but concrete form. For LoRA, the frozen projection $B \in \mathbb{R}^{r \times k}$3 receives a low-rank update

$B \in \mathbb{R}^{r \times k}$4

with forward pass

$B \in \mathbb{R}^{r \times k}$5

In PeftCD, LoRA is injected into every MHSA $B \in \mathbb{R}^{r \times k}$6 projection of each Transformer block, with $B \in \mathbb{R}^{r \times k}$7, $B \in \mathbb{R}^{r \times k}$8, and LoRA-dropout = 0.1. For ViT-L/16 ($B \in \mathbb{R}^{r \times k}$9), the summary gives approximately 16 384 trainable parameters per projection, multiplied by 3 projections and then by $r \ll \min(d,k)$0 layers. The Adapter alternative inserts a two-layer bottleneck MLP in parallel to each Transformer block,

$r \ll \min(d,k)$1

with $r \ll \min(d,k)$2, $r \ll \min(d,k)$3, $r \ll \min(d,k)$4, and $r \ll \min(d,k)$5. The trainable parameters per block are approximately $r \ll \min(d,k)$6, reported as $r \ll \min(d,k)$7 if $r \ll \min(d,k)$8 (Dong et al., 11 Sep 2025).

5. Training protocol, evaluation, and reported performance in remote sensing

The remote-sensing PeftCD formulation uses standard binary pixel-wise cross-entropy between prediction and ground truth $r \ll \min(d,k)$9,

$W = W_0 + BA.$0

The paper explicitly notes that it did not add an explicit IoU loss term; they report IoU only as a metric. Optimization uses AdamW, $W = W_0 + BA.$1, weight decay = $W = W_0 + BA.$2, batch = 16, and 3 000-step linear warmup, cosine or constant thereafter. Regularization includes LoRA dropout = 0.1 and standard transformer dropout in attention/MLP (Dong et al., 11 Sep 2025).

The primary metrics are Intersection over Union (IoU), F1 score, and Overall Accuracy (OA), with the summary also giving the standard Boundary F1 definition while noting that it is not explicitly reported. Pseudo-change suppression is quantified via Precision; the summary states that the paper repeatedly notes PeftCD’s Precision rises, for example from ~82%→96% on SYSU-CD, indicating effective filtering of seasonal or illumination changes (Dong et al., 11 Sep 2025).

The reported IoU values across seven public datasets are as follows.

Dataset	IoU
SYSU-CD	73.81%
WHU-CD	92.05%
MSRSCD	64.07%
MLCD	76.89%
CDD	97.01%
S2Looking	52.25%
LEVIR-CD	85.62%

The paper describes these numbers as state-of-the-art performance across all seven. Examples given in the summary include SYSU-CD IoU 73.81 vs prior 72.90 (CDNeXt), WHU-CD IoU 92.05 vs 90.73, CDD IoU 97.01 vs 96.49, and S2Looking IoU 52.25 vs 51.51. Ablations further report that on DINOv3, LoRA yields +2.8 pts IoU on SYSU-CD, +1.1 pts on S2Looking, +1.1 pts on MSRSCD, while on SAM2, Adapter slightly better on seasonal dataset CDD (97.01 IoU) while LoRA excels on LEVIR-CD. For the DINO3CD decoder, the MFCE decoder (deep attention + ASPP) adds +3.3 pts IoU on SYSU-CD, +1.9 pts on WHU-CD, +1.7 pts on S2Looking, among other improvements (Dong et al., 11 Sep 2025).

The efficiency claim is central. Table 11 reports PeftCD (DINOv3 LoRA): 2.86 M, PeftCD (DINOv3 Adapter): 3.67 M, PeftCD (SAM2 LoRA): 10.15 M, and PeftCD (SAM2 Adapter): 11.0 M trainable parameters, with all other pretrained weights frozen. These are contrasted with full fine-tuning baselines such as CDNeXt: 242.7 M trainable, CGNet etc: 38–50 M, and P2V, ChangeFormer: ~40 M. The summary concludes that PeftCD uses 1–2 orders of magnitude fewer trainable parameters, while the LoRA/Adapter overhead is negligible compared to a full ViT pass, empirical inference speed matches or exceeds baselines on RTX 4090, and in practice, PeftCD runs at real-time or faster (Dong et al., 11 Sep 2025).

6. AdaPaD and the generalized “PeftCD” interpretation

AdaPaD is not itself named PeftCD in its title, but its final section explicitly argues that AdaPaD’s self-correcting parallel deflation and data-driven module-wise rank discovery form a natural backbone for a broader “PeftCD” (Parameter-Efficient Fine-Tuning via Component Decomposition) framework. In AdaPaD, the target matrix is $W = W_0 + BA.$3, the inputs are $W = W_0 + BA.$4, and the update after communication round $W = W_0 + BA.$5 is

$W = W_0 + BA.$6

Worker $W = W_0 + BA.$7 constructs a private deflated target using only the predecessors’ contributions from the previous round,

$W = W_0 + BA.$8

The paper’s central claim is self-correction: each worker refines its component against a target built from the latest predecessor estimates, so deflation errors converge to zero over rounds rather than persisting as fixed residuals. Proposition 4.2 bounds the deflation mismatch and shows that it tends to zero, in contrast to sequential deflation, which the paper says freezes early errors permanently (Su et al., 11 May 2026).

On top of this backbone, AdaPaD adds advance learning and per-module dynamic rank discovery. Advance learning allows a worker that would otherwise be idle to run its Rank1 subroutine privately on its current deflation target, accumulating a warm initialization before activation. Dynamic rank discovery introduces a per-module raw importance score

$W = W_0 + BA.$9

with two EMAs, both using $W_q$0,

$W_q$1

and an allocation score $W_q$2. Under a global budget $W_q$3, modules with $W_q$4 are grown by selecting the top-$W_q$5 modules according to this score; each selected module promotes its reserve to committed, increments active rank, and spawns a new reserve component for further advance learning (Su et al., 11 May 2026).

The theoretical results separate algorithmic and statistical error. Theorem 4.1 gives an exponential decay guarantee after warm-up for each component’s total error,

$W_q$6

with $W_q$7, so the error decays exponentially after a modest warm-up period. The noiseless and noisy recovery bounds then decompose overall error into a vanishing algorithmic term and an irreducible statistical floor. The paper therefore argues that a generalized PeftCD system can allocate a global parameter budget across modules while driving algorithmic recovery error toward zero before encountering the noise floor (Su et al., 11 May 2026).

AdaPaD’s empirical results give a concrete sense of what that generalized interpretation entails. On GLUE classification (DeBERTaV3-base, ∼184 M frozen params, adapters attach to 72 linear layers), AdaPaD discovers module-wise ranks up to $W_q$8 under total budget $W_q$9, corresponding to ≈0.34 M trainable params, and reaches 89.34 average GLUE-dev score, compared with 85.36 for LoRA $W_k$0, 88.99 for IncreLoRA, 89.03 for AdaLoRA, 88.87 for SoRA, and 88.01 for dEBORA. On Qwen3-0.6B + SQuAD v1.1/v2, AdaPaD is reported as competitive with fixed-rank LoRA while using on average 30.7% fewer adapter parameters. Component-parallel sharding across 4 × H200 GPUs delivers up to 2.66× speedup (per-batch), with near-linear scaling up to 3 GPUs and still 2.45× at 4 GPUs (Su et al., 11 May 2026).

The same section also records open questions for any future PeftCD platform built on these ideas. The current theory applies to the bilinear regression subproblem $W_k$1 rather than the full nonlinear fine-tuning loss; deflation relies on a positive spectral gap $W_k$2; synchronous communication of all $W_k$3 components can become a bottleneck; and the EMA hyperparameters and growth interval $W_k$4 remain practical tuning knobs. These caveats delimit the current scope of the broader PeftCD concept while clarifying the research agenda implied by the AdaPaD formulation (Su et al., 11 May 2026).