--- title: 'PeftCD: Parameter-Efficient Fine-Tuning' url: https://www.emergentmind.com/topics/peftcd type: topic --- # PeftCD: Parameter-Efficient Fine-Tuning PeftCD is a name applied in recent arXiv literature to parameter-efficient fine-tuning systems that adapt foundation models under tight trainable-parameter budgets. In 2025, the label denotes both a Whisper-based framework for cleft lip and palate speech detection and severity classification and a vision foundation model framework for remote-sensing change detection; in 2026, AdaPaD explicitly presents its self-correcting rank-discovery machinery as a natural backbone for a broader “PeftCD” paradigm, expanding the term from domain-specific instantiations toward a more general PEFT-by-component-decomposition interpretation [2507.14898][2509.09572][2605.10741]. ## 1. Terminology and scope The term has multiple, related uses. In speech pathology, the detailed summary of “Parameter-Efficient Fine-Tuning of Foundation Models for CLP Speech Classification” emphasizes **PeftCD**, expanded there as **Parameter-Efficient Fine-Tuning for Cleft Detection**. In remote sensing, **“PeftCD: Leveraging Vision Foundation Models with Parameter-Efficient Fine-Tuning for Remote Sensing Change Detection”** uses the same label for a VFM-based change detection framework. AdaPaD then introduces a broader phrase, **“PeftCD” (Parameter-Efficient Fine-Tuning via Component Decomposition)**, in its implications section [2507.14898][2509.09572][2605.10741]. | Usage | Expansion or task | Basis | |---|---|---| | Speech CLP classification | Parameter-Efficient Fine-Tuning for Cleft Detection | Whisper-small with LoRA or DoRA | | Remote sensing | Remote sensing change detection | SAM2 or DINOv3 with LoRA or Adapter | | Generalized interpretation | Parameter-Efficient Fine-Tuning via Component Decomposition | AdaPaD implications | This suggests that **PeftCD is not yet a fully standardized term**. Rather, it presently names a family of PEFT configurations that share a common design principle: freeze a large pretrained backbone, inject lightweight trainable modules, and concentrate adaptation capacity where task-specific signal is expected to be strongest. ## 2. PeftCD in cleft lip and palate speech classification In the CLP setting, the motivation for PEFT is tightly tied to the phonetic and acoustic structure of disordered speech. The paper states that in CLP, **nasalization increases with severity due to the abnormal passage between the oral and nasal tracts; this causes oral stops to be replaced by glottal stops and alters formant trajectories and vowel space**. The detailed summary further specifies four domain characteristics: **nasalization and leakage**, with increased nasal airflow especially on vowels and raised nasal formant energy around **~250 Hz**; **glottal substitution**, where oral stops such as **/p/, /t/, /k/** are often replaced by glottal stops; **formant trajectories**, where **F1, F2** become flatter and more erratic; and **vowel space reduction**, where the corner vowels **/i/, /a/, /u/** show reduced separation in the **F1–F2** plane, with severity correlating with smaller vowel quadrilateral area [2507.14898]. The model comparison spans both handcrafted and foundation-model representations. The speech study evaluates **Wav2Vec2-base (facebook/wav2vec2-base-960h)**, described as a self-supervised model trained to predict quantized speech units over long contexts; **WavLM-base (microsoft/wavlm-base)**, which augments Wav2Vec2 objectives with utterance-level denoising and speaker tasks; and **Whisper-small (openai/whisper-small)**, a weakly supervised sequence-to-sequence model trained on paired audio-text for ASR, whose encoder latent representations capture linguistically meaningful features. The paper’s rationale is that foundation models trained for grapheme prediction or long-term quantized representation prediction may better discriminate CLP severity when fine-tuned on domain-specific data [2507.14898]. Two datasets are used. | Dataset | Speakers and utterances | Train / Dev / Eval | |---|---|---| | NMCPC (English) | 65 children, 1 463 utterances | 929 / 235 / 299 | | AIISH (Kannada) | 60 children, 2 726 utterances | 1 731 / 508 / 487 | For NMCPC, the speakers are **65 children (9–13 years), 41 CLP (22 M/19 F), 24 normal (20 M/4 F)**, with **76 prompts, up to 5 words**. For AIISH, the speakers are **60 children (7–12 years), 31 normal (19 F/12 M), 29 CLP (9 F/20 M)**, with **19 prompts, up to 3 words**. The train, dev, and eval splits are reported per class for both datasets, and the evaluation protocol uses **5-fold cross-validation on train+dev** with final testing on a held-out eval split [2507.14898]. The handcrafted baselines are extracted with **openSMILE**: **eGeMAPS** with **88 dimensions**, **ComParE** with **6 373 dimensions**, and a **Merged (eGeMAPS + ComParE)** representation with **6 461 dims**, reduced to **100 dims via PCA (preserving > 90% variance)**. Classification uses **SVM with RBF kernel (C=1.0, gamma='scale'), balanced class weights**. For frozen foundation features, the pipeline resamples speech to **16 kHz**, pads or truncates to **30 s**, converts Whisper inputs to log-Mel via its **AutoProcessor**, and **mean-pools last encoder hidden states** into a **768-dim vector per utterance**, followed again by the same SVM/RBF classifier [2507.14898]. ## 3. LoRA and DoRA as the speech PeftCD adaptation mechanism The speech PeftCD system chooses **Whisper-small** as the best frozen baseline and then fine-tunes it with **Low-Rank Adapter (LoRA)** and **Decomposed Low-Rank Adapter (DoRA)**. For LoRA, the summary gives the formulation by freezing the original transformer weights $W_0 \in \mathbb{R}^{d \times k}$ and learning only a low-rank update $\Delta W = BA$, where $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times k}$, and $r \ll \min(d,k)$. The updated weight is $$ W = W_0 + BA. $$ The adapters are inserted into the **query ($W_q$), key ($W_k$), and value ($W_v$)** projection matrices of each multi-head attention block, with **rank $r = 8$ chosen for all experiments**. Training backpropagates **cross-entropy loss only through $A$ and $B$**, with the rest of $W_0$ frozen, and **after training, $BA$ is merged with $W_0$ for inference** [2507.14898]. For DoRA, the paper describes an extension of LoRA that **factorizes $BA$ into magnitude and direction components, pruning unimportant single-rank modules during training**. The formulation is $$ U = W_0 + BA,\qquad D = U / \|U\|,\qquad M = \|U\|,\qquad W = M \cdot D. $$ The same insertion points are used, again with **rank $r = 8$**, and cross-entropy is optimized over **$M$ and $BA$’s components**. The detailed summary states that DoRA **learns a normalization step that decouples scale from orientation, enabling dynamic pruning of low-contribution directions**, and **further reduces trainable parameters by keeping only top-k rank-1 updates** [2507.14898]. The PEFT fine-tuning pipeline is **raw speech → foundation encoder + adapters → mean pooling → FC layer → softmax → cross-entropy**. The added head is an **FC layer to 2 classes (detection) or 4 classes (severity)**. Optimization uses **Adam**, **learning rate = $8 \times 10^{-5}$**, **batch size = 16**, **epochs = 10**, **early stopping on dev macro-F1**, plus **mixed-precision**, **gradient clipping at 1.0**, and **warmup LR over first 10% steps** [2507.14898]. The quantitative results are reported as **macro-average F1** on the eval sets. For **Whisper (frozen) + SVM**, detection/severity is **0.92 / 0.53** on NMCPC and **0.93 / 0.49** on AIISH. For **Whisper + LoRA (PEFT)**, detection/severity becomes **0.93 / 0.65** on NMCPC and **0.94 / 0.52** on AIISH. For **Whisper + DoRA (PEFT)**, detection/severity is **0.93 / 0.67** on NMCPC and **0.93 / 0.51** on AIISH. The best handcrafted baseline, **eGeMAPS + SVM**, gives **0.80 / 0.41** on NMCPC and **0.85 / 0.34** on AIISH. Relative improvements over the best foundation baseline (**Whisper + SVM**) are **+1.1% / +26.4%** on NMCPC and **+1.1% / +6.1%** on AIISH; over the best handcrafted baseline (**eGeMAPS + SVM**), the gains are **+16.3% / +63.4%** on NMCPC and **+10.6% / +52.9%** on AIISH. A **paired two-tailed t-test over 5 folds** yields **$p < 0.05$ for severity gains on NMCPC with DoRA**, with **error bars ($\pm 0.01$–$0.02$)** reported in the supplementary material [2507.14898]. A plausible implication is that, in this CLP setting, PEFT contributes most strongly to the **severity classification** problem rather than to the already strong **binary detection** baseline. That interpretation follows directly from the asymmetry of the reported relative gains. ## 4. PeftCD as a vision foundation model framework for change detection In remote sensing, PeftCD is a multi-temporal change detection framework explicitly built on **Vision Foundation Models (VFMs)** with **Parameter-Efficient Fine-Tuning (PEFT)**. The input is a co-registered image pair **$X_1, X_2 \in \mathbb{R}^{H \times W \times 3}$**. The core encoder is a **weight-sharing Siamese encoder** derived from either **SAM2 (“sam2_hiera_large”)**, described as a ViT-based segmentation foundation model with hierarchical tokens, or **DINOv3 (“dinov3_vitl16”)**, described as a self-supervised ViT pretrained on **17B images**. The two streams share all pretrained weights; **only the PEFT modules (LoRA or Adapters) are trainable** [2509.09572]. A distinctive component is **cross-temporal feature exchange**: after encoding, the framework can optionally **swap (“exchange”) features from even-numbered layers between the two streams to encourage the network to focus on change signals**. Decoding is deliberately lightweight. In the **SAM2CD** variant, the decoder **builds an FPN over the multi-scale SAM2 features, fuses them, then applies a small ResBlock-based upsampling head** to produce a probability map **$P_i \in [0,1]^{H \times W}$**. In the **DINO3CD** variant, the decoder **selects same-resolution features $F_5, F_{11}, F_{17}, F_{23} \in \mathbb{R}^{B \times C \times H/16 \times W/16}$**, performs **Deep Attention Fusion**, applies an **ASPP context module** with parallel dilated convolutions at rates **$\{1,6,12,18\}$**, and then performs **progressive upsampling ($\times 2$ twice) via bilinear + depthwise separable convolutions back to $H \times W$**. Each branch yields **$P_1$** and **$P_2$**, and at inference the final prediction is pixel-wise averaged: $$ P_{\text{final}} = (P_1 + P_2)/2. $$ The PEFT modules are specified in standard but concrete form. For LoRA, the frozen projection **$W_0 \in \mathbb{R}^{d \times k}$** receives a low-rank update $$ \Delta W = (\alpha/r)BA,\qquad A \in \mathbb{R}^{r \times k},\quad B \in \mathbb{R}^{d \times r},\quad r \ll \min(d,k), $$ with forward pass $$ h = (W_0 + \Delta W)x = W_0x + (\alpha/r)B(Ax). $$ In PeftCD, LoRA is injected into **every MHSA $q,k,v$ projection of each Transformer block**, with **$r=8$**, **$\alpha=32$**, and **LoRA-dropout = 0.1**. For **ViT-L/16 ($d=k=1024$)**, the summary gives approximately **16 384** trainable parameters per projection, multiplied by **3 projections** and then by **$L$ layers**. The **Adapter** alternative inserts a two-layer bottleneck MLP in parallel to each Transformer block, $$ h_{\text{out}} = h_{\text{in}} + W_{\text{up}}\,f(W_{\text{down}}\,h_{\text{in}}), $$ with **$W_{\text{down}} \in \mathbb{R}^{r \times d}$**, **$W_{\text{up}} \in \mathbb{R}^{d \times r}$**, **$r=32$**, and **$f=\mathrm{GELU}$**. The trainable parameters per block are approximately **$2 \cdot d \cdot r$**, reported as **$\approx 65\,536$ if $d=1024, r=32$** [2509.09572]. ## 5. Training protocol, evaluation, and reported performance in remote sensing The remote-sensing PeftCD formulation uses **standard binary pixel-wise cross-entropy** between prediction and ground truth **$Y \in \{0,1\}^{H \times W}$**, $$ L_{\mathrm{CE}} = -\sum_{h,w} \left[y_{hw}\log p_{hw} + (1-y_{hw})\log(1-p_{hw})\right]. $$ The paper explicitly notes that it **did not add an explicit IoU loss term; they report IoU only as a metric**. Optimization uses **AdamW**, **$\mathrm{lr}=3 \times 10^{-4}$**, **weight decay = $1 \times 10^{-2}$**, **batch = 16**, and **3 000-step linear warmup, cosine or constant thereafter**. Regularization includes **LoRA dropout = 0.1** and **standard transformer dropout in attention/MLP** [2509.09572]. The primary metrics are **Intersection over Union (IoU)**, **F1 score**, and **Overall Accuracy (OA)**, with the summary also giving the standard **Boundary F1** definition while noting that it is **not explicitly reported**. **Pseudo-change suppression** is quantified via **Precision**; the summary states that the paper repeatedly notes PeftCD’s Precision rises, **for example from ~82%→96% on SYSU-CD**, indicating effective filtering of seasonal or illumination changes [2509.09572]. The reported IoU values across seven public datasets are as follows. | Dataset | IoU | |---|---| | SYSU-CD | 73.81% | | WHU-CD | 92.05% | | MSRSCD | 64.07% | | MLCD | 76.89% | | CDD | 97.01% | | S2Looking | 52.25% | | LEVIR-CD | 85.62% | The paper describes these numbers as **state-of-the-art performance across all seven**. Examples given in the summary include **SYSU-CD IoU 73.81 vs prior 72.90 (CDNeXt)**, **WHU-CD IoU 92.05 vs 90.73**, **CDD IoU 97.01 vs 96.49**, and **S2Looking IoU 52.25 vs 51.51**. Ablations further report that on **DINOv3**, **LoRA yields +2.8 pts IoU on SYSU-CD, +1.1 pts on S2Looking, +1.1 pts on MSRSCD**, while on **SAM2**, **Adapter slightly better on seasonal dataset CDD (97.01 IoU) while LoRA excels on LEVIR-CD**. For the **DINO3CD** decoder, the **MFCE decoder (deep attention + ASPP)** adds **+3.3 pts IoU on SYSU-CD, +1.9 pts on WHU-CD, +1.7 pts on S2Looking**, among other improvements [2509.09572]. The efficiency claim is central. Table 11 reports **PeftCD (DINOv3 LoRA): 2.86 M**, **PeftCD (DINOv3 Adapter): 3.67 M**, **PeftCD (SAM2 LoRA): 10.15 M**, and **PeftCD (SAM2 Adapter): 11.0 M** trainable parameters, with all other pretrained weights frozen. These are contrasted with full fine-tuning baselines such as **CDNeXt: 242.7 M trainable**, **CGNet etc: 38–50 M**, and **P2V, ChangeFormer: ~40 M**. The summary concludes that **PeftCD uses 1–2 orders of magnitude fewer trainable parameters**, while the **LoRA/Adapter overhead is negligible compared to a full ViT pass**, **empirical inference speed matches or exceeds baselines on RTX 4090**, and **in practice, PeftCD runs at real-time or faster** [2509.09572]. ## 6. AdaPaD and the generalized “PeftCD” interpretation AdaPaD is not itself named PeftCD in its title, but its final section explicitly argues that **AdaPaD’s self-correcting parallel deflation and data-driven module-wise rank discovery form a natural backbone for a broader “PeftCD” (Parameter-Efficient Fine-Tuning via Component Decomposition) framework**. In AdaPaD, the target matrix is **$Y \in \mathbb{R}^{m \times n}$**, the inputs are **$X \in \mathbb{R}^{d \times n}$**, and the update after communication round $t$ is $$ \Delta W^{(t)} = \sum_{i=1}^r v_i^{(t)}u_i^{(t)\top},\qquad R^{(t)} = Y - \Delta W^{(t)}X. $$ Worker $i$ constructs a private deflated target using only the predecessors’ contributions from the previous round, $$ R_i^{(t)} = Y - \sum_{j