Acoustic-to-Semantic Progressive Fine-Tuning

• Acoustic-to-Semantic Progressive Fine-Tuning is a staged paradigm that first anchors models with acoustic objectives before specializing them for semantic tasks.
• It integrates multiple methodologies including CTC-only adaptation, attention-based transduction, and teacher–student alignment to bridge ASR and semantic understanding.
• Empirical results demonstrate improved accuracies in dialogue act classification, intent recognition, and acoustic scene analysis while mitigating error propagation.

Acoustic-to-Semantic Progressive Supervised Fine-Tuning denotes a staged training paradigm in which supervision is organized so that models are first anchored by acoustic, phonetic, lexical, or alignment-oriented objectives and are then specialized toward semantic prediction. In the cited literature, this progression appears in several concrete forms: CTC-only adaptation followed by joint CTC and SLU optimization over self-supervised acoustic encoders (Wang et al., 2023); ASR pretraining followed by words-plus-concepts decoding and then concept–value transduction in attention-based spoken language understanding (Pelloin et al., 2021); and ASR pretraining followed by cross-modal teacher–student alignment and few-shot downstream adaptation in end-to-end SLU (Denisov et al., 2020). Related work extends the same logic to unsupervised semantic augmentation of speech encoders (Xu et al., 2022), curriculum-based audio prompting for frozen LLMs and VLMs (Liang et al., 2023), progressive compression of acoustic sequences for ASR and speech translation (Xu et al., 2023), and semantically structured acoustic scene representations learned by contrastive fine-tuning and distillation (Yuan et al., 4 Oct 2025). This suggests the term functions as a family of staged training schemes rather than a single fixed architecture.

1. Core idea and supervisory progression

The defining feature of the paradigm is that supervision is introduced in an ordered sequence from lower-level acoustic structure toward higher-level semantic structure. In the joint CTC-SLU formulation, training proceeds in two phases: first, CTC-only fine-tuning with $\alpha_{CTC}=1$ and $\alpha_{SLU}=0$; second, end-to-end joint optimization with $\alpha_{CTC}>0$ and $\alpha_{SLU}>0$, so that the same frame features become more semantically informative while preserving ASR competence (Wang et al., 2023). In the attention-based semantic transduction formulation, the stages are ASR pretraining, then AllWords-C, then specialization to either SupWords-C or NormValues-C, with successive stages reusing and extending previously learned parameters rather than reinitializing them (Pelloin et al., 2021). In the teacher–student SLU formulation, the sequence is ASR pretraining, cross-modal alignment between speech-side utterance embeddings and text-side sentence embeddings, and then progressive few-shot downstream fine-tuning (Denisov et al., 2020).

This staged organization distinguishes the paradigm from both conventional ASR$\rightarrow$NLU pipelines and purely semantic end-to-end systems that omit intermediate textual or alignment guidance. Pipeline systems optimize ASR and NLU independently and therefore expose downstream semantics to error propagation, whereas purely acoustic-to-semantic models may lack lexical grounding. The literature repeatedly treats intermediate supervision—CTC, words-plus-concepts decoding, or text-embedding alignment—as the mechanism that bridges these regimes (Wang et al., 2023).

Formulation	Stages	Supervisory signal
Joint CTC-SLU	CTC-only $\rightarrow$ joint CTC+SLU	transcripts $\rightarrow$ utterance labels
Attention transduction	ASR $\rightarrow$ AllWords-C $\rightarrow$ SupWords-C or NormValues-C	characters/words $\rightarrow$ concept symbols $\alpha_{SLU}=0$0 normalized values
Teacher–student SLU	ASR $\alpha_{SLU}=0$1 speech/text embedding alignment $\alpha_{SLU}=0$2 few-shot task tuning	transcriptions $\alpha_{SLU}=0$3 latent-space matching $\alpha_{SLU}=0$4 class labels
Audio prompting curriculum	alignment pretraining $\alpha_{SLU}=0$5 single-audio tasks $\alpha_{SLU}=0$6 multi-audio reasoning	audio-text matching and generation $\alpha_{SLU}=0$7 seq2seq supervision $\alpha_{SLU}=0$8 reasoning supervision

2. Objective functions and optimization schedules

The joint CTC-SLU instantiation makes the acoustic-to-semantic progression explicit at the loss level. Its CTC objective is

$\alpha_{SLU}=0$9

while utterance classification uses

$\alpha_{CTC}>0$0

and the joint loss is

$\alpha_{CTC}>0$1

Stage 1 fine-tunes the pretrained encoder with CTC only and early-stops if the ASR loss does not improve for 5 epochs; Stage 2 turns on the SLU loss and trains for up to 50 epochs, with AdamW and task-specific settings such as $\alpha_{CTC}>0$2 for DSTC2 and SLURP, and $\alpha_{CTC}>0$3 for Speech Commands V2 (Wang et al., 2023).

The attention-based transduction formulation uses a standard sequence-to-sequence cross-entropy objective,

$\alpha_{CTC}>0$4

with additive attention aligning decoder steps to encoder states. Its progressive stages do not change the training criterion so much as the target serialization: the model is first pretrained as character-level ASR, then retrained to emit all pronounced words plus concept tokens, and finally specialized either to supports-only words plus concepts or to concepts plus normalized values (Pelloin et al., 2021).

The teacher–student alignment formulation uses distance-based latent-space matching rather than token-level decoding losses. The speech-side student embedding $\alpha_{CTC}>0$5 is aligned to the text-side teacher embedding $\alpha_{CTC}>0$6 using cosine distance, $\alpha_{CTC}>0$7 distance, or $\alpha_{CTC}>0$8 distance, with $\alpha_{CTC}>0$9 selected as the best overall objective:

$\alpha_{SLU}>0$0

The downstream classifier is then trained on sentence embeddings from transcriptions and transferred to speech embeddings, after which few-shot fine-tuning can update the classifier alone or the classifier plus the two top former ASR encoder layers (Denisov et al., 2020).

Later work generalizes the same staged logic to other objective families. The unsupervised semantic augmentation approach combines adversarial and auxiliary bridge-training losses,

$\alpha_{SLU}>0$1

before downstream task fine-tuning (Xu et al., 2022). The audio prompting curriculum combines language modeling, contrastive alignment, audio-text matching, and audio-grounded text generation objectives across stages (Liang et al., 2023). The acoustic scene formulation uses

$\alpha_{SLU}>0$2

for the teacher and

$\alpha_{SLU}>0$3

for the student, making the progression run from semantic structuring to semantic-preserving distillation (Yuan et al., 4 Oct 2025).

3. Representative architectures

One representative architecture couples self-supervised speech encoders to a minimal utterance classifier. A pretrained Wav2Vec 2.0 or HuBERT backbone maps the waveform $\alpha_{SLU}>0$4 to frame-level hidden features $\alpha_{SLU}>0$5. A linear layer then produces per-frame logits $\alpha_{SLU}>0$6 over subword units plus the blank, with

$\alpha_{SLU}>0$7

The SLU head consumes either the frame-level logits $\alpha_{SLU}>0$8 or the hidden states $\alpha_{SLU}>0$9, performs temporal max-pooling to obtain $\rightarrow$0, applies two fully connected layers with GELU activations and 128 units each, and then predicts the utterance label with a linear classifier and softmax. The method uses Wav2Vec 2.0 for DSTC2 and Speech Commands, and HuBERT for SLURP; in practice, hidden states and logits perform similarly, but logits are slightly preferred and empirically better than post-softmax probabilities (Wang et al., 2023).

A second representative architecture is an attention-based encoder–decoder that directly serializes semantics. Its input is a sequence of 40-dimensional Mel filterbank vectors extracted with a 25 ms Hamming window and 10 ms frame shift. The encoder applies 4 two-dimensional convolutional blocks with batch normalization, followed by 4 biLSTM layers. The decoder is a stack of 4 LSTM layers followed by 2 fully connected layers and a softmax over a character vocabulary augmented with special concept symbols. No BIO schema is used: each concept label maps to a dedicated special character, while values are emitted as normalized character sequences in the NormValues-C setting. Support boundaries are handled implicitly by attention and by the positions at which concept tokens are emitted (Pelloin et al., 2021).

A third architecture realizes the progression through latent-space transfer. The acoustic encoder is the encoder block of a pretrained end-to-end ASR Transformer with attention dimension $\rightarrow$1, feed-forward dimension 2048, 8 heads, and 12 encoder blocks, operating on 80-dimensional log-Mel features plus 3-dimensional pitch. A learnable linear projection $\rightarrow$2 maps these outputs into the hidden size of Sentence-BERT, whose self-attention blocks then process the projected sequence and whose MEAN pooling produces the speech-side utterance embedding. The text-side teacher is bert-base-nli-stsb-mean-tokens, frozen during alignment, and only selected student parameters are updated (Denisov et al., 2020).

Broader variants preserve the same acoustic-to-semantic direction while altering the interface. The unsupervised semantic augmentation model inserts an ASR-U bridge and BART-derived semantic stream between the speech encoder and downstream task decoder (Xu et al., 2022). Acoustic Prompt Tuning uses Audio-MAE, an instruction-aware audio aligner with learnable queries, a fixed prompt length of $\rightarrow$3, and a learnable <AUDIO> token to inject audio-derived soft prompts into a frozen Vicuna-based LLM (Liang et al., 2023). Progressive Down-Sampling alternates Conv1D down-sampling with Transformer or Conformer interaction blocks so that fine-grained acoustic frames are gradually compressed into coarser units (Xu et al., 2023).

4. Inference regimes, efficiency, and granularity

A central distinction within the literature concerns whether the acoustic-to-semantic pathway remains non-autoregressive at inference. In the joint CTC-SLU formulation, no ASR decoding is required for SLU inference. The model performs one encoder pass, uses frame-level logits or hidden states directly as “text-like” embeddings, applies temporal max-pooling and a small MLP, and predicts the utterance label without explicit CTC decoding, best-path alignment, or beam search. The paper contrasts this with prior end-to-end SLU systems that consume decoder-side embeddings from sequence-to-sequence ASR models and therefore incur token-by-token autoregressive decoding and often beam search (Wang et al., 2023).

By contrast, the attention-based semantic transduction formulation remains explicitly autoregressive. Decoding uses beam search with shallow fusion,

$\rightarrow$4

where the external LLM is LSTM-based with look-ahead word probabilities. The weight $\rightarrow$5 is tuned on the development set and is reported to be close to zero for some configurations, especially the concept–value model. This is an important corrective to a common simplification: progressive fine-tuning in the literature does not imply a single decoding regime, and some realizations remain attention-decoder systems with external LLMs (Pelloin et al., 2021).

The granularity perspective extends the same principle beyond SLU heads and decoder interfaces. Progressive Down-Sampling defines stage-wise compression by

$\rightarrow$6

with total compression ratio $\rightarrow$7, and reduces self-attention cost from $\rightarrow$8 to $\rightarrow$9 on the compressed representation. A multi-level fusion module then reconstructs information across stages as

$\rightarrow$0

The reported result is compression to $\rightarrow$1 of the initial length while maintaining better or comparable ASR performance and yielding inference speedups ranging from $\rightarrow$2 to $\rightarrow$3 (Xu et al., 2023). This suggests that progressive fine-tuning can also be realized as progressive semantic densification of the acoustic sequence.

5. Empirical results and ablation evidence

On utterance-level spoken language understanding, the joint CTC-SLU approach reports strong gains. For dialogue act classification on DSTC2, the baseline state of the art from Wei et al., 2022 is 93.6% accuracy, while the proposed model reaches 97.6% with hidden states as input and 97.5% with logits as input, an absolute improvement of approximately 4.0%. For Google Speech Commands V2 keyword spotting, the model achieves 98.0%, matching the HTS-AT baseline. For intent classification on SLURP, baselines include ESPnet-SLU at 86.3% and Seo et al., 2022 at 86.9%, while the proposed HuBERT-based model reaches 88.1% with hidden states and 88.2% with logits, an absolute improvement of 1.3% over prior state of the art; statistical significance is not reported (Wang et al., 2023).

Its ablations identify the mechanisms responsible for those gains. On SLURP, HuBERT LARGE with joint CTC+SLU achieves 88.18%. Cascaded baselines remain weaker despite larger NLU heads: a BiLSTM NLU with 24.7M parameters reaches 83.95%, and a BERT NLU with 110M parameters reaches 87.44%. An end-to-end SLU model without CTC reaches 84.81%, indicating the importance of textual supervision from CTC. Freezing ASR after CTC and training only the SLU head performs poorly at 72.16%, while using post-softmax probabilities as SLU input gives 87.00%, below logits or hidden states. A larger CNN textual encoder with 1.9M parameters does not help relative to the small MLP, giving 88.05% versus 88.18%. After joint training, ASR improves from WER 18.2% to 17.4% and CER 8.5% to 7.8%, and the model predicts correct intents for 83.4% of utterances that still contain ASR errors (Wang et al., 2023).

On the French MEDIA corpus, the attention-based transduction formulation reports Concept Error Rate and Concept–Value Error Rate rather than utterance accuracy. With shallow fusion, AllWords-C reaches 13.6% CER and 18.5% CVER on test, corresponding to an absolute 2.8-point CER reduction relative to the prior state of the art at 16.4% CER. Without the LLM, AllWords-C yields 15.6% CER and 20.4% CVER. NormValues-C reaches 15.4% CER and 21.6% CVER with shallow fusion, and 15.4% CER and 21.7% CVER without it. SupWords-C does not help in this attention-based architecture, unlike earlier CTC-based systems, because the attention mechanism already focuses on relevant acoustic spans without requiring frame-synchronous output behavior (Pelloin et al., 2021).

The teacher–student alignment formulation emphasizes zero-shot transfer and few-shot adaptation. The pipeline baseline using ASR output attains 57.23% on SwBD, 64.06% on MRDA, and 94.57% on FSC. After cross-modal alignment with the best $\rightarrow$4 objective and no downstream speech labels, the end-to-end SLU model reaches 58.60% on SwBD, 60.18% on MRDA, and 91.12% on FSC, which is comparable to the pipeline on two benchmarks and lower on MRDA. With 10 labeled speech examples per class and fine-tuning of the output layer plus hidden layers, performance rises to 60.22% on SwBD, 61.32% on MRDA, and 95.49% on FSC, outperforming the pipeline on SwBD and FSC while remaining below it on MRDA (Denisov et al., 2020).

6. Extensions, misconceptions, and open problems

A frequent misconception is that acoustic-to-semantic progressive supervised fine-tuning necessarily requires labeled transcripts at every stage. The unsupervised semantic augmentation line explicitly removes supervised ASR pairs: an SSL speech encoder is connected to a CNN generator, a WFST bridge from phonemes to subwords, and BART-derived semantic embeddings, after which lightweight downstream decoders are trained. On SLURP intent classification, the paper reports 49.97% $\rightarrow$5 63.64% for w2v2l14 and 58.11% $\rightarrow$6 64.33% for HuBERT after introducing semantics; it also states that the unsupervised method achieves similar downstream performance to supervised w2v2-ASR trained on 100 hours of labeled audio transcripts on several tasks (Xu et al., 2022). The original method is unsupervised, but the paper explicitly describes how limited supervision could be introduced progressively by first fine-tuning the bridge and adapters and then selectively unfreezing higher speech layers. This suggests that the paradigm can be implemented even when full transcript supervision is unavailable.

A second misconception is that the paradigm is confined to intent or dialogue-act classification. Acoustic Prompt Tuning formulates a broader curriculum: Stage 0 learns audio–text alignment on AudioSet and WavCaps; Stage 1 uses single-clip seq2seq tasks such as tagging, captioning, audio QA, temporal retrieval, and counting; Stage 2 uses interleaved multi-clip reasoning for few-shot classification and Natural Language Audio Reasoning. Using a frozen Vicuna 7B backbone, Audio-MAE, and an instruction-aware audio aligner with $\rightarrow$7 prompt vectors per clip, the method reports 91.0% accuracy on 5-way 5-shot ESC-50, 62.9% and 63.8% NLAR accuracy with APT-Vicuna v1.1 and v1.5, and 59.7% on audio–video question answering when extending BLIP-2 without finetuning (Liang et al., 2023). Here, the progression runs from alignment to single-audio semantics to multi-audio reasoning rather than from ASR to SLU.

A third misconception is that the “semantic” target must always be linguistic. In acoustic scene classification, ContrastASC first performs supervised contrastive fine-tuning of a BEATs teacher and then contrastive representation distillation into compact CP-Mobile students. The teacher preserves closed-set TAU22 accuracy at 62.5% while improving TUT17 few-shot performance from 60.1% to 62.3% in the 5-shot setting and from 70.4% to 72.4% in the 20-shot setting; the CP-Mobile 126K student improves from 53.0% to 56.3% on TUT17 5-shot and from 62.9% to 66.5% on TUT17 20-shot when contrastive fine-tuning and CRD replace conventional fine-tuning and KD (Yuan et al., 4 Oct 2025). This suggests that “acoustic-to-semantic” can also denote semantically structured category geometry rather than language-conditioned semantics.

The literature also converges on several open problems. The joint CTC-SLU model is limited to utterance-level classification and explicitly does not address sequence labeling such as slot filling; future work is directed toward sequence labeling and differentiable, efficient, non-autoregressive recovery of entity names from CTC outputs (Wang et al., 2023). The attention-based transduction model still exhibits frequent insertions and deletions of logical connectors and co-reference markers, and gains from external language modeling are limited for concept–value transduction (Pelloin et al., 2021). The teacher–student system remains below the ASR pipeline on MRDA after few-shot tuning (Denisov et al., 2020). The unsupervised semantic augmentation framework depends on WFST mappings and GAN stability (Xu et al., 2022), while Acoustic Prompt Tuning degrades on very long interleaved sequences and must be trained separately for different LLM or VLM embedding spaces (Liang et al., 2023). Across these formulations, the recurring technical question is how to preserve acoustic fidelity while injecting progressively stronger semantic structure without sacrificing efficiency, generalization, or alignment quality.