VARAN: Variational Inference for Speech Fine-Tuning

• VARAN is a framework for fine-tuning self-supervised speech models using data-dependent layer aggregation framed as a variational inference problem.
• It employs layer-specialized probing heads to preserve unique transformer layer information, thereby mitigating information bottlenecks.
• Integrating with LoRA, VARAN dynamically weights layer contributions per input, leading to improved performance in tasks like ASR and SER.

Searching arXiv for the primary VARAN paper and closely related background papers so the article can cite current sources accurately. VARAN is a framework for fine-tuning self-supervised speech models on downstream tasks by treating layer aggregation as a variational-inference problem rather than selecting the final layer or using a static weighted sum. Introduced in "VARAN: Variational Inference for Self-Supervised Speech Models Fine-Tuning on Downstream Tasks" (Diatlova et al., 16 Aug 2025), it addresses two limitations identified in conventional aggregation schemes: information bottlenecks and static feature weighting for all dataset examples. Its central mechanism is to infer, for each input $x$, a data-dependent posterior over transformer layers and to combine this with layer-specialized probing heads, thereby preserving layer-specific information while enabling flexible feature utilization.

1. Problem Setting and Core Idea

VARAN is formulated for self-supervised speech backbones whose hidden states $h_i$ at different transformer layers encode different information. The framework is motivated by the claim that using only the final layer or a static weighted sum imposes a fixed aggregation strategy on all inputs. VARAN replaces that fixed strategy with an input-conditioned one: the model infers which layers should matter for a given example and weights them accordingly.

The formal starting point is the marginal predictive distribution

$p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$

where the discrete latent variable $l\in\{0,\ldots,n\}$ denotes the choice of layer. Rather than selecting a single layer, VARAN introduces an approximate posterior $q_\theta(l|x)$ over layers. Each layer $i$ has its own lightweight probing head $\psi_i$, with

$p_\theta(y|x,l=i)=\psi_i(h_i)\,.$

This construction makes the aggregation mechanism data-dependent and per-sample, rather than globally fixed across a dataset.

A central design claim of the framework is that the trade-off between preserving layer-specific information and enabling flexible feature utilization can be resolved by decoupling the two roles: separate heads preserve layer-local structure, while $q_\theta(l|x)$ supplies adaptive layer weighting.

2. Variational Formulation and ELBO Objective

VARAN optimizes a lower bound on $\log p(y|x)$ by introducing $h_i$0 and a prior $h_i$1. The derivation begins with Jensen’s inequality:

$h_i$2

This yields the negative ELBO used for training:

$h_i$3

Following $h_i$4-VAE, the KL term is scaled:

$h_i$5

Substituting the probing heads gives the working form

$h_i$6

In this objective, the first term is the expected task loss under the inferred posterior over layers, while the second regularizes layer usage toward a chosen prior. The paper uses a discretized reversed $h_i$7 distribution for $h_i$8, and reports that this prior can be tuned to favor different parts of the network: middle-layer priors for speech emotion recognition and top-layer priors for automatic speech recognition (Diatlova et al., 16 Aug 2025).

3. Data-Dependent Weighting and Layer-Specialized Heads

At training and inference time, VARAN sets

$h_i$9

so each input receives its own mixture weights over layers. The posterior predictor operates on pooled layer representations. If $p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$0 with $p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$1, the procedure is:

1. Mean-pool across time:

$p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$2

2. Stack pooled representations:

$p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$3

3. Apply Multi-Head Self-Attention across the layer dimension:

$p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$4

4. Project to logits and normalize:

$p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$5

$p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$6

At inference, these weights are used to aggregate log-probabilities:

$p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$7

The paper notes an alternative in which one first forms a weighted feature sum $p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$8 and then applies a shared classifier, but the implementation instead keeps probing heads separate to avoid information bottlenecks.

The probing heads are small independent modules attached to each transformer layer. In the implementation they are two fully-connected layers with hidden size 256 and a final softmax for classification or CTC for ASR. During training, each head contributes its per-sample loss weighted by $p(y|x)=\sum_{i=0}^n p(y|x,l=i)\cdot p(l=i)\,,$9. The intended effect is that different heads can preserve and exploit different levels of the pretrained hierarchy rather than forcing all information through a single shared bottleneck.

4. Integration with LoRA and Fine-Tuning Regimes

VARAN is described as orthogonal to the choice between full fine-tuning and parameter-efficient adaptation. In the LoRA regime, the original backbone weights $l\in\{0,\ldots,n\}$0 are frozen and replaced by

$l\in\{0,\ldots,n\}$1

with

$l\in\{0,\ldots,n\}$2

where $l\in\{0,\ldots,n\}$3, $l\in\{0,\ldots,n\}$4, and $l\in\{0,\ldots,n\}$5. Only $l\in\{0,\ldots,n\}$6, $l\in\{0,\ldots,n\}$7, and bias terms are trained.

Within this setup, VARAN applies LoRA to the feed-forward sublayers and simultaneously trains the layer-posterior MHSA and the probing heads $l\in\{0,\ldots,n\}$8. The variational objective remains unchanged. The paper reports that LoRA + VARAN best preserves the pretrained feature hierarchy and maximizes the multi-layer benefit. This positioning is consistent with the broader architectural rationale: LoRA limits disturbance to the pretrained backbone, while VARAN extracts task-relevant information from multiple layers without collapsing them into a single static representation.

The framework therefore defines two interacting axes of adaptation. One axis concerns how much of the backbone is updated: full fine-tuning or LoRA. The other concerns how information is read from the backbone: last-layer only, static weighted sum, or VARAN’s input-conditioned posterior with specialized heads. The reported gains are strongest in the LoRA regime, which the paper attributes to better preservation of pretrained layer features.

5. Experimental Configuration and Reported Performance

The reported experiments use WavLM (Base/Large) and Data2Vec (Base/Large) as backbones. Downstream tasks are automatic speech recognition and speech emotion recognition. For ASR, training uses GigaSpeech XS and evaluation reports WER on GigaSpeech test, plus LibriSpeech test-clean and test-other. For SER, the benchmarks are RAVDESS (7-class) and IEMOCAP (4-class, 10-fold speaker-leave-one-out). Baselines are last-layer only and static weighted sum. Fine-tuning is evaluated under both full fine-tuning and LoRA. The primary metrics are Accuracy and Weighted F1 for SER, and WER for ASR (Diatlova et al., 16 Aug 2025).

Setting	VARAN	Baseline comparison
RAVDESS, Data2Vec-Base + LoRA	0.60 Acc, 0.60 F1	0.52 Acc, 0.49 F1 for weighted-sum
IEMOCAP, Data2Vec-Base + LoRA	0.71 Acc, 0.70 F1	0.71 Acc, 0.69 F1
GigaSpeech, Data2Vec-Base + LoRA	36.04% WER	42.39% static, 37.92% last

On RAVDESS with Data2Vec-Base + LoRA, VARAN attains 0.60 Accuracy and 0.60 F1 versus 0.52 and 0.49 for weighted-sum, reported as a +15% relative improvement. On IEMOCAP with Data2Vec-Base + LoRA, the Accuracy is 0.71 for both VARAN and the comparison, while F1 is 0.70 versus 0.69. On GigaSpeech with Data2Vec-Base + LoRA, VARAN reports 36.04% WER versus 42.39% for static weighting and 37.92% for last-layer-only, corresponding to a 15% relative WER reduction versus static weighting. The paper further states that similar consistent gains are observed for WavLM and Data2Vec-Large, especially in the LoRA regime.

6. Ablations, Hyperparameters, and Interpretation

The paper reports several ablations intended to isolate the contribution of each design element. Removing the separate probing heads and collapsing all $l\in\{0,\ldots,n\}$9 into a single vector degrades performance by up to 10%. Replacing data-dependent $q_\theta(l|x)$0 with fixed static weights yields a 5–15% relative drop. Varying $q_\theta(l|x)$1 produces a U-shaped curve: too small a value overfits per-sample weighting, while too large a value collapses the posterior toward the prior.

Hyperparameters reported in the paper include $q_\theta(l|x)$2, $q_\theta(l|x)$3 degrees of freedom in $q_\theta(l|x)$4, LoRA rank $q_\theta(l|x)$5, and learning rates from $q_\theta(l|x)$6 to $q_\theta(l|x)$7. The prior over layers is a discretized reversed $q_\theta(l|x)$8 distribution, chosen via grid search. Figure 1 of the paper is described as showing that SER tasks favor middle-layer priors, while ASR favors top layers.

These ablations sharpen the interpretation of VARAN’s architecture. The separate heads are not merely an implementation choice; they are the mechanism that prevents the information bottleneck associated with reducing all layer states to a single representation. The posterior predictor is likewise not merely a weighting module; it is the component that turns layer aggregation into an input-conditioned latent-variable inference problem. The resulting framework adds only a small overhead—an MHSA module plus $q_\theta(l|x)$9 lightweight heads—while preserving the formal structure of end-to-end variational training.

7. Position Within Self-Supervised Speech Fine-Tuning

VARAN can be understood as a general recipe for downstream adaptation when intermediate representations of a self-supervised speech model carry heterogeneous information. Its distinctive contribution is to cast layer aggregation itself as the latent structure to be inferred. This differs from a final-layer-only approach, which fixes the relevant abstraction level in advance, and from static weighted sums, which allow multiple layers but impose one global weighting scheme on all inputs.

The paper’s central claim is therefore architectural as much as probabilistic: preserving layer-specific information and enabling flexible feature utilization are not competing objectives if they are assigned to different parts of the model. In VARAN, layer-specialized probing heads preserve the distinct contribution of each transformer layer, while the approximate posterior $i$0 supplies per-example flexibility. Within the reported experiments, this formulation is effective for both ASR and SER, and particularly effective when combined with LoRA, suggesting that dynamic aggregation is most beneficial when the pretrained hierarchy remains comparatively intact (Diatlova et al., 16 Aug 2025).