OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL

Abstract: We propose Online Latent prediction with Invariant Views and rEconstruction (OLIVE), a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE combines view-augmented masked latent prediction with waveform reconstruction under a unified objective. Reconstruction constrains early encoder features to retain signal-level information, while masked latent prediction shapes later contextual representations toward invariance for robust downstream performance. We show that these objectives enable representations that support a broad range of tasks. In particular, OLIVE improves results on generation and speaker tasks, maintains competitive performance on recognition and semantic tasks, and improves waveform reconstruction.

• The paper introduces OLIVE, which unifies analysis and synthesis objectives to support both high-fidelity speech reconstruction and robust downstream tasks.
• It employs view-augmented masked latent prediction alongside a HiFi-GAN V2 based synthesis branch to capture abstract invariance and preserve signal-level details.
• Experimental evaluations on the SUPERB benchmark show improved performance in speaker verification, emotion recognition, and generation tasks, with enhanced perceptual quality.

OLIVE: Unified Analysis–Synthesis Self-Supervision for Speech Representation Learning

Introduction and Motivation

This work introduces OLIVE (Online Latent prediction with Invariant Views and rEconstruction), a framework for self-supervised speech representation learning that integrates both analysis and synthesis objectives during pre-training. Traditional SSL approaches for speech—such as wav2vec 2.0, HuBERT, WavLM, and data2vec—focus mainly on discriminative objectives, aiming to extract representations tailored for downstream predictive tasks (e.g., ASR, speaker identification). While such methods yield strong performance for recognition and classification, they often neglect the preservation of information necessary for high-fidelity speech synthesis or signal generation.

OLIVE proposes a dual-objective framework in which early encoder features are constrained for signal-level information via waveform reconstruction, while deeper layers are shaped for invariance using a masked latent prediction task. This directly targets the longstanding analysis–synthesis paradigm in speech processing, extending it into the SSL domain by jointly optimizing for both discriminative and generative capabilities within the same pre-trained backbone.

Figure 1: OLIVE pre-training framework. Two independently augmented waveform views are passed through a shared local feature extractor; the analysis branch uses view-augmented masked distillation, and the synthesis branch reconstructs the waveform via a neural vocoder.

Methodology

Analysis Branch: View-Augmented Masked Latent Prediction

The analysis pathway generalizes data2vec 2.0’s self-distillation objective by employing two independently augmented waveform views. The student is trained on masked latent features of one view, and the teacher provides targets from a contextualized encoding of the other (unmasked) augmented view. The augmentations (e.g., mixup, gain perturbations) act as priors over desired invariances. Targets are computed by averaging instance-normalized activations from the top K teacher layers. The student minimizes mean squared error at masked positions between its predictions and these teacher targets.

Synthesis Branch: Waveform Reconstruction

The synthesis pathway employs a HiFi-GAN V2 neural vocoder conditioned on early (pre-contextual) encoder features, aiming to reconstruct the original waveform. The vocoder is trained adversarially, using multi-period and multi-scale discriminators in combination with mel-L1, adversarial, and feature-matching losses. Conditioning the vocoder on shallow representations enforces the preservation of signal-level detail lost in purely discriminative pipelines.

Joint Objective and Training

The final objective is a weighted sum of the analysis loss (masked regression on invariance) and the synthesis loss (adversarial waveform reconstruction). The encoder and prediction head are updated via both losses, while the teacher is maintained via EMA. Synthesis affects early encoder layers, whereas the analysis objective mainly governs the later (more abstract) Transformer layers. This separation allows OLIVE to simultaneously retain information for both generation and discrimination.

Experimental Results

Downstream Task Performance

Evaluation is performed using the SUPERB benchmark across content, speaker, paralinguistic, semantic, and generation tasks. OLIVE’s analysis-only and joint models are compared to baseline SSL methods (wav2vec 2.0, HuBERT, WavLM, data2vec, data2vec 2.0). All models are "Base" scale and use the LibriSpeech 960h dataset to isolate objective contributions.

OLIVE-J (joint analysis–synthesis) achieves the highest WavLM-style aggregate score and excels in speaker and generation tasks, while OLIVE-A (Mix+Gain) obtains the highest SUPERB$_s$ normalized aggregate score. Notably, OLIVE-J matches or surpasses baseline SSL models in automatic speaker verification (ASV), emotion recognition (ER), and voice conversion (VC)—indicating successful retention of speaker- and signal-specific detail—while preserving competitive recognition (ASR) and semantic (IC, SF, ST) performance.

Figure 3: Representative SUPERB layer-combination weights. ASR and slot filling (SF) emphasize later contextual layers, while ASV and speech enhancement (SE) rely on earlier layers.

The learned layer-weight profiles demonstrate that content and semantic tasks draw on deep (contextual) layers, whereas speaker and enhancement tasks utilize early (local) layers. This functional separation highlights OLIVE’s conditioning strategy.

Waveform Reconstruction Quality

Waveform reconstruction is evaluated using matched HiFi-GAN V2 vocoders trained on representations from each baseline as well as OLIVE models. OLIVE-J's jointly trained vocoder, as well as its frozen-feature variant, substantially outperform prior SSL methods on spectral (Mel-L1, MCD, LSD), pitch (F0 MAE), intelligibility (STOI), quality (PESQ, UTMOS, ViSQOL), and SNR metrics. The combination of analysis and synthesis objectives leads to both superior objective scores and improved perceptual fidelity.

Figure 2: Spectrograms for three reference utterances and reconstructions from the frozen feature-conditioned OLIVE-J vocoder and the integrated OLIVE-J vocoder.

Figure 4: Spectrograms for one reference utterance and reconstructions from HiFi-GAN V2 vocoders conditioned on local features from the baseline and OLIVE models.

OLIVE-based representations retain significantly more signal-level detail, as evidenced by lower reconstruction errors and improved perceptual quality. The functional persistence of the decoder post-pretraining further sets OLIVE apart.

Theoretical and Practical Implications

OLIVE demonstrates that incorporating explicit signal-level generative supervision during SSL pre-training leads to representations that support both high-fidelity generation and robust downstream discrimination. This contrasts with traditional SSL methods, which may discard signal information crucial for reconstruction during latent compression. The decomposition of invariance and reconstructivity via architectural separation (early vs. late layers) provides insight into the representational tradeoffs inherent in SSL.

Practically, OLIVE enables a single model to serve a wider downstream task portfolio—including voice conversion, enhancement, and speech generation—without sacrificing recognition accuracy. The preservation of a functional vocoder as part of the pre-trained model also enables efficient deployment of generation tasks in downstream settings, offering the basis for effective universal speech analysis-synthesis backbones.

Theoretically, this work points to the benefit of aligning SSL objectives more closely with the inherent duality of speech as both a communicative and generative medium. It raises questions regarding optimal invariance and information retention trade-offs and how these might adapt across languages, domains, and architectures.

Future Directions

• Scaling and Multilingual Pre-training: Extending OLIVE to larger models and diverse datasets to evaluate robustness and generalization.
• Task-Specific Augmentations: Tailoring augmentations for application-specific invariance (e.g., environmental robustness, accent generalization).
• Broader Generative Evaluation: Applying OLIVE representations to additional generative scenarios, including zero-shot text-to-speech and cross-lingual synthesis.
• Decoder Reusability and Adaptation: Exploring transferability of the integrated vocoder and fine-tuning for in-domain generation tasks.

Conclusion

OLIVE systematically unifies self-supervised analysis and synthesis objectives for speech, producing representations that maintain discriminative strength while preserving the signal-level information required for high-quality generation. By combining view-augmented latent prediction and adversarial waveform reconstruction under joint optimization and architectural separation, OLIVE simultaneously advances performance for recognition and synthesis tasks, broadening the practical scope of self-supervised speech representation models.