Non-Causal Self-Supervised Speech Encoder

• The paper showcases a non-causal, bidirectional architecture that uses full-sequence self-attention to aggregate past and future context for robust speech representation.
• It employs self-supervised contrastive objectives along with data augmentation and k-NN positive pair mining to effectively capture phonetic similarities without labeled data.
• Non-causal encoders achieve state-of-the-art performance in tasks like query-by-example, speech inpainting, and speaker verification, though they face challenges in streaming and low-latency applications.

A non-causal self-supervised speech encoder is a neural architecture for generating robust, high-level representations of speech sequences using self-supervision, with the defining property that each output embedding can leverage both past and future context within a given input window. These models eschew autoregressive constraints, enabling bidirectional modeling via mechanisms such as full-sequence self-attention. This paradigm has become foundational for state-of-the-art speech representation learning, query-by-example retrieval, speech inpainting, and other downstream tasks.

1. Architectural Principles of Non-Causal Self-Supervised Speech Encoders

Non-causal self-supervised speech encoders typically exhibit two unifying properties: (a) their core sequence model is fully bidirectional, and (b) their self-supervised training objective enables learning without labeled data. Architectures instantiate these by stacking non-causally masked self-attention blocks (e.g., Transformers, GALR), often preceded by convolutional or LSTM-based feature encoders.

The model in "Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning" employs a LayerNorm and 1D convolution front-end (512 channels, kernel size 4), followed by sinusoidal positional encoding and a single Transformer block with four attention heads (model dim 512). Crucially, this Transformer applies bidirectional self-attention—no masking—over the entire input subsequence. For fixed-size output, a temporal max-pooling layer reduces the encoded sequence to a single vector per input segment. The end-to-end structure is non-causal: to emit an embedding $z$, the model must see the full $80\,\mathrm{ms}$–$1\,\mathrm{s}$ window (Algayres et al., 2022).

"Wav2vec-C" uses an LSTM-based feature encoder ($3 \times 768$-dim LSTM), product quantization, and a context module comprising a stack of five standard Transformers (dim 1024, 16 heads). There is no causal attention mask in the context stack, enabling bidirectional encoding of temporal dependencies. Segment-level masking, implemented via SpecAugment, further enforces context exploitation (Sadhu et al., 2021).

HuBERT (as used in speech inpainting (Asaad et al., 2024)) adopts a convolutional prenet for frame extraction, followed by a deep Transformer stack (24 layers, hidden size 1024). Critically, all self-attention blocks are fully non-causal.

Contrastive Separative Coding (CSC) utilizes globally-attentive locally-recurrent (GALR) blocks with global self-attention (unmasked) and bi-directional local RNNs, followed by global cross-attention and a unidirectional LSTM aggregator. All context modeling up to the global aggregation step is non-causal (Wang et al., 2021).

2. Self-Supervised Training Objectives and Positive Pair Mining

Non-causal speech encoders rely on self-supervised objectives that encourage the model to build representations predictive of masked or transformed parts of the signal, without explicit supervision. Most commonly, the NTXent (InfoNCE) contrastive loss is employed: each anchor embedding is contrasted with its positive partner and other negatives.

In (Algayres et al., 2022), positive pairs are curated either by on-the-fly data augmentation (time stretching to yield phonetic-invariant subsequences) or via iterative k-nearest neighbor (k-NN) mining, where subsequences with similar embeddings are presumed to share phonetic content. The NTXent loss is defined as

$$L_{nce}(z_i) = - \log \frac{\exp(\mathrm{sim}(z_i, z^+_i) / \tau)}{\sum_{j=1,\dots,2n,\,j\neq i} \exp(\mathrm{sim}(z_i, z_j) / \tau)}$$

where $\mathrm{sim}$ denotes cosine similarity and $\tau$ the temperature hyperparameter.

"Wav2vec-C" formulates its loss as a sum of contrastive, quantizer regularization, and consistency (reconstruction) losses. The contrastive component is again InfoNCE-style, with temporal masking and discrete latent targets obtained via vector quantization and/or Gumbel-Softmax. The commitment and diversity losses ensure full codebook usage, regularized by a VQ-VAE-style consistency network (Sadhu et al., 2021).

HuBERT (Asaad et al., 2024) uses a masked prediction pretext task: random contiguous frame ranges are replaced by a learned mask token, and the model predicts quantized teacher targets from the unmasked input using a cross-entropy loss. This encourages the model to integrate information from the full temporal context.

CSC (Wang et al., 2021) introduces a contrastive separative loss that maximizes the mutual information between per-segment representations and a global speaker vector, forming positives via class membership (speaker identity) and obviating the need for explicit negative sampling.

3. Positive Pair Mining: Data Augmentation and k-NN Iteration

The effectiveness of contrastive learning depends critically on the selection of positive pairs. In (Algayres et al., 2022), the model is initially bootstrapped using stretch-invariant data augmentation: vocal segments are time-stretched with independently sampled scaling factors ($\in[0.5, 1.8]$), ensuring the resultant positive pairs are robust to local temporal deformations.

After this pretraining, k-NN iteration is employed: frame-level embeddings of subsequences from all vocal segments are indexed (with FAISS) using cosine distance. For each subsequence, non-overlapping nearest neighbors are selected, pruned by non-maximal suppression and cosine similarity thresholding (target 50% coverage), and these (anchor, neighbor) tuples are treated as positive training pairs for the next training round. The procedure is repeated for up to two cycles, converging rapidly. This self-labelling strategy exploits the non-causal context encoder’s ability to discover fine-grained, language-independent phonetic similarity without supervision (Algayres et al., 2022).

4. Integration with Downstream Tasks and Evaluation Protocols

Non-causal self-supervised speech encoders attain state-of-the-art results on diverse downstream tasks due to their access to bidirectional context over input segments:

• Query-by-Example: Subsequence embeddings (80 ms–1 s) are used for large-scale nearest-neighbor retrieval. On LibriSpeech, the model from (Algayres et al., 2022) improves mean average precision (MAP) from a max-pooling baseline (0.07) and CAE-Siamese state-of-the-art (0.212) to 0.399 after k-NN training; the supervised topline reaches 0.784.
• Spoken Term Discovery: Using ZeroSpeech NED/COV metrics, discovered segment pairs yield operating points that strictly dominate prior submissions across all tested languages (Algayres et al., 2022).
• Speech Inpainting: HuBERT-based non-causal encoding, combined with the HiFiGAN vocoder, reconstructs waveforms with masked segments up to 200 ms (and sometimes 400 ms) with high intelligibility and perceptual naturalness. Two adaptation schemes (encoder-fine-tuning vs. decoder-fine-tuning) perform best respectively for single- and multi-speaker data (Asaad et al., 2024).
• Speaker Verification under Adverse Conditions: CSC yields significantly lower EER (≈6.2%) and higher AUC (0.97) on overlapped Libri-2mix compared to SincNet_mix and CE_mix (≈12.5–11.3% EER, 0.90–0.92 AUC). Ablations confirm the value of cross-attention and non-causal context modeling (Wang et al., 2021).

5. Analysis of Non-Causality vs. Streaming and Implications

The bidirectional (non-causal) structure allows the models to extract maximally informative representations by aggregating both past and future context. In (Algayres et al., 2022), the Transformer encoder and subsequent pooling necessitate the entire segment as input prior to embedding emission; a causal alternative is not explored. In "Wav2vec-C," the context network’s full self-attention similarly disables streaming operation—each $c_t$ attends to all temporal positions, maximizing context for contrastive discrimination but precluding on-line processing (Sadhu et al., 2021).

HuBERT’s masked prediction setup inherently precludes strict causality, as any input frame can be masked irrespective of location, and all attention is unmasked (Asaad et al., 2024). CSC’s GALR and cross-attention stacks aggregate over the full utterance; only the LSTM aggregator is causal, but preceding context encoding remains non-causal (Wang et al., 2021).

A plausible implication is that while non-causal encoders offer superior performance for offline tasks requiring holistic sequence understanding, they are less suited to latency-constrained streaming settings. The literature suggests that introducing causal masking or chunk-based inference with lookahead could provide a trade-off, but empirical studies on streaming adaptation remain to be conducted (Algayres et al., 2022).

6. Architectural Variations and Representational Characteristics

A representative sample of non-causal self-supervised speech encoders and their design elements is given in the following table:

Model	Main Encoder	Attention	Pooling/Aggregation
(Algayres et al., 2022)	Conv1D + Transformer	Bidirectional	Max-over-time
(Sadhu et al., 2021)	LSTM + Transformer (context)	Bidirectional	Token (frame)-wise
(Asaad et al., 2024)	Conv prenet + Transformer	Bidirectional	Frame-wise
(Wang et al., 2021)	GALR + Cross Attention	Bidirectional	LSTM/global pooling

These architectures universally exploit unmasked self-attention or global attention mechanisms in the core feature encoder, ensuring non-causality, while combining local recurrence or convolutional modules for feature extraction. Downstream modules (e.g., LSTM aggregators, vocoders) may optionally be causal, but the main representational bottleneck is computed in a non-causal fashion.

Codebook quantization (e.g., Gumbel-Softmax, k-means) and VQ-VAE–style consistency networks further diversify the representations, as detailed in "Wav2vec-C" (Sadhu et al., 2021). Data shows that including a reconstruction loss dramatically expands codebook usage (up to 100%) and correlates with higher downstream task performance.

7. Current Limitations and Future Directions

Current non-causal self-supervised speech encoders demonstrate strong performance for offline and batch inference, but streaming and low-latency applications remain out of reach due to global context dependencies. Masking strategies may blur fine-grained prosodic details, and codebook discretization can limit signal fidelity in generation and inpainting tasks for gaps >400 ms (Asaad et al., 2024). Blind inpainting and structured missing-data imputation are open research areas.

Prospective directions include jointly learning streaming-capable variants (by combining causal and non-causal SSL losses), developing hybrid discrete-continuous representations to capture nuanced signal attributes, and extending non-causal architectures to other sequential modalities (e.g., music, multimodal settings) (Asaad et al., 2024).

---

Key references:

• "Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning" (Algayres et al., 2022)
• "Wav2vec-C: A Self-supervised Model for Speech Representation Learning" (Sadhu et al., 2021)
• "Fill in the Gap! Combining Self-supervised Representation Learning with Neural Audio Synthesis for Speech Inpainting" (Asaad et al., 2024)
• "Contrastive Separative Coding for Self-supervised Representation Learning" (Wang et al., 2021)