Textless Spoken Language Processing

• Textless spoken language processing is a paradigm that bypasses text, using self-supervised learning to extract and quantize speech into discrete units for robust analysis.
• It employs advanced architectures such as Transformer encoders with task-prompt vectors and unit language constructions to enable end-to-end speech-to-speech translation and understanding.
• Recent advances integrate prosody, emotion, and speaker style preservation, achieving notable improvements in BLEU, MOS, and SLU metrics while supporting under-resourced languages.

Textless spoken language processing refers to computational methods that operate on speech audio directly, eschewing intermediate text-based representations such as orthographic transcripts, phoneme sequences, or explicit linguistic annotations. This paradigm aims to enable speech analysis, generation, translation, and understanding in scenarios where written resources are limited or unavailable, and to leverage the full richness of spoken communication—including prosody, emotion, and paralinguistic cues—often lost in text-based pipelines. The field has seen rapid methodological advances, driven by developments in self-supervised learning, discrete speech unit discovery, and generative modeling.

1. Discrete Speech Unit Discovery and Representation

Textless spoken language processing fundamentally relies on extracting compact, content-centric representations from continuous speech. The dominant approach leverages self-supervised models such as HuBERT, wav2vec 2.0, and their multilingual variants to produce frame-level hidden representations, which are then quantized into discrete units via k-means clustering or learned codebooks (e.g., VQ-VAE) (Lee et al., 2021, Kim et al., 2023, Duret et al., 2023).

A typical process involves:

• Feature extraction: Downsampling audio into short frames (e.g., 20 ms), extracting continuous embeddings $h_t$, often from mid-to-late layers of SSL models.
• Quantization: Assigning each $h_t$ to a discrete codeword $u_t$ in a codebook of size $K$ (commonly 1000) via $u_t = \arg\min_{k} \|h_t - c_k\|^2$.
• Post-processing: Collapsing consecutive duplicate units to capture phone-like or syllable-like structure, and potentially grouping units into larger "unit-words" (see unit language below).
• Normalization and speaker-invariance: Applying approaches such as CTC-fine-tuned speech normalizers to reduce acoustic variation due to channel, speaker, or accent, yielding "norm-units" that encode lexical content robustly (Lee et al., 2021).

The resulting sequences of discrete units serve as a spoken analogue of text for downstream modeling, enabling both unit-level language modeling and sequence prediction within entirely speech-based pipelines (Wu et al., 2023, Roy et al., 3 Jul 2025).

2. Core Architectures and Modeling Frameworks

Textless spoken language processing has introduced several novel model architectures across understanding, generation, and translation tasks:

• Textless Speech-to-Speech Translation (S2ST): End-to-end models directly map source-language audio to target-language audio via intermediate discrete speech units. Architectures typically involve a self-supervised speech encoder, unit extractor, sequence-to-sequence unit translation model (e.g., Transformer), and neural unit vocoder (e.g., HiFi-GAN) (Lee et al., 2021, Zhang et al., 21 May 2025, Kim et al., 2023).
• Unit Language: To impose text-like structure and facilitate long-range alignment, "unit-words"—n-gram groupings of contiguous speech units optimized by language-model likelihood—serve as pseudo-tokens, enabling multi-task learning across cross-modal (CM) and cross-lingual (CL) objectives within a unified framework (Zhang et al., 21 May 2025).
• Task-Prompted Multitask Models: To resolve conflicts between auxiliary tasks (e.g., unit reconstruction for CM and translation for CL), learnable prompt vectors are injected into Transformer encoders, guiding layers to specialize in either noise filtering or semantic alignment (Zhang et al., 21 May 2025).
• Joint Semantic and Acoustic Modeling: Flow-SLM jointly predicts future discrete semantic tokens and frame-level continuous acoustic vectors via a flow-matching objective, bridging linguistic likelihood and fine acoustic detail in generation (Chou et al., 12 Aug 2025).

Model/Component	Input Representation	Intermediate Representation	Output/Task
S2ST (baseline, S2UT)	Audio (log-mel or waveform)	Discrete units (k-means)	Unit seq.→unit vocoder→waveform
Unit Language S2ST	Audio & units	N-gram unit-words	S2ST with multi-task guidance
Flow-SLM	Audio, past frames	Discrete tokens + acoustic	Speech generation/prompted synthesis

3. Applications and Benchmarks

Textless spoken language processing enables a spectrum of applications previously dependent on text resources:

• Speech-to-Speech Translation: State-of-the-art textless S2ST achieves SacreBLEU scores in the 15–25 range on Europarl-ST and VoxPopuli, with normalized units providing up to +6 BLEU over unnormalized baselines and tightening the gap to cascaded ASR–MT–TTS pipelines (Lee et al., 2021, Kim et al., 2023, Zhang et al., 21 May 2025).
• Speech Resynthesis and Compression: Unit-based systems provide strong intelligibility (ASR WER ≈ 8% at 200 kb/s for HuBERT-200), and decouple bitrate and speaker information, conserving lexically salient content while neutralizing identity (Kharitonov et al., 2022).
• Spoken Language Understanding (SLU): Textless SLU using unit-sequence auxiliary guidance surpasses pure end-to-end models lacking transcripts, yielding +3 to +5 F1 improvements across multiple datasets and enhancing few-shot robustness (Wu et al., 2023).
• Dependency Parsing and Semantic Tasks: CTC-based textless models directly predict syntactic or semantic sequences from speech, preserving prosodic cues for structural disambiguation (Kando et al., 2024).
• Dialogue Generation: Dual-tower Transformers trained on conversational audio, or hybrid approaches integrating text-based LLMs and speech LLMs, produce naturalistic, turn-synchronized dialogues without text supervision (Mai et al., 8 Jan 2025, Lu et al., 1 Jan 2025, Nguyen et al., 2022).

Task	Key Benchmark	Notable Metric (Typical Value)	Reference
S2ST (Es→En, norm-units)	Europarl-ST	BLEU +5.7 over baseline (18.8)	(Lee et al., 2021)
SLU (SLURP, unit-guided)	Intent/Slot Filling	+4.7 SLU-F1 over baseline, 67.9%	(Wu et al., 2023)
S2ST (unit language, +prompts)	VoxPopuli (4 langs)	+1.2 avg BLEU (to 21.5)	(Zhang et al., 21 May 2025)
Dialogue Generation	Fisher	Naturalness MOS 3.7–4.1	(Nguyen et al., 2022, Lu et al., 1 Jan 2025)

4. Design Trade-offs and Feature Selection

Key design decisions in textless processing critically affect downstream performance:

• Choice of Unit Granularity and Encoder Layer: Translation efficacy increases with finer quantization (e.g., $k=1024$ in HuBERT Base, layer 6) but does not align with resynthesis quality (MOS), which may peak at lower cluster counts (e.g., $k=512$) (Duret et al., 2024). Moderate negative correlation ($r \approx -0.47$) is observed between unit-based ASR CER and S2ST BLEU.
• Unit Language Construction: Fixed-length $K$-mer groupings for unit-words strike a trade-off between sequence compression and alignment expressivity; $K=3$ yields significant compression (from ≈358 raw units to ≈130 tokens) while maintaining translation capability (Zhang et al., 21 May 2025).
• Task Interference: Simultaneous inclusion of CM and CL objectives induces destructive interference during multi-task training; learnable prompt vectors reliably disentangle these tasks, enabling co-training (Zhang et al., 21 May 2025).
• Acoustic Detail vs. Semantic Robustness: Flow-SLM improves acoustic quality and prosody but still lags large text-based models in abstract semantic modeling; predicting multiple future semantic tokens ($k=4$) is essential for preserving linguistic structure (Chou et al., 12 Aug 2025).

5. Advances in Expressivity, Robustness, and Multilinguality

Recent work addresses critical limitations of prior textless systems:

• Expressivity Transfer: Augmenting S2ST with language-agnostic emotion embeddings and explicit prosody (pitch, duration) control yields improved MOS and MUSHRA scores, outperforming TTS baselines in expressivity while maintaining BLEU (Duret et al., 2023).
• Speaker and Style Preservation: Multi-task decoders (e.g., MSLM) couple preservation of speaker style (measured by WavLM embedding similarity, ≈0.40) with high translation BLEU (e.g., 24.78 in Es→En) in monolithic models (Peng et al., 2024).
• Scalability to Under-Resourced Languages: Work targeting 24 unwritten and low-resource languages demonstrates fully textless architectures—spectrogram, wavelet, scalogram, and unit models—augmented by Multiscale Audio-Semantic Transforms (MAST) and fractional diffusion, offering robustness under noise and domain shift (Tembine et al., 3 Jun 2025).
• Few-Shot and In-Context Learning (ICL): Warmup schemes with prompt tuning equip small textless LMs with genuine ICL capacity, yielding 35–60% accuracy on unseen speech classification tasks—often matching SVMs trained on the same demonstrations (Hsu et al., 2023).

6. Toolkits, Evaluation Paradigms, and Future Directions

• Modular Toolkits: textless-lib provides APIs for the entire pipeline—self-supervised encoding, quantization, vocoding, and downstream modeling—with support for arbitrary unit vocabularies and languages (Kharitonov et al., 2022).
• Evaluation: Standard metrics include BLEU (for S2ST), MOS (for naturalness), ABX (for phonetic discriminability), Word/Phoneme Error Rate (for resynthesis), F1/Intent Accuracy (for SLU), and embedding-based similarity (for style) (Duret et al., 2024, Wu et al., 2023, Peng et al., 2024).
• Limitations: Textless S2ST still lags state-of-the-art text-based models by ∼10 BLEU (Zhang et al., 21 May 2025). Human evaluation of prosody, semantic adequacy, and intelligibility is incomplete. Robust, real-time and streaming generation remains challenging (Mai et al., 8 Jan 2025).
• Research Areas: Extensions include richer neural unit merging, curriculum-driven task scheduling, style and prosodic modeling, direct joint learning of units and translation, human-in-the-loop evaluation, and cross-modal (audio–vision) textless retrieval (Zhang et al., 21 May 2025, Xie et al., 9 Sep 2025, Chou et al., 12 Aug 2025).

7. Significance, Challenges, and Prospects

Textless spoken language processing unlocks new possibilities for speech applications in scenarios where text, scripts, and annotated resources are unavailable or irrelevant. It enables end-to-end pipelines for translation, synthesis, understanding, and dialogue in unwritten or marginalized languages, and preserves spoken phenomena integral to communication. Continuing challenges include bridging the semantic gap with text-based models, explicitly modeling paralinguistic and prosodic detail, optimizing unit representations for specific downstream tasks, and achieving scalability and robustness under low-data and noisy conditions. The field is converging towards unified, modular architectures that leverage self-supervised learning, discrete units, and multi-objective optimization to approach the performance and flexibility of traditional text-mediated pipelines while operating fully in the audio domain (Zhang et al., 21 May 2025, Lee et al., 2021, Kim et al., 2023, Tembine et al., 3 Jun 2025).