ASR with Discretized Input

Updated 13 October 2025

ASR with Discretized Input is a paradigm that converts continuous acoustic signals into discrete tokens, enabling efficient, robust, and privacy-preserving speech recognition.
It utilizes self-supervised feature extraction and quantization methods, such as k-means clustering and neural codecs, to compress data and reduce model complexity.
The approach integrates advanced architectures like transformer decoders and LLM-based speech adapters, achieving competitive accuracy and improved cross-domain performance.

Automatic Speech Recognition (ASR) with Discretized Input refers to ASR system architectures and learning paradigms where the continuous-valued acoustic input or intermediate features are transformed into sequences of discrete tokens before being processed by the recognizer or downstream components. This trend, increasingly prominent with advances in self-supervised learning and neural codec technology, enables practical benefits for scalability, privacy, model efficiency, cross-domain robustness, and integration with natural language processing techniques.

1. Rationale for Discretization in ASR

The foundational motivation for discretizing speech input in ASR stems from several convergent factors: the need for more compact and privacy-preserving input representations, the capability to exploit powerful NLP methodologies on tokenized sequences, and the technical facilitation of robust learning across heterogeneous domains. Discrete speech representations can be constructed via unsupervised clustering (e.g., k-means over SSL features (Chang et al., 2023, Shon et al., 13 Jun 2024)), learned quantization in neural codecs (Dhawan et al., 3 Jul 2024), or post-processing steps such as deduplication and subword modeling (Chang et al., 2023, Sukhadia et al., 19 Jun 2024). Discretization aligns well with the finite-alphabet signal assumption, relevant for both classical Bayesian inference (Dai et al., 2019) and modern neural speech processing.

Discrete tokens often abstract away speaker identity and other paralinguistic details (Aloufi et al., 2021, Sukhadia et al., 19 Jun 2024), compress data size for efficient computation and transfer (Chang et al., 2023), facilitate structured input for transformer decoders (Song et al., 2020), and serve as a bridge to LLM architectures for spoken language understanding (Shon et al., 13 Jun 2024).

2. Construction of Discretized Speech Representations

Discrete input is synthesized via a pipeline involving self-supervised feature extraction, quantization, and optional sequence post-processing:

Self-Supervised Feature Extraction: Models such as WavLM, HuBERT, CPC, and wav2vec2 produce high-dimensional embeddings that preserve both phonetic and some linguistic properties (Chang et al., 2023, Shon et al., 13 Jun 2024, Sukhadia et al., 19 Jun 2024).
Quantization Mechanisms: Common techniques include k-means clustering over feature vectors to produce token indices (Chang et al., 2023, Sukhadia et al., 19 Jun 2024, Shon et al., 13 Jun 2024), residual vector quantization (RVQ), and finite scalar quantization (FSQ) within neural audio codecs (Dhawan et al., 3 Jul 2024).
Embedding and Sequence Processing: Token sequences may undergo deduplication (collapsing consecutive identical tokens), subword modeling (e.g., Sentencepiece unigram segmentation), and meta-token formation to reduce sequence length and redundancy (Chang et al., 2023, Sukhadia et al., 19 Jun 2024).
Privacy Filtering: Discretization, by omitting fine-grained continuous features, inherently suppresses paralinguistic and speaker-specific information, lowering privacy leakage to the level of random guessing for trained classifiers (Aloufi et al., 2021).

The process is exemplified mathematically by:

$z_t = \arg\min_{j \in \{1,...,K\}} \lVert h_t - \mu_j \rVert$

where $h_t$ is the feature vector at time t, and $\mu_j$ the centroid for cluster j (Shon et al., 13 Jun 2024, Chang et al., 2023).

3. Architectures for Discretized Input in ASR

ASR systems utilizing discretized input typically employ one or more of the following architectural variants:

Joint CTC/Attention Models: Discrete token sequences are embedded and provided to an encoder-decoder ASR network with alignment managed via Connectionist Temporal Classification (CTC) and attention mechanisms (Chang et al., 2023, Sukhadia et al., 19 Jun 2024).
FastConformer/Transducer Systems: Acoustic codes from neural codecs are mapped to embeddings and supplied to advanced Conformer-based architectures for robust end-to-end speech recognition (Dhawan et al., 3 Jul 2024).
LLMs with Speech Adapters: Discrete Speech Units (DSU), post-quantization, are remapped by speech adapters to the token embedding space of LLMs, enabling direct speech-to-text or speech understanding tasks (Shon et al., 13 Jun 2024).
Alternating Bayesian Inference Schemes: Sparse Bayesian Learning (SBL) frameworks integrate discretization enforcing priors for finite-alphabet signal recovery, supported by variational Bayesian inference and alternating optimization (Dai et al., 2019).

Embedding strategies, aggregation approaches (e.g., stacking and averaging across codebooks in RVQ/FSQ systems), and codebook initialization significantly affect final ASR performance (Dhawan et al., 3 Jul 2024).

4. Performance, Efficiency, and Robustness

Key empirical findings from recent literature include:

System/Domain	Data Type	Model Size / Savings	WER/CER Performance	Additional Benefits
K-means on WavLM tokens (Chang et al., 2023)	Discretized tokens	23 min/epoch, ~60% seq length reduction	WER: 3.1% (test-clean), 6.9% (test-other)	100× storage reduction
Codec-ASR RVQ (Dhawan et al., 3 Jul 2024)	Acoustic codes	<140M params	CER: 21% (multi-lingual benchmark)	Surpasses Encodec and XLSR-128
Children’s ASR (Sukhadia et al., 19 Jun 2024)	Discrete tokens	40M params (~83% size reduction)	ΔWER ≈ 0.67–0.95 vs continuous	Maintains generalization, privacy
DSU-LLM (Shon et al., 13 Jun 2024)	DSU/MFCC	Variable	Robust WER and BLEU across domains	Length reduction, cross-domain
Privacy-preserving ASR (Aloufi et al., 2021)	Discrete phonemes	Modular, independent modules	WER within range of continuous system	Paralinguistic leakage ≈ random

Results consistently indicate that discretized input yields competitive ASR accuracy—often within 0.67–1.0 WER points of conventional continuous-feature front-ends—while facilitating substantial computational and storage benefits. Discrete-code models demonstrate enhanced privacy, efficient deployment in low-resource or edge settings, and strong robustness to cross-domain or unseen condition generalization (Sukhadia et al., 19 Jun 2024, Shon et al., 13 Jun 2024).

5. Advanced Algorithmic and Statistical Techniques

Optimizing the entire discretized-ASR pipeline entails several algorithmic considerations:

Sparse Bayesian Learning (SBL): Discretization enforcing priors integrated in SBL allow recovery of finite-alphabet signals under uncertainty, with variational inference and alternating updates for hierarchical parameters (Dai et al., 2019). Ideal delta-function priors are approximated with Gaussians of large precision. The generalized approximate message passing (GAMP) algorithm offers further computational improvements when matrix assumptions are met.
Self-Supervised Pre-training and Data Selection: Discrete tokens from SSL quantizers support contrastive data selection strategies, improving relevancy and domain-matching for pre-training (Lu et al., 2022). The probabilistic scoring function

$\text{Score}(q) = \frac{\log P_T(q) - \log P_G(q)}{\text{length}(q)}$

where $P_T$ and $P_G$ are LM probabilities on target/general domains, is used for efficient unsupervised data curation.

Embedding Layer Initialization and Aggregation: RVQ/FSQ codebook-based embedding initialization and code aggregation (averaging vs stacking) drive improvements in recognition accuracy and robustness (Dhawan et al., 3 Jul 2024).
Subword and Meta-token Modeling: NLP approaches (such as Sentencepiece unigram segmentation) further compress token sequences and can enhance error resilience (Chang et al., 2023, Sukhadia et al., 19 Jun 2024).

6. Privacy, Configurability, and Application Scope

Discretization plays a pivotal role in augmenting privacy preservation by minimizing overrepresentation of speaker identity and paralinguistic traits (Aloufi et al., 2021, Sukhadia et al., 19 Jun 2024). Configurable privacy through tuning of the discretization granularity and post-processing parameters allows domain-specific trade-offs between linguistic utility and privacy risk (Aloufi et al., 2021). These principles extend application potential to:

Voice assistants in privacy-sensitive scenarios.
Children's speech recognition, medical, and legal transcription (Sukhadia et al., 19 Jun 2024).
Low-resource and edge device deployment (Chang et al., 2023).
Multilingual, cross-domain transfer and universal spoken language understanding (Dhawan et al., 3 Jul 2024, Shon et al., 13 Jun 2024).
Upstream generalization for spoken question answering without explicit ASR supervision (Shon et al., 13 Jun 2024).

7. Limitations, Challenges, and Future Directions

Principal limitations arise from hyperparameter sensitivity (number of clusters, codebook size, aggregation methods), performance gaps on noisy or challenging subsets, and dependency on quantizer quality (Chang et al., 2023, Lu et al., 2022, Dhawan et al., 3 Jul 2024). Accuracy may degrade slightly in difficult conditions (e.g., non-i.i.d. matrices for SBL-GAMP (Dai et al., 2019), large-alphabet scenarios, or shallow DSU extraction (Shon et al., 13 Jun 2024)). Future work is expected to focus on:

Adaptive and ensemble discretization strategies (Chang et al., 2023).
More sophisticated quantization and neural codebook training (Dhawan et al., 3 Jul 2024).
Joint optimization of sequence processing pipelines (Sukhadia et al., 19 Jun 2024).
Extending to broader language coverage and challenging real-world conditions (Dhawan et al., 3 Jul 2024).
Integrating acoustic and semantic codes for multi-task learning (Dhawan et al., 3 Jul 2024).
Pre-training paradigms for speech–text foundational models (Shon et al., 13 Jun 2024).

In summary, ASR with discretized input is a rapidly emerging paradigm that leverages advances in self-supervised learning, quantization, and modularization to deliver privacy, efficiency, and robust performance across diverse speech processing tasks. The approach encompasses both technically rigorous Bayesian frameworks and scalable neural models, promising widespread impact for next-generation speech recognition architectures.