Voxtral Realtime ASR

• Voxtral Realtime is a real-time automatic speech recognition system that uses delayed streams modeling with a causal encoder and adaptive delay conditioning to synchronize audio and text.
• It achieves near offline-quality transcription at sub-second latency, with competitive word error rates across 13 languages and diverse benchmarks.
• The open-source framework supports flexible deployment modalities, enabling integration in real-time applications via API and local processing.

Voxtral Realtime is a natively streaming automatic speech recognition (ASR) model architected to achieve offline-quality transcription at sub-second latency in a fully open-source, multilingual framework. It utilizes the Delayed Streams Modeling (DSM) paradigm with a causal audio encoder and adaptive delay conditioning to align audio and textual streams precisely in time. Training is conducted at scale across 13 languages to ensure robustness and consistent performance. Voxtral Realtime represents a modern approach to real-time ASR, providing a publicly available solution that supports a range of deployment modalities and application requirements (Liu et al., 11 Feb 2026).

1. Model Architecture and Delayed Streams Modeling

Voxtral Realtime is built on the DSM framework, which synchronizes audio and text via explicitly constructed dual streams. The system operates on frame-aligned audio embeddings $(a_1, a_2, \ldots, a_T)$ and token sequences $(y_1, y_2, \ldots, y_T)$ sampled at a consistent frame rate $f_a$. The decoder is trained to emit a token per frame, using special tokens $[P]$ (padding) for frames without new words and $[W]$ (word boundary) to indicate completed words:

• If no word boundary has been reached, $\hat{y}_t = [P]$.
• After a word completes and the allowed delay $\tau$ elapses, $\hat{y}_t = [W], w_1, w_2, \ldots$ where the $w$ are subword tokens.

The architecture comprises the following components:

• Causal audio encoder: 16 kHz input → 128-bin log-Mel spectrogram (10 ms hop), passed through two 1D convolutions (causal, kernel size 3). Each encoder output at time $t$ depends on frames $(y_1, y_2, \ldots, y_T)$0 to $(y_1, y_2, \ldots, y_T)$1, ensuring strict causality.
• Transformer backbone: 32 causal self-attention layers with a 750-frame (15 s) sliding window and output rate of 50 Hz (20 ms embedding stride). Additional features include RMSNorm, SwiGLU FFNs, and rotary position embeddings.
• Decoder: Receives MLP-downsampled audio features along with previous token embeddings, augmented by adaptive conditioning on the delay parameter $(y_1, y_2, \ldots, y_T)$2.

This architecture ensures that all computations respect real-time streaming constraints, in contrast to typical offline models that rely on bidirectional attention or chunk-based adaptation (Liu et al., 11 Feb 2026).

2. Adaptive Delay-Conditioned Normalization

A defining feature of Voxtral Realtime is its use of Adaptive RMS-Norm in the language decoder, which enables dynamic adjustment to arbitrary token emission delays without retraining. For model dimension $(y_1, y_2, \ldots, y_T)$3, delay $(y_1, y_2, \ldots, y_T)$4, and feedforward block input $(y_1, y_2, \ldots, y_T)$5 in decoder layer $(y_1, y_2, \ldots, y_T)$6:

1. The attention branch computes $(y_1, y_2, \ldots, y_T)$7 and $(y_1, y_2, \ldots, y_T)$8.
2. The feedforward branch applies a learned scaling $(y_1, y_2, \ldots, y_T)$9:

$f_a$0

$f_a$1

where $f_a$2 denotes element-wise multiplication.

This mechanism allows the same model to be run at a range of delays---from low-latency 80 ms up to 2400 ms---with no additional training, providing flexibility for deployment on varying hardware or in environments with different responsiveness requirements (Liu et al., 11 Feb 2026).

3. Training Regimen and Data Composition

Voxtral Realtime is pretrained on approximately 370,000 hours of speech data per batch, spanning 13 languages (Arabic, German, English, Spanish, French, Hindi, Italian, Japanese, Korean, Dutch, Portuguese, Russian, and Chinese). Key aspects of the training design include:

• Uniform language sampling: Each batch is balanced across all languages, promoting multilingual generalizability.
• Delay-sampling: During training, the target delay $f_a$3 is sampled uniformly from the set $f_a$4 ms to ensure robustness across latency settings.
• Objective function: The primary loss is framewise cross-entropy between predicted tokens and aligned targets, supplemented by a z-loss regularizer to prevent logit explosion:

$f_a$5

$f_a$6

$f_a$7

The design eliminates the need for forced alignments or non-streaming supervision and directly learns robust audio-to-text alignments (Liu et al., 11 Feb 2026).

4. Inference Pipeline and Latency Analysis

The inference path is fully pipelined for real-time operation:

• Audio ingestion: 80 ms frames serve as the atomic input unit.
• Encoder and decoder scheduling: The causal encoder outputs an embedding every 20 ms, and the decoder generates subword tokens at 80 ms intervals (12.5 Hz).
• Token emission: Each decoder step produces the next token or padding, handling the emission of words as soon as delay constraints allow.

Inference latency per frame on NVIDIA A100 (batch size 1) is dominated by: | Component | Time (ms) | |----------------------------|------------| | 80 ms audio buffer | 80 | | Feature extraction/conv | ~2 | | Causal encoder (self-attn) | 4–6 | | Adapter downsample | <1 | | Decoder step (3.4B param) | 8–10 | | Net overhead (I/O, queue) | ~2 |

The total overhead beyond $f_a$8 is typically 17–19 ms. The observed real-time factor (RTF) is approximately 0.24 at 480 ms delay---meaning one second of audio requires 240 ms of compute (Liu et al., 11 Feb 2026).

5. Transcription Performance and Multilingual Capabilities

At a delay of 480 ms, Voxtral Realtime achieves word error rates closely matching the best offline systems on both short and long English test segments, as well as standard multilingual benchmarks:

Model	En-Short	En-Long	FLEURS	MCV
Whisper	8.39 %	7.97 %	8.23 %	14.25 %
Scribe v2	7.33 %	7.43 %	8.34 %	20.85 %
Voxtral	8.47 %	7.73 %	8.72 %	15.24 %

Voxtral Realtime notably outperforms prior open-source streaming models (DSM, Nemotron) at similar delays and delivers matching quality to Whisper on the majority of tested conditions, but with strict streaming guarantees (Liu et al., 11 Feb 2026).

6. Deployment, API Integration, and Licensing

Voxtral Realtime's model weights are released under the Apache 2.0 license, with prebuilt checkpoints (e.g., mistralai/Voxtral-Mini-4B-Realtime-2602) hosted for direct loading. The runtime is optimized for deployment via the vLLM RealtimeSession API, supporting:

• Resumable streaming using paged attention for managing encoder-decoder cache rate mismatches.
• Token-by-token output over WebSocket or local Python environments, enabling real-time downstream integration.
• Inference at fixed or adaptive delay using a dialable $f_a$9 parameter via Ada RMS-Norm.
• Plug-in compatibility with LangChain and OpenAI-style APIs.

A canonical usage example in Python: $[P]$0 The model operates fully locally, at scale, and with robust support for 13 languages (Liu et al., 11 Feb 2026).

7. Significance, Limitations, and Outlook

Voxtral Realtime demonstrates that stream-synchronous ASR is achievable at quality levels previously only accessible to offline bidirectional models. Its architecture makes no use of forced alignments, avoids chunk-based approximation, and generalizes natively to multilingual data and low-latency operation. The open-source release under a permissive license further facilitates transparent integration and research use.

Limitations include a modest trailing in WER relative to Scribe v2 on some multilingual datasets (e.g., MCV) and the requirement for GPU-class hardware to match reported latencies.

A plausible implication is that DSM-based stream-synchronous architectures, combined with flexible normalization for delay conditioning, may become the primary foundation for future real-time ASR deployments—especially in multilingual and open-source contexts (Liu et al., 11 Feb 2026).