FireRedVAD: Robust Voice Activity Detection

• FireRedVAD is a robust voice activity detection module within FireRedASR2S that segments audio to identify speech, singing, and music.
• It employs a DFSMN-based architecture with about 0.6 million parameters, achieving efficient processing and high accuracy in challenging acoustic environments.
• It supports both binary and multi-label classification modes with tailored post-processing for offline and streaming applications.

FireRedVAD is the voice activity detection component of the industrial-grade, all-in-one speech system FireRedASR2S, which also includes ASR, spoken language identification, and punctuation prediction. In that system, FireRedVAD operates on the raw waveform before the downstream modules, detecting where voice is present, segmenting audio into voice and non-voice regions, and, in its multi-label configuration, distinguishing speech, singing, and music at the frame level. The design emphasizes industrial robustness for user-generated content, including long recordings, silence, noise, background music, and singing, while maintaining an ultra-lightweight DFSMN-based footprint of approximately $0.6$ million parameters per model (Xu et al., 11 Mar 2026).

1. Position within FireRedASR2S

Within FireRedASR2S, FireRedVAD is the front-end segmentation module. It runs first on the raw waveform, identifies where “voice” is present, and outputs segments that are then passed to FireRedLID for language or dialect prediction, to FireRedASR2 for transcription of only the voice segments, and to FireRedPunc for punctuation restoration (Xu et al., 11 Mar 2026). This arrangement reflects the system-level assumption that robust segmentation is critical in real-world speech processing pipelines, particularly for long-form and acoustically heterogeneous user-generated content.

The paper explicitly places FireRedVAD alongside three other modules: FireRedASR2, FireRedLID, and FireRedPunc. In this architecture, segmentation is not merely a preprocessing convenience; it is a control point for compute allocation and error propagation. The text states that long recordings, silence, noise, background music, and singing make segmentation especially important, and that many industrial VAD systems rely on weak supervision from ASR forced alignment, whereas FireRedVAD is trained on thousands of hours of human-annotated acoustic events. This suggests that the authors view front-end segmentation quality as a principal determinant of downstream ASR stability under complex acoustic conditions.

A common misconception is to treat FireRedVAD as a single binary classifier. In fact, the term denotes a family of three related DFSMN-based models: non-streaming VAD, streaming VAD, and non-streaming multi-label VAD (mVAD) (Xu et al., 11 Mar 2026).

2. Task definitions and label ontology

FireRedVAD supports three operational modes: offline binary segmentation, online low-latency segmentation, and frame-level acoustic event recognition. The two standard VAD variants perform binary frame-level classification, while mVAD performs frame-level multi-label classification.

For standard VAD, the positive class is defined as the union of speech and singing, and the negative class as music, silence, and noise (Xu et al., 11 Mar 2026). This is an application-specific ontology tuned for ASR usage in user-generated content: singing is treated as “voice” because it should be passed to the recognizer, whereas pure music and non-speech acoustic content should be filtered out. Both the streaming and non-streaming VAD models use this binary labeling; they differ only in the amount of future context available at inference time.

The mVAD variant retains the original three-event ontology and is formulated as a frame-level multi-label classification problem over speech, singing, and music. Its outputs are independent posterior probabilities,

$$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$

where $t$ indexes time frames (Xu et al., 11 Mar 2026). These posteriors are not constrained to sum to $1$, so overlapping activity patterns can in principle be represented, such as simultaneous singing and music. Event segments are then obtained via event-wise post-processing.

Although the binary VAD and mVAD tasks are derived from the same human annotation ontology, they are trained as separate models with distinct objectives and post-processing procedures. The paper states that mVAD uses the original three-class human labels directly, while binary VAD collapses speech and singing into voice and maps music, silence, and noise to non-voice (Xu et al., 11 Mar 2026). A plausible implication is that FireRedVAD is designed not only as a speech gatekeeper for ASR, but also as a lightweight acoustic event segmentation subsystem for routing or filtering decisions.

3. Supervision and training data

All FireRedVAD models are trained on “thousands of hours of human-annotated acoustic event data,” with each utterance annotated with time boundaries for speech, singing, and music (Xu et al., 11 Mar 2026). The paper contrasts this supervision strategy with the weak supervision used by many industrial VAD systems, where labels are approximated via ASR forced alignment. The stated motivation is greater reliability under complex acoustic conditions and better discrimination among speech, singing, music, and non-speech background content.

The supervision protocol differs by task. For mVAD, the three-event labels are used directly as frame-level multi-label targets. For binary VAD, the same annotation is remapped so that speech and singing correspond to voice and music, silence, and noise correspond to non-voice (Xu et al., 11 Mar 2026). The paper does not explicitly specify the loss functions in equation form, but it notes the output structure and label semantics in a way consistent with independent sigmoid-based frame-level posterior estimation.

This reliance on direct human annotation is central to the system’s claimed robustness. The authors explicitly associate it with improved accuracy on non-speech, improved distinction between speech, singing, and music, and greater robustness in varied acoustics such as background music and user-generated noise (Xu et al., 11 Mar 2026). Because the paper does not enumerate ablation results isolating annotation quality, these causal claims should be read as design rationale rather than as independently quantified attribution.

4. DFSMN architecture and efficiency profile

All three FireRedVAD variants use the same Deep Feedforward Sequential Memory Network backbone. The model consumes the same acoustic features as FireRedASR2: 80-dimensional log Mel filterbanks, a 25 ms window, a 10 ms frame shift, and global CMVN (Xu et al., 11 Mar 2026). This shared front-end standardizes feature extraction across VAD, LID, and ASR within FireRedASR2S.

The paper states that FSMN memory blocks are implemented using depthwise 1-D convolutions with dilation to model temporal context, together with residual connections for stable optimization (Xu et al., 11 Mar 2026). Conceptually, the DFSMN layer augments a projected hidden state with a learnable aggregation of past and, in non-streaming variants, future frames: $\tilde{\mathbf{h}}_t = \mathbf{V}\mathbf{h}_t,$

$$\mathbf{m}_t = \sum_{i=0}^{N_1}\mathbf{a}_i \odot \tilde{\mathbf{h}}_{t-i} + \sum_{j=1}^{N_2}\mathbf{c}_j \odot \tilde{\mathbf{h}}_{t+j},$$

$\mathbf{h}'_t = \mathbf{h}_t + \mathbf{m}_t.$

In FireRedVAD, this memory mechanism is realized efficiently through depthwise temporal convolutions rather than recurrent state transitions.

The configuration is fixed across the family: 8 DFSMN blocks, hidden size 256, projection size 128, and dropout 0.05 (Xu et al., 11 Mar 2026). All variants use a look-back order of 20 with stride 1. The look-ahead order is 20 for non-streaming VAD and mVAD, and 0 for streaming VAD. With a 10 ms frame shift, this corresponds to approximately 200 ms of past context for all variants and an additional 200 ms of future context for the offline models. The paper interprets this as yielding smoother and more accurate offline decisions, while the causal streaming variant remains suitable for real-time use.

The output layer is a linear projection to logits followed by sigmoid activation: $p_k(t)=\sigma(z_k(t))=\frac{1}{1+e^{-z_k(t)}},$ with one output dimension for binary VAD and three output dimensions for mVAD (Xu et al., 11 Mar 2026).

A defining property of FireRedVAD is its compactness. Each model has approximately $0.6$ million parameters and occupies approximately $2.2$ MB in float32 (Xu et al., 11 Mar 2026). The paper explicitly links this footprint to minimal memory and compute overhead, suitability for high-QPS cloud deployment, and applicability to edge or low-resource devices.

Variant	Context regime	Output space
Non-streaming VAD	Look-ahead $$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$0, look-back $$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$1	Binary voice / non-voice
Streaming VAD	Look-ahead $$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$2, look-back $$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$3	Binary voice / non-voice
Non-streaming mVAD	Look-ahead $$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$4, look-back $$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$5	Speech, singing, music

5. Inference mechanics and post-processing

The streaming and non-streaming variants differ primarily in how temporal context is used during inference. Streaming VAD is strictly causal because its look-ahead is $$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$6. Each DFSMN layer maintains a per-layer cache of preceding frames up to the look-back order, and inference appends new frames to the cache while computing outputs only for new frames (Xu et al., 11 Mar 2026). This keeps latency low and memory bounded. The paper identifies live captioning and low-latency ASR services as representative uses.

The non-streaming VAD and mVAD variants use 20 frames of future context and therefore can exploit both past and future information during segmentation. The paper states that this generally yields better accuracy and smoother decisions than the streaming model, making these variants appropriate for batch processing, offline transcription of long recordings, or offline acoustic event analysis (Xu et al., 11 Mar 2026).

The raw network outputs are frame-level posterior probabilities, which are converted into time segments through a multi-stage post-processing pipeline. First, a moving-average filter smooths the posterior sequence to reduce local fluctuations. Second, thresholding is applied, with the example threshold given as $$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$7, to obtain frame-level labels (Xu et al., 11 Mar 2026). Third, “a finite-state postprocessor enforces minimum voice and silence duration constraints,” suppressing short voice blips and merging short silences into surrounding voice. This is the system’s hangover-style stabilization mechanism.

The pipeline then performs segment refinement by merging short gaps between voice segments, extending boundaries slightly to include near-edge speech that may be underconfident, and splitting overly long segments (Xu et al., 11 Mar 2026). The paper states that splitting long segments is useful for ASR because long segments can lead to heavy ASR load and timing drift, whereas shorter segments improve latency and stability.

Outputs differ by mode. Non-streaming VAD produces a set of voice segments with start and end timestamps. Streaming VAD produces frame-level decisions and voice start or end events incrementally as audio arrives. mVAD applies the same smoothing, thresholding, and duration rules independently to each event and produces event-wise segments for speech, singing, and music (Xu et al., 11 Mar 2026). This suggests an event-aware front-end capable of dispatching different acoustic content to different downstream paths.

6. Evaluation, benchmark results, and limitations

FireRedVAD is evaluated on binary speech activity detection using the label space voice versus non-voice. The benchmark introduced in the paper is FLEURS-VAD-102, which is based on the FLEURS test set and contains approximately 100 audio files for each of 102 languages, for a total of 9,443 audio files, each manually annotated with binary VAD labels (Xu et al., 11 Mar 2026). Evaluation uses 25 ms analysis windows and a 10 ms frame shift, matching the feature setup used by FireRedVAD itself.

The reported metrics are frame-level AUC-ROC, F1, FAR, and MR. The paper defines F1 as the harmonic mean of precision and recall, FAR as the false alarm rate over total non-voice frames, and MR as the miss rate over total voice frames (Xu et al., 11 Mar 2026). For neural VAD systems, a fixed threshold of $$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$8 is used for all models without per-system tuning. The baselines are Silero-VAD, TEN-VAD, FunASR-VAD, and WebRTC-VAD.

Metric	FireRedVAD	Comparison note
AUC-ROC (%)	99.60	Higher than Silero-VAD 97.99 and TEN-VAD 97.81
F1 (%)	97.57	Higher than Silero-VAD 95.95, TEN-VAD 95.19, FunASR-VAD 90.91, WebRTC-VAD 52.30
FAR (%)	2.69	Lower than Silero-VAD 9.41, TEN-VAD 15.47, FunASR-VAD 44.03; comparable to WebRTC-VAD 2.83
MR (%)	3.62	Higher than TEN-VAD 2.95 and FunASR-VAD 0.42; lower than WebRTC-VAD 64.15

The authors characterize these results as state of the art on FLEURS-VAD-102, emphasizing especially the combination of 97.57% frame-level F1, 99.60% AUC-ROC, and a model size of approximately $$p_{\text{speech}}(t), \quad p_{\text{singing}}(t), \quad p_{\text{music}}(t),$$9 million parameters (Xu et al., 11 Mar 2026). They also interpret the metric trade-offs explicitly: FunASR-VAD attains a very low MR by aggressively labeling speech, but incurs a very high FAR of 44.03%, while WebRTC-VAD achieves low FAR at the cost of an extremely high MR of 64.15%. In contrast, FireRedVAD is presented as operating at a more balanced point.

The broader significance of these results lies in the stated integration benefits for FireRedASR2S. Clean VAD segments are said to benefit FireRedLID by reducing confusion from noise and music, reduce computational cost for FireRedASR2 by avoiding transcription of silence and music, and indirectly help FireRedPunc because segment boundaries often align with sentence or clause boundaries (Xu et al., 11 Mar 2026). The paper also notes that FireRedVAD can be deployed as a standalone segmentation front-end.

The paper does not provide a dedicated empirical failure analysis for FireRedVAD, but it does imply several limitations. Event granularity in mVAD is currently restricted to speech, singing, and music. More fine-grained acoustic events, including overlapping speech and specific noise types, are not modeled. Although multi-label outputs allow overlap in principle, implementation details and performance under heavy overlap are not elaborated. Additional benchmark coverage for extreme noise, far-field microphones, or more unusual acoustic environments is likewise not reported (Xu et al., 11 Mar 2026). These omissions do not contradict the reported multilingual robustness, but they delimit the scope of the current evidence.