FireRedASR2S: Unified Speech Pipeline

• FireRedASR2S is a modular, all-in-one speech system that integrates voice activity detection, language identification, automatic speech recognition, and punctuation prediction.
• It builds on the FireRedASR lineage with dual variants (AED and LLM) enhanced by data scaling and architectural refinements, achieving state-of-the-art CER benchmarks.
• The system robustly handles both non-streaming and streaming modes, offering independent, deployable components for practical multilingual, dialect-rich, and singing transcription scenarios.

FireRedASR2S most specifically denotes an industrial-grade, all-in-one speech processing system that unifies Voice Activity Detection (VAD), Spoken Language Identification (LID), Automatic Speech Recognition (ASR), and Punctuation Prediction (Punc) in a single pipeline, with open-source code and model weights released by the FireRedTeam (Xu et al., 11 Mar 2026). The system is organized around FireRedASR2, which offers an Encoder–Adapter–LLM variant and an Attention-based Encoder–Decoder variant, and is designed for Mandarin, Chinese dialects and accents, English, code-switching, and singing transcription. The nomenclature is not entirely stable across the literature: the earlier FireRedASR release explicitly stated that “FireRedASR2S” was not an official model designation in that work (Xu et al., 24 Jan 2025), while a separate later study reused the same string for a fairness-prompted second-language English ASR finetuning pipeline (Swain et al., 21 Oct 2025). In current FireRedTeam usage, however, the term refers to the four-module unified system introduced in 2026 (Xu et al., 11 Mar 2026).

1. Nomenclature, lineage, and scope

The immediate predecessor of FireRedASR2S is the FireRedASR family, which comprised two named variants—FireRedASR-LLM and FireRedASR-AED—each offered in XS, S, M, and L configurations (Xu et al., 24 Jan 2025). In that earlier release, the authors stated that “FireRedASR2S” was not presented as an official model variant, alias, or repository naming. The later FireRedASR2S report formalizes a different object: not a single recognizer checkpoint, but a modular speech stack that integrates VAD, LID, ASR, and punctuation restoration in one production-oriented pipeline (Xu et al., 11 Mar 2026).

This lineage matters technically because FireRedASR2 reuses and extends the architectural logic of FireRedASR. The 2025 FireRedASR release centered on Mandarin-specialized large-scale ASR with two principal designs: an Encoder–Adapter–LLM model for maximal accuracy and an AED model for efficiency and reuse as a speech representation module (Xu et al., 24 Jan 2025). FireRedASR2 preserves this bifurcation, but the later report attributes its gains primarily to scaling supervised training from approximately $70\text{k}$ to approximately $200\text{k}$ hours, broadening dialect and accent coverage, and adding timestamp support to the AED branch through a post-hoc CTC head (Xu et al., 11 Mar 2026).

A recurrent source of confusion is the reuse of the same string in a distinct fairness study on second-language English ASR. That work defines “FireRedASR2S” as a pipeline combining ERM, Spectral Decoupling, Group-DRO, and IRM under lightweight adapter finetuning for Whisper and Seamless-M4T backbones (Swain et al., 21 Oct 2025). This is a separate methodological usage rather than the FireRedTeam’s all-in-one multilingual speech system. A plausible implication is that the term should be disambiguated by citation context rather than assumed to denote a unique model family.

2. Unified pipeline and system organization

FireRedASR2S processes raw waveform through four sequential modules: FireRedVAD, FireRedLID, FireRedASR2, and FireRedPunc (Xu et al., 11 Mar 2026). FireRedVAD first segments the audio timeline into voice regions and filters non-voice content; FireRedLID then assigns a language label and, for Chinese, optionally a dialect label; FireRedASR2 transcribes each segment; and FireRedPunc restores punctuation. The final structured output includes punctuated text, segment-level start and end timestamps, ASR confidence, optional language and dialect labels with confidences, and token or word timestamps when the AED ASR variant is used.

The architecture is explicitly modular. The report states that the components are independently deployable and composable, which is central to its “industrial-grade” characterization. Robustness is pursued through human-annotated acoustic event labels for VAD, hierarchical LID, large-scale supervised ASR training spanning multilingual, dialectal, and singing data, and confidence scoring. Scalability is addressed by keeping VAD ultra-lightweight, making LID decoding very short, and using post-hoc CTC for AED timestamps so that timestamping does not require retraining the base recognizer (Xu et al., 11 Mar 2026).

The system supports both non-streaming and streaming modes. In non-streaming mode, segmentation can exploit look-ahead; in streaming mode, VAD operates online with a causal DFSMN and zero look-ahead, and downstream modules decode segment by segment as voice regions arrive. The report is careful, however, to localize most streaming behavior to the segmentation layer: default evaluations use non-streaming, and streaming ASR decoding details are not presented as a fully developed subsystem. This corrects a common misconception that FireRedASR2S is a fully streaming end-to-end ASR model in the strict sense.

3. FireRedASR2 as the core transcription engine

FireRedASR2 is provided in two variants (Xu et al., 11 Mar 2026). FireRedASR2-AED is a $1\text{B}+$-parameter model built from a Conformer encoder with initial convolutional subsampling and stacked Conformer blocks, followed by a Transformer decoder with cross-attention over encoder states. It uses a mixed tokenizer—Chinese characters for Chinese and BPE for English—with an updated vocabulary size of 8,667. FireRedASR2-LLM is an $8\text{B}+$-parameter Encoder–Adapter–LLM model in which a Conformer encoder, initialized from the AED encoder, produces acoustic representations that are mapped by a lightweight adapter into the embedding space of a pretrained text LLM, which then autoregressively emits transcript tokens.

The training corpus is approximately $200\text{k}$ hours of supervised data spanning Mandarin, English, $20+$ Chinese dialects and accents, code-switching, speech, and singing (Xu et al., 11 Mar 2026). The report states that this expanded diversity is a major contributor to accent robustness and dialect coverage. Singing is not treated as a peripheral task: the training set includes singing speech, and the system reports dedicated results on opencpop.

The AED branch uses standard attention-based sequence prediction:

$$c_t = \sum_i \alpha_{t,i} h_i, \qquad \alpha_{t,i} = \mathrm{softmax}(e(s_{t-1}, h_i)),$$

with training loss

$$L_{\mathrm{att}} = -\sum_t \log p(y_t \mid y_{<t}, x).$$

Timestamping is added after base AED training by attaching a linear CTC projection to frozen encoder outputs and optimizing

$$L_{\mathrm{CTC}} = -\log \sum_{\pi \in B^{-1}(y)} \prod_t p(\pi_t \mid x).$$

The report explicitly states that this is not joint CTC–AED training; the base model is trained only with $L_{\mathrm{att}}$, and CTC is optimized post hoc for alignment (Xu et al., 11 Mar 2026).

Decoding differs across the two branches. The AED model uses beam search for token generation and derives token timestamps by CTC forced alignment to the $200\text{k}$0-best AED hypothesis; word timestamps are then obtained by merging BPE subwords in English or treating each Chinese character as a word unit. The LLM branch follows the FireRedASR-LLM pattern of prompt-conditioned autoregressive generation and does not report any external LM rescoring beyond the LLM itself. Utterance-level confidence for AED is derived from decoder posteriors as a geometric mean over hypothesis tokens (Xu et al., 11 Mar 2026).

4. Auxiliary modules: VAD, LID, and punctuation restoration

The non-ASR modules are not ancillary utilities; they are treated as first-class components with independent benchmarks and architectural designs (Xu et al., 11 Mar 2026).

FireRedVAD is based on the Deep Feedforward Sequential Memory Network. The report specifies 8 DFSMN blocks plus 1 feed-forward layer, hidden size 256, projection 128, dropout 0.05, look-back order 20, and stride 1. Non-streaming VAD uses look-ahead order 20, while streaming VAD sets look-ahead to 0. The memory mechanism is written as

$200\text{k}$1

The module supports both binary VAD and multi-label VAD, where voice is defined as speech $200\text{k}$2 singing and non-voice includes music, silence, and noise. For multi-label training over $200\text{k}$3, the loss is

$200\text{k}$4

Post-processing applies moving-average smoothing, fixed thresholding, finite-state hangover logic, gap merging, boundary extension, and splitting of overly long segments.

FireRedLID uses an Encoder–Decoder architecture with a Conformer encoder initialized from FireRedASR2-AED and a Transformer decoder trained from scratch. It covers $200\text{k}$5 languages, and for Chinese it performs hierarchical two-token decoding: first a language token, then a dialect token from $200\text{k}$6 dialects grouped into 8 clusters. Decoding length is at most 2 tokens, beam search is used, and confidence is the mean posterior over emitted label tokens excluding $200\text{k}$7 and $200\text{k}$8. The training loss is token-level cross-entropy:

$200\text{k}$9

FireRedPunc is a BERT-style punctuation prediction model initialized from LERT. It uses a 5-class tag set $1\text{B}+$0, with full-width punctuation for Chinese output. Its training corpus contains 18.57B Chinese characters and 2.20B English words, and its token-classification objective is

$1\text{B}+$1

At inference, punctuation is inserted after each token in the ASR hypothesis (Xu et al., 11 Mar 2026).

5. Benchmarks and reported performance

The report presents the system as achieving state-of-the-art results on the evaluated benchmarks (Xu et al., 11 Mar 2026). The headline results are summarized below.

Component	Benchmark or aggregate	Reported result
FireRedASR2-LLM	Avg-All-24 ASR	9.67 CER
FireRedASR2-LLM	Avg-Mandarin-4 ASR	2.89 CER
FireRedASR2-LLM	Avg-Dialect-19 ASR	11.55 CER
FireRedASR2-LLM	opencpop singing	1.12 CER
FireRedVAD	FLEURS-VAD-102	97.57 F1, 99.60 AUC-ROC
FireRedLID	FLEURS (82 languages)	97.18% accuracy
FireRedPunc	Chinese/English average	78.90 F1

For ASR, FireRedASR2-LLM reports 9.67 CER on Avg-All-24, 2.89 CER on Avg-Mandarin-4, 11.55 CER on Avg-Dialect-19, and 1.12 CER on opencpop. The AED variant reports 9.80, 3.05, 11.67, and 1.17 respectively. On the 4 Mandarin public benchmarks, the LLM variant gives 0.64 CER on AISHELL-1, 2.15 on AISHELL-2 (iOS), 4.44 on WenetSpeech Internet, and 4.32 on WenetSpeech Meeting. The same report states that these numbers outperform competitive baselines including Doubao-ASR, Qwen3-ASR, and Fun-ASR on the evaluated aggregates (Xu et al., 11 Mar 2026).

Relative to the earlier FireRedASR family, the ASR core shows incremental but concrete improvements on the corresponding 4-benchmark Mandarin aggregate. FireRedASR-LLM-L reported 3.05% Average-4 CER, while FireRedASR2-LLM reports 2.89% on Avg-Mandarin-4; FireRedASR-AED-L reported 3.18%, while FireRedASR2-AED reports 3.05% (Xu et al., 24 Jan 2025). The later report attributes such gains qualitatively to data scaling, broader dialect coverage, and related design refinements (Xu et al., 11 Mar 2026).

For VAD, frame-level evaluation on FLEURS-VAD-102 yields 99.60% AUC-ROC, 97.57% F1, 2.69% FAR, and 3.62% MR. These values exceed the reported F1 or AUC figures of Silero-VAD, TEN-VAD, FunASR-VAD, and WebRTC-VAD on that benchmark. For LID, FireRedLID reaches 97.18% utterance-level accuracy on FLEURS with 82 languages, 92.07% on CommonVoice with 74 languages, and 88.47% on a merged Chinese dialect benchmark built from KeSpeech and MagicData. For punctuation, FireRedPunc reports 82.96 F1 on multi-domain Chinese, 74.83 on multi-domain English, and 78.90 average F1 across languages, compared with 62.77 for FunASR-Punc (Xu et al., 11 Mar 2026).

6. Limitations, ambiguities, and research significance

The report does not include formal ablation experiments (Xu et al., 11 Mar 2026). Instead, it identifies several observed drivers of performance: data scaling from approximately $1\text{B}+$2 to approximately $1\text{B}+$3 hours for ASR, hierarchical two-token LID, human-annotated event labels for VAD, and post-hoc CTC for AED timestamps. This suggests that the system’s contribution is as much infrastructural and data-centric as it is architectural.

Several constraints remain explicit. The $1\text{B}+$4 LLM-based ASR variant is computationally heavy and may require GPU clusters for deployment, while even the $1\text{B}+$5 AED model remains substantial relative to small on-device recognizers. Streaming behavior is dominated by the VAD front end; strict-latency streaming ASR is not fully characterized. Quantization and pruning are not reported. Acoustic edge cases continue to matter: overlapping speech, extreme noise or music, rapid code-switching, and dialectal confusions—particularly in difficult subsets such as Shanghai conversational data—can still degrade performance (Xu et al., 11 Mar 2026).

A separate limitation is terminological rather than algorithmic. Because “FireRedASR2S” was not an official designation in the earlier FireRedASR paper and was later reused for a fairness-oriented L2 English finetuning method, references to the term can be ambiguous unless anchored to the specific arXiv paper (Xu et al., 24 Jan 2025). In the fairness work, the name denotes a multi-objective adapter finetuning regime for Whisper and Seamless-M4T on 26 accent groups from EdAcc, reporting macro-averaged WER reductions of 58.7% and 58.5% relative to pretrained Whisper-large and Seamless-large, and 9.7% and 7.8% relative to ERM finetuning (Swain et al., 21 Oct 2025). That usage is methodologically interesting, but it is not the same system as the 2026 all-in-one multilingual pipeline.

Within the FireRedTeam line, FireRedASR2S is best understood as a modular, reproducible speech stack that elevates auxiliary components—segmentation, language routing, and punctuation restoration—to the same level of engineering and benchmarking emphasis as the recognizer itself. Its research significance lies not only in reported CER, F1, and accuracy numbers, but also in presenting a unified open-source configuration in which VAD, LID, ASR, and Punc are co-designed for long-form, multilingual, dialect-rich, and music-contaminated speech scenarios (Xu et al., 11 Mar 2026).