FireRedLID: Hierarchical LID for 100+ Languages

• FireRedLID is an encoder–decoder spoken language identification module that uses a hierarchical two-token formulation to classify over 100 languages and 20 Chinese dialects.
• The architecture leverages a Conformer-based encoder and a lightweight Transformer decoder, enabling extremely fast decoding with a total of approximately 183 M parameters.
• Benchmark results show outstanding performance with 97.18% on FLEURS, 92.07% on CommonVoice, and 88.47% on Chinese dialects, surpassing competing systems.

FireRedLID is an Encoder–Decoder spoken language identification (LID) module within the FireRedASR2S pipeline, designed for utterance-level classification across 100+ languages and 20+ Chinese dialects and accents. It uses a hierarchical two-token formulation in which the model first predicts a language token and, only when the language is Chinese, optionally predicts an additional dialect token. In the reported evaluation, it achieves 97.18% utterance-level accuracy on FLEURS (82 languages), 92.07% on CommonVoice (74), and 88.47% on Chinese dialects, outperforming Whisper, SpeechBrain, and Dolphin on the listed benchmarks (Xu et al., 11 Mar 2026).

1. System role and representational design

FireRedLID is positioned as the spoken language and Chinese dialect identification component of FireRedASR2S, an all-in-one automatic speech recognition system that integrates ASR, Voice Activity Detection, Spoken Language Identification, and Punctuation Prediction. Within that broader pipeline, FireRedLID functions as a routing-oriented classifier for multilingual and dialectal speech, rather than as a transcription model (Xu et al., 11 Mar 2026).

A defining design choice is its hierarchical two-token formulation. Instead of a flat 120-way softmax, the model first emits a single-token language code such as “zh”, “en”, or “ja”, and then—only if the token is “zh”—emits one additional dialect code such as “mandarin”, “yue”, or “wu”. The stated rationale is that this simplifies learning and reduces confusion between languages and dialects. A plausible implication is that the architecture separates coarse-grained language discrimination from finer-grained intra-Chinese dialect discrimination, while also constraining the effective output space.

The output structure is deliberately short. The longest output is only two tokens, namely 〈Language〉 and, optionally, 〈Dialect〉. This short target length informs multiple downstream properties reported for the module, including the shallow decoder design and the claim that decoding is extremely fast.

2. Architecture

The acoustic front end uses 80-dim log-Mel filterbanks with CMVN. The core encoder is a Conformer-based acoustic encoder initialized from FireRedASR2-AED. In practice, the encoder is a stack of $N_e = 12$ Conformer blocks, each consisting of multi-head self-attention with $H = 8$ heads and model dimension $d = 512$, a convolution module with depthwise convolution and kernel size 31, position-wise feed-forward networks with hidden dimension $2 \times d = 1024$, and layer normalization with residual connections (Xu et al., 11 Mar 2026).

The decoder is a lightweight Transformer decoder with $N_d = 2$ layers, model dimension $d = 512$, 8 heads, and feed-forward dimension 1024. The decoder token embeddings also have size $d = 512$. The shallow decoder is explicitly motivated by the short output length: because the longest output is only two tokens, the decoder is deliberately kept shallow.

The parameterization is reported as approximately 180 M parameters for the encoder and approximately 3 M parameters for the decoder and output head, for a grand total of approximately 183 M parameters. In the FireRedASR2S pipeline, this model footprint is described as sitting comfortably in the middle between the ultra-lightweight VAD at 0.6 M parameters and the large ASR-LLM at 8 B parameters. This suggests a design point oriented toward strong multilingual discrimination without approaching the scale of the speech recognizer.

3. Learning objective and decoding formulation

The model is trained as a sequence-to-sequence classifier. Let $X = (x_1,\dots,x_T)$ denote acoustic feature frames and $Y = (y_1,y_2)$ denote the one- or two-token target sequence. Training uses the standard sequence-to-sequence cross-entropy loss (Xu et al., 11 Mar 2026):

$$\mathcal{L}_{\rm CE} \;=\; -\sum_{t=1}^{|Y|}\,\sum_{v\in V} \; \mathbf{1}\{y_t=v\}\,\log p_\theta(y_t=v \mid y_{<t}, X)\,.$$

Because $H = 8$0, decoding is reported as extremely fast. The model also computes an utterance-level confidence by averaging the per-token softmax probabilities:

$H = 8$1

This confidence construction follows directly from the hierarchical tokenization scheme. For non-Chinese utterances, the output may consist of only the language token; for Chinese utterances, the confidence aggregates both the language and dialect decisions. A plausible implication is that the confidence score reflects both coarse and fine classification certainty when the dialect token is present.

4. Training data and coverage

FireRedLID is trained on approximately 200 k hours of labeled speech. The reported coverage includes 100+ languages, including zh, en, es, fr, ja, ko, and ru, and 20+ Chinese dialects grouped into 8 clusters: mandarin, yue, wu, min, xiang, north, xinan, and bo (Xu et al., 11 Mar 2026).

The data sources combine proprietary multi-domain corpora and public multilingual resources. The proprietary portion includes broadcast news, conversational speech, and read speech. The public sources include a portion of FLEURS and CommonVoice. For Chinese dialect training, the listed datasets include KeSpeech and MagicData subsets for Yue, Chuan, Wu, and others.

The balancing and augmentation strategy is specified in operational terms. Uniform sampling across language buckets is used to avoid high-resource dominance. Standard acoustic augmentation, specifically SpecAugment and speed-perturbation, is applied equally to all languages. Short-segment oversampling is used for languages and dialects with less than 1 k hours. The strengths section later attributes exceptional accuracy on both high-resource and low-resource languages to this large-scale, balanced training data; this suggests that the sampling policy is not incidental but central to the reported generalization behavior.

5. Benchmarks and reported performance

Evaluation is reported in terms of utterance-level identification accuracy on three benchmarks. The results are summarized below (Xu et al., 11 Mar 2026).

Test set	FireRedLID	Baselines
FLEURS (82 lang)	97.18	Whisper-large-v3: 79.41; SpeechBrain LID: 92.91
CommonVoice (74)	92.07	Whisper-large-v3: 80.81; SpeechBrain LID: 78.75
Chinese dialects	88.47	Dolphin: 69.01

On FLEURS, FireRedLID outperforms Whisper by +17.8 pp and exceeds SpeechBrain by +4.3 pp. On CommonVoice, it outperforms Whisper by +11.3 pp. On Chinese dialects, the hierarchical design is reported to yield +19.5 pp over the next best public system, Dolphin.

These comparisons are central to the module’s positioning. In the abstract, FireRedLID is described as supporting 100+ languages and 20+ Chinese dialects and accents, and its FLEURS result is presented as outperforming Whisper and SpeechBrain. The Chinese dialect result is especially notable because it is explicitly linked to the hierarchical formulation rather than only to scale.

6. Inference behavior, strengths, limitations, and projected extensions

FireRedLID operates in non-streaming mode as an utterance-level classifier and is not designed for frame-wise streaming. For typical 5 s inputs, the reported end-to-end LID latency is less than 20 ms on a single V100 GPU, corresponding to a real-time factor of approximately 0.004. Reported throughput is 3 k utterances/s on A100 and 2 k utt/s on V100 at batch size 32. The model footprint is approximately 183 M parameters, or approximately 700 MB in float32 (Xu et al., 11 Mar 2026).

The reported strengths are threefold. First, the module shows exceptional accuracy on both high-resource and low-resource languages due to large-scale, balanced training data. Second, hierarchical two-token decoding reduces confusion between close dialects and keeps vocabulary small. Third, the lightweight decoder and short output sequences yield negligible decoding overhead.

The stated limitations are also specific. Very short clips, defined as less than 0.5 s, or heavily accented speech occasionally mis-classify, especially for under-sampled dialects such as Xiang. The model provides no streaming or continuous-LID support, because the entire utterance must be buffered. Performance degrades gracefully but noticeably when signal-to-noise ratio drops below 10 dB.

Several improvements are suggested. These include integrating a contrastive pre-training stage, for example pretrained speech representations, to boost low-resource dialect accuracy; adding streaming LID support via segmental convolution or sliding-window decoding; incorporating adversarial noise augmentation to improve robustness in real-world noisy environments; and exploring a joint ASR+LID multi-task model to share more representations and reduce overall pipeline latency. These proposals indicate that the current system is optimized for high-accuracy, utterance-level routing in a unified ASR stack, while leaving streaming and noise robustness as explicit future extensions.