A Comparative Study of Pre-trained Speech Encoders and Training Objectives for Large-Scale Indic Spoken Language Identification

Abstract: Spoken language identification (LID) for Indian languages is a challenging problem due to the large number of languages, significant phonetic overlap among related varieties, and the scarcity of labeled data for many low-resource languages. In this work, we present a systematic comparative study of two pre-trained speech encoders -- Whisper and FastConformer -- combined with a linear classifier for large-scale Indic LID spanning 42 languages across four linguistic families. We evaluate both encoders in frozen (linear probing) and fine-tuned settings, and compare three training objectives: cross-entropy (CE), supervised contrastive loss with cross entropy (CE + supCon), and hierarchical softmax (HSM). Models are trained on the Vaani dataset and evaluated in a cross-corpus setting on Vaani-Test (held-out), FLEURS, and Kathbath, providing insights into domain generalization. The frozen FastConformer encoder achieves over 90\% macro accuracy on FLEURS and Kathbath without any task-specific adaptation, substantially outperforming Whisper on out-of-domain benchmarks, while fine-tuned Whisper yields stronger in-domain performance. HSM consistently outperforms CE and CE+SupCon for both encoders across all benchmarks, with the largest gains on out-of-domain test sets. CE+SupCon degrades FastConformer's cross-corpus generalization, suggesting that the contrastive objective over-specializes representations to in-domain conditions. Per-family analysis shows that Central Indo-Aryan varieties are the hardest to discriminate, with Hindi--Urdu and the Sadri--Chhattisgarhi--Surgujia cluster being the dominant confusion pairs.

• The paper systematically compares pre-trained encoders and training objectives, revealing that FastConformer's frozen variant offers superior cross-lingual generalization compared to fine-tuned Whisper.
• It employs a unified encoder-classifier architecture evaluated on 42 Indic languages using in-domain and cross-corpus benchmarks, highlighting the effectiveness of Hierarchical Softmax.
• The findings underline the importance of objective design for out-of-domain performance, particularly addressing challenges in discriminating similar Central Indo-Aryan languages.

Comparative Evaluation of Pre-trained Speech Encoders and Training Objectives for Large-Scale Indic Spoken Language Identification

Introduction

Spoken Language Identification (LID) in the Indic context presents acute challenges due to the sheer number of target languages, substantial phonological overlap, and the dearth of labeled data for most low-resource speech communities. This study systematically evaluates two state-of-the-art pre-trained speech encoders---Whisper and FastConformer---and three principal training objectives (Cross-Entropy (CE), CE + Supervised Contrastive (SupCon), and Hierarchical Softmax (HSM)), utilizing a unified encoder--classifier architecture. The evaluation spans 42 Indic languages drawn from four linguistic families, relying on rigorous cross-corpus protocols to meaningfully probe generalization beyond the training domain.

Model Architectures and Training Objectives

The study adopts a modular encoder-classifier design: utterance-level embeddings, generated by either the Whisper or FastConformer pre-trained encoder via self-attention pooling, feed into a linear classification head. Both encoders are compared in frozen (linear probing) and end-to-end fine-tuned settings. The three main loss functions are:

• Cross-Entropy (CE): Standard multiclass objective, offering direct supervision only at the classification head.
• CE + SupCon: Encourages embedding space clustering via supervised contrastive loss, weighted along with CE to jointly shape the representation and classification layers.
• Hierarchical Softmax (HSM): Exploits the structured taxonomy of languages, decomposing the classification over the linguistic hierarchy; this approach is geared toward improved separation of closely related language varieties.

Experimental Protocol

Training exclusively utilizes a balanced subset (10 hours/language) from Vaani, one of the largest naturalistic corpora representative of Indian linguistic diversity. Evaluation benchmarks are:

• Vaani-Test: In-domain held-out set.
• FLEURS: Standard cross-lingual benchmark (13 Indic languages).
• Kathbath: Out-of-domain test set (11 languages).

All results are macro-averaged for accuracy, treating every language equally irrespective of its corpus frequency.

Quantitative Results and Analysis

Encoder and Fine-tuning Effects

A marked divergence is observed between the two encoder architectures. Fine-tuning significantly enhances Whisper's performance, whereas FastConformer’s best generalization is realized when it remains frozen. Fine-tuned FastConformer exhibits notable overfitting to the in-domain Vaani dataset—with cross-corpus accuracy dropping substantially post-fine-tuning. Importantly, the frozen FastConformer surpasses fine-tuned Whisper on FLEURS and Kathbath, underscoring its superior cross-lingual robustness:

• Frozen FastConformer: FLEURS 94.2%, Kathbath 90.9%
• Fine-tuned Whisper: FLEURS 72.7%, Kathbath 68.3%

This denotes a strong cross-domain transferability of FastConformer's pre-trained representations for Indic speech, despite a relatively modest in-domain showing.

Impact of Training Objectives

Across both encoder families and all evaluation protocols, HSM provides the highest macro-accuracy, with especially prominent gains on out-of-domain test sets (e.g., Whisper-HSM on Kathbath: 75.8%, +7.5% over CE).

• CE + SupCon consistently underperforms HSM and, for FastConformer, even lags behind standard CE on cross-corpus tasks. This suggests that while supervised contrastive learning can yield a more clustered embedding space, it can inadvertently cause over-specialization to the training domain, harming generalization.

Comparison against external LID baselines (Facebook MMS and SpeechBrain ECAPA-TDNN) reveals that FastConformer-HSM matches or outperforms these systems on FLEURS and Kathbath, despite the latter covering a far smaller set of languages.

Cross-corpus Generalization

FastConformer provides an overwhelming advantage in generalization. Its frozen variant attains >90% macro-accuracy on both FLEURS and Kathbath, compared to Whisper's ~60%. This substantial gap corroborates the hypothesis that the architectural and pre-training recipe of FastConformer yields representations less sensitive to domain shift, while Whisper's representations, perhaps due to the nature of its weakly supervised pre-training, have a higher domain-dependence.

Per-family and Error Analysis

A granular breakdown by linguistic family reveals stark accuracy variation:

• Sino-Tibetan: 97.1%
• Dravidian: 85.9%
• Indo-Aryan overall: 67.6%
  • Central Indo-Aryan: 58.7% (lowest-performing subgroup)

This pattern reflects the relative distinctiveness of Sino-Tibetan and Dravidian phonologies versus the dense, overlapping feature space of Central Indo-Aryan languages.

Examination of confusion matrices highlights two dominant error modes:

1. Hindi–Urdu Confusion: Extremely high mutual confusion, especially in FastConformer, mirroring their near-identical spoken forms.
2. Sadri–Chhattisgarhi–Surgujia Cluster: Substantial overlap leads to diffuse predictions across this subset, again most pronounced in FastConformer.

Figure 1: Confusion matrices for Central Indo-Aryan languages: Whisper + HSM vs. FastConformer + HSM. Darker diagonals indicate superior discrimination.

Bajjika and Halbi, in contrast, are well-identified due to greater distinctiveness and/or more robust training support. These findings underscore the limits of encoder architectures in disentangling true phonological similarity under current pre-training regimes.

Theoretical and Practical Implications

The study refines several foundational LID claims. Primarily, pre-trained feature extractors are not uniformly effective across tasks requiring fine-grained discrimination, especially when faced with closely-related language varieties. The decisive superiority of HSM for cross-domain robustness indicates that architectural and objective alignment with linguistic hierarchies is beneficial, a result congruent with theory in hierarchical classification.

Furthermore, success in generalization is dictated more by pre-training data coverage and domain diversity than by mere parameter count, with FastConformer's pre-training on diverse Indic data enabling superior transfer. However, the persistent confusion among Central Indo-Aryan languages marks a bottleneck for practical deployment, particularly in applications serving dialectally-contiguous regions.

Future Directions

Critical future research directions include: (i) data augmentation and adversarial training focusing on low-resource Central Indo-Aryan varieties, (ii) integration of phonotactic or sub-word linguistic priors into encoder objectives, and (iii) explicit domain adaptation and multi-corpus fine-tuning strategies. The role of hierarchical objectives in minimizing intra-family confusion warrants deeper investigation, as does the optimization of cross-corpus robustness for encoder architectures underrepresented in current pre-training datasets.

Conclusion

This comparative analysis demonstrates that model and objective selection are pivotal for robust large-scale Indic LID. FastConformer, particularly in its frozen regime, yields markedly better cross-domain generalization than Whisper. Hierarchical softmax outperforms both CE and contrastive methods, especially on out-of-domain speech. The central challenge for future LID systems remains the fine-grained discrimination among highly similar Central Indo-Aryan varieties, indicating a pressing need for enhanced data and objective-driven learning tailored to the typology of Indo-Aryan speech.