Efficiently Adapting Spoken Language Models for the Singaporean Context

Abstract: Spoken LLMs (SLMs) unify speech perception and reasoning, but adapting them to sensitive domains is underexplored, especially when the original training data is inaccessible and the use case demands multilingual, spoken-query interaction. We adapt an open-source SLM to the Singaporean Home Team context across five speech tasks in Singapore's four official languages, combining LoRA fine-tuning, a surrogate text-QA dataset that guards against catastrophic forgetting, and a multi-task objective that adapts the CoBa reweighting scheme to speech. We also build HTD-multilingual-QA, a 504,853 sample multilingual QA dataset in text and spoken form. The resulting HT-Moonstone (5B) matches or outperforms SLMs up to 7x its size on most tasks, attains the best accent and gender recognition among all models evaluated, and loses under 2\% of its original speech QA ability.

• The paper presents a parameter-efficient multi-task adaptation pipeline using LoRA to fine-tune spoken language models for the Singaporean context.
• It introduces the HTD-multilingual-QA dataset, covering four official languages and code-mixed speech to support diverse SLM tasks.
• The approach mitigates catastrophic forgetting and dynamically balances tasks with CoBa, achieving state-of-the-art performance in accent and gender recognition.

Efficient Adaptation of Spoken LLMs for the Singaporean Context

Introduction

The paper "Efficiently Adapting Spoken LLMs for the Singaporean Context" (2607.10092) systematically investigates strategies for localizing open-source Spoken LLMs (SLMs) to the complex, resource-constrained, and multilingual environment of Singapore, where privacy constraints prohibit training from scratch and limit the availability of large-scale native corpus data. The authors address crucial gaps in SLM adaptation for sensitive domains, notably the lack of established methodology for adapting both perception (e.g., ASR, accent/gender recognition) and reasoning (e.g., spoken QA) tasks without incurring catastrophic forgetting in foundational model capabilities.

Central to the study are three key innovations: (1) a parameter-efficient multi-task adaptation pipeline using LoRA, (2) the development of the HTD-multilingual-QA dataset spanning all four official Singaporean languages in both text and speech, and (3) the deployment of CoBa-based reweighting for dynamic task-balancing during fine-tuning. The resulting model, HT-Moonstone, establishes a new state-of-the-art for resource-constrained SLM adaptation in multilingual, code-mixed real-world conditions.

Singaporean Speech Tasks and SLMs: Complexity and Gaps

Singapore presents a particularly challenging environment for SLMs due to its multilingual landscape (English, Mandarin, Bahasa Melayu, Tamil), prevalent code-switching (e.g., Singlish), and operational requirements for spoken-query, text-context paradigms. Existing open-source SLMs such as Kimi-Audio, Audio-Flamingo-Next, MERaLiON-2, and Voxtral exhibit limitations—either optimizing for singular tasks or lacking support for mixed-modality interactions critical for security and conversational AI in the Home Team context. The absence of a native benchmark spanning multi-turn, spoken-query tasks in the full linguistic range further stymies robust adaptation.

Figure 1: Illustration of multimodal, multi-task interaction handled by HT-Moonstone.

Adaptation Methodology and Dataset Construction

The adaptation pipeline is grounded on fine-tuning Voxtral-Mini-3B-2507 with LoRA to maximize parameter-efficiency. Task selection is guided by operational priorities: ASR, accent recognition (AR), gender recognition (GR), spoken QA, and speech QA, with evaluation mainly drawn from MNSC components and newly created spoken QA evaluation sets tailored to the Singaporean context.

A pivotal contribution is the construction of HTD-multilingual-QA. Starting from locally grounded, persona-based Singaporean text QA, the dataset is extended via translation and code-mixing into Mandarin, Bahasa Melayu, and Tamil variants using GPT-4o, then rendered into speech through speaker-matched voice cloning (OmniVoice). This resource, comprising over 500k QA samples in both modalities, directly supports adaptation for authentic Singaporean spoken interaction.

Mitigating Catastrophic Forgetting with Surrogate Text QA

Preserving original SLM reasoning faculties in the absence of the original pretraining data is a critical technical challenge. The study benchmarks multiple surrogate instruction datasets for use in parameter-efficient fine-tuning (PEFT), finding that Nemotron-SFT-Instruction-Following-Chat-v2, a text-only corpus, uniquely supports robust retention of spoken QA and speech QA capabilities during adaptation. Thus, despite downstream tasks emphasizing spoken modality, text-based surrogates can serve as an effective anchor against degradation from multi-task adaptation.

Multi-Task Training with Task-Convergence-Aware Balancing

Adapting SLMs to heterogeneous tasks introduces loss-balancing issues due to disparities in sequence length, convergence dynamics, and downstream value. The authors implement and empirically validate a two-fold approach:

• Token-normalized per-domain loss aggregation: Each task domain's cross-entropy is normalized per answer length before aggregation, preventing longer QA domains from dominating over shallower ones like AR/GR.
• Dynamic, convergence-based weighting (CoBa): Per-domain weights $\omega_i(t)$ are dynamically modulated based on relative convergence status, reducing overfitting and allocating compute to as-yet unconverged tasks. Redistribution of training proportionality further prioritizes late-converging reasoning tasks.

Figure 2: Per-dataset domain evaluation loss curves, analyzing the effectiveness of CoBa and dynamic task reweighting.

Empirical Results and Performance Analysis

After full-scope multi-task fine-tuning (1M samples, ∼3k hours audio, 500M tokens, LoRA, 2 x H100), the HT-Moonstone model demonstrates robust improvements and competitive benchmarks:

• Accent Recognition: AR accuracy increased from 24.7 to 72.6, significantly outperforming all baseline SLMs regardless of parameter count.
• Gender Recognition: GR accuracy rose sharply from 15.7 to 93.9, surpassing all other models, including those with over 7$\times$ parameters.
• Spoken QA: Accuracy improved from 42.2 to 52.1—second only to Qwen3-Omni (7$\times$ larger).
• Automatic Speech Recognition (ASR): HT-Moonstone delivers a WER of 6.59, improved over its base and competitive with models up to double its size.
• Speech QA: Retained within 1 point of base accuracy despite extensive adaptation, indicating minimal catastrophic forgetting.
• Multilingual ASR: Average error rate reduced from 52.4 (base) to 14.6 on all four local languages—trailing only specialized models trained from scratch on the same benchmarks.

Highlight: The model’s performance on accent and gender recognition constitutes the strongest results, particularly remarkable given the focus on transfer and adaptation rather than training from scratch.

Limitations and Future Directions

Outstanding limitations include the single-scale evaluation (5B parameters), incomplete coverage of tasks (e.g., emotion recognition, audio-scene QA not explored), absence of architectural ablations, and non-exhaustive search over training mixture hyperparameters. There is also an intrinsic data imbalance: Malay and Tamil ASR data scarcity precludes deeper scaling and potentially biases generalization across languages.

The authors propose extending the work to larger-scale models, a broader suite of spoken and text tasks, more optimal domain-proportion search strategies, and exploration of alternative encoder/decoder architectures for further gains.

Implications and Prospective Impact

Practically, the work demonstrates that privacy- and resource-sensitive organizations can feasibly adapt publicly available SLMs for complex, highly multilingual domains using modest compute. Theoretically, the successful application of text-only surrogate data and CoBa-based loss scheduling in spoken settings establishes new best practices for low-resource SLM adaptation and multi-task learning. This may shape future development of regionally customized, privacy-compliant conversational agents and multimodal SLMs, with implications for governmental, clinical, and other sensitive-use scenarios internationally.

Conclusion

This study establishes a rigorous, efficient pathway for adapting open-source SLMs to the challenging, multilingual Singaporean context—without access to proprietary data or massive compute. By integrating LoRA-based adaptation, dynamic task balancing with CoBa, and the introduction of extensive multilingual QA resources, HT-Moonstone achieves performance that meets or exceeds models up to seven times its size across critical perception and reasoning tasks. The empirical evidence highlights that with carefully engineered fine-tuning protocols and strategic data curation, high-performance SLMs are attainable for specialized, sensitive applications, setting new standards for regional adaptation of large neural architectures.

---

References:

(2607.10092) Efficiently Adapting Spoken LLMs for the Singaporean Context (Gong et al., 2024) CoBa: Convergence Balancer for Multitask Finetuning of LLMs (Liu et al., 17 Jul 2025) Voxtral (Wang et al., 2 Jan 2025) Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models (Chen et al., 6 Nov 2025) CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonese