CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents

Abstract: Task-oriented voice agents need to map spoken user requests to structured outputs such as semantic frames, executable actions, and function calls. A common approach is to cascade ASR with a text-based LLM, but transcription errors can propagate to downstream structured output generation, especially under noisy conditions. Spoken LLMs (SLMs) offer a direct speech-based alternative, yet adapting them to new tasks typically requires paired speech-target annotations. Motivated by this gap, we present CORTIS, a text-only adaptation framework for task-oriented voice agents. CORTIS fine-tunes SLMs using text-form task supervision, enabling speech-based structured output generation at inference time without task-specific speech-target annotations during adaptation. We evaluate CORTIS on two Qwen2.5-Omni backbones and three task-oriented speech datasets, including an in-house product dataset, and compare it with matched ASR-LLM cascades trained with the same text-form task supervision. Results show that CORTIS performs competitively with matched cascades and offers clearer advantages under acoustic degradation, particularly in preserving high-level task semantics. These findings suggest that text-only fine-tuning of SLMs can serve as a practical adaptation strategy for voice agents when paired speech-target data are costly to collect.

• The paper presents a novel text-only adaptation strategy that enables spoken language models to generate structured outputs from speech without paired speech annotations.
• It leverages pretrained speech–text alignment by freezing the speech encoder and fine-tuning only the language model component via next-token prediction.
• Experimental results demonstrate significant improvements in semantic frame accuracy and intent recognition under noise compared to traditional ASR–LLM cascades.

Text-Only Adaptation of Spoken LLMs for Task-Oriented Voice Agents: The CORTIS Framework

Motivation and Problem Setting

Task-oriented voice agents require robust mapping from spoken user queries to structured outputs such as semantic frames, executable actions, or function calls. The prevailing industry architecture involves cascading automatic speech recognition (ASR) with a text-based LLM, but this design introduces vulnerability to ASR errors which propagate downstream and particularly degrade performance under adverse acoustic conditions. Spoken LLMs (SLMs) offer direct processing from speech to structured outputs, yet conventional adaptation mandates paired speech–target annotations, increasing annotation cost and limiting scalability for new tasks.

CORTIS addresses this gap by enabling annotation-efficient adaptation via text-only supervised instruction fine-tuning. It leverages the pretrained cross-modal alignment of SLMs, allowing them to be adapted by text-form task supervision and used for speech-based inference without task-specific speech–target data.

Figure 1: Overview of the CORTIS framework, showcasing text-only fine-tuning of SLMs for structured output generation from speech at inference.

Methodological Contributions

CORTIS formalizes an adaptation pipeline comprising (1) text-only task supervision, (2) frozen speech-related modules of the SLM during fine-tuning, and (3) strict prompt-position consistency between training and inference. During adaptation, only the LLM component receives gradients, trained via next-token prediction on task instructions paired with serialized structured outputs in a unified prompt template. At inference, the spoken input substitutes the text query within the same template, enabling end-to-end speech-to-structure mapping without an intermediate transcription step.

Implementation constraints include explicitly freezing the speech encoder and modality adapter to preserve cross-modal transfer without incurring mismatched optimization from text-only batches. The architecture leverages Qwen2.5-Omni SLM backbones and maintains consistent task schema across training and inference, exploiting the speech–text alignment learned during SLM pretraining.

Experimental Design

The evaluation benchmark includes three task-oriented speech datasets: (1) Fluent Speech Commands (FSC), (2) SLURP, and (3) an in-house smart home function calling dataset. Metrics span exact match (EM) for semantic frames and function calls, as well as intent accuracy, entity F1, and SLU-F1 for SLURP, reflecting both high-level and slot-value prediction reliability.

Baselines are matched ASR–LLM cascades utilizing Whisper large-v3 for transcription and Qwen2.5-Instruct LLMs for downstream processing, trained on identical text-form task supervision as CORTIS. Models are tested across clean and babble-noise conditions (from 15 dB to 0 dB SNR), assessing robustness and performance tradeoffs. Oracle-text settings (using gold transcripts) provide an upper-bound for downstream modeling in the absence of ASR errors.

Results: Quantitative Analysis

Figure 2: Performance of CORTIS-tuned SLMs versus ASR-LLM cascades under increasing babble noise across FSC, SLURP, and the in-house dataset.

On FSC, CORTIS consistently surpasses the cascade baseline under all noise regimes: with Qwen2.5-Omni-7B, the EM gain increases from 1.00 under clean speech to 5.59 at 0 dB SNR. In SLURP, CORTIS yields pronounced improvements in intent accuracy (up to 8.78 percentage points over the cascade at 5 dB SNR), though gains in entity F1 and SLU-F1 are more modest. In the in-house dataset, CORTIS demonstrates superior robustness: in the 7B setting, CORTIS maintains EM of 55.33 at 0 dB compared to 44.84 for the cascade, an absolute gain of 10.49.

The results substantiate a contradictory claim against traditional expectations—text-only adapted SLMs match or exceed cascaded systems trained on the same supervision, especially for command understanding and semantic prediction under degraded speech. However, slot-level entity grounding remains challenging for direct speech inference, with occasional errors in fine-grained slot values.

Qualitative Analysis

Qualitative examples highlight that CORTIS-tuned SLMs are able to recover high-level task semantics even when ASR outputs are corrupted (e.g., “Heat down” mis-transcribed as “Feet down”), illustrating resilience to recognition errors. Nonetheless, failures persist in exact slot-value prediction, which can be superior in cascaded systems when ASR transcripts are clean.

Practical and Theoretical Implications

CORTIS demonstrates that text-only adaptation of SLMs is a practical strategy when collecting paired speech–target task data is prohibitive. For real-world deployment, CORTIS enables rapid agent adaptation for emerging domains by leveraging existing text data and maintaining speech input capability. Robustness under noise and the efficiency of adaptation underpin its utility for on-device and user-facing agents.

Theoretical implications center on the cross-modal transfer efficiency in SLMs: pretrained speech–text alignment can generalize task instruction fine-tuning from text data to speech inference. This raises future questions on modality-grounded parameterization, schema generalization, and the limits of cross-modal transfer for slot-level accuracy.

Directions for Future Research

Future work should explore hybrid adaptation strategies, including text denoising objectives, curriculum learning, or limited paired speech data supplementation, to further enhance robustness and slot-value prediction. Generalization across different SLM families and architectures warrants systematic investigation, as preliminary findings report varying efficacy (e.g., Qwen2-Audio underperforms in the in-house product domain).

Additional real-world robustness testing, encompassing reverberation, far-field input, and device-specific noise, is critical for characterizing deployment performance.

Conclusion

CORTIS delivers a text-only adaptation framework for task-oriented voice agents, achieving competitive or superior performance compared to matched ASR–LLM cascades for high-level semantic prediction under noise. Its annotation efficiency facilitates scalable agent development, though exact slot-value grounding remains a challenge. CORTIS embodies a viable, practical, and robust methodology that advances the paradigm of speech-based structured output generation in multimodal LLMs.