StarDrinks: An English and Korean Test Set for SLU Evaluation in a Drink Ordering Scenario

Abstract: LLMs and speech assistants are increasingly used for task-oriented interactions, yet their evaluation often relies on controlled scenarios that fail to capture the variability and complexity of real user requests. Drink ordering, for example, involves diverse named entities, drink types, sizes, customizations, and brand-specific terminology, as well as spontaneous speech phenomena such as hesitations and self-corrections. To address this gap, we introduce StarDrinks, a test set in English and Korean containing speech utterances features, transcriptions, and annotated slots. Our dataset supports speech-to-slots SLU, transcription-to-slots NLU, and speech-to-transcription ASR evaluation, providing a realistic benchmark for model robustness and generalization in a linguistically rich, real-world task.

• The paper presents a novel dataset capturing realistic, multilingual drink orders with detailed slot annotations to enhance SLU evaluation.
• It uses a data-centric pipeline that processes receipt-derived orders, crowdsourced natural speech recordings, and rigorous multi-stage annotation.
• Baseline experiments reveal that ASR noise and rare named entities significantly reduce slot-filling accuracy, underlining challenges in adaptation and cross-lingual robustness.

StarDrinks: A Multilingual Test Set for Realistic SLU Evaluation in Drink Ordering

Motivation and Context

The assessment of Spoken Language Understanding (SLU) systems in practical deployment domains, such as task-oriented dialog agents, suffers from a notable lack of test sets mirroring authentic user interactions—especially in linguistically nuanced scenarios like drink ordering. Existing benchmarks center primarily on clean speech and text with limited entity coverage and minimal representation of the diverse, spontaneous linguistic phenomena observed in realistic conversations. The "StarDrinks: An English and Korean Test Set for SLU Evaluation in a Drink Ordering Scenario" (2604.26500) initiative proposes a dataset capturing these complexities in both English and Korean, with a granular structure supporting several evaluation paradigms (SLU, NLU, and ASR).

Dataset Construction and Characteristics

StarDrinks adopts a data-centric pipeline (Figure 1) grounded in real-world drink order receipts from a popular South Korean coffee chain. The data generation proceeds through three main stages: structured order extraction and expansion, natural utterance recording by native speakers, and rigorous annotation/validation of both slot labels and textual transcriptions. This yields a test corpus of 550+ annotated spoken orders, evenly split between English and Korean.

Figure 1: Overview of the StarDrinks data pipeline, mapping from real-world order receipts to spoken utterance collection and annotation.

Recordings were crowdsourced using Prolific, ensuring naturalness by instructing participants to order drinks based solely on the receipt attributes, allowing for lexical variability and spontaneous phenomena (e.g., hesitations, self-corrections). Notably, the dataset introduces coverage of multiple simultaneous drinks, a rich customization schema (e.g., temperature, milk type, syrup type/amount, topping customizations), and numerous named entities, including rare menu items and adaptation-prone terminology.

Figure 2: Example utterances from both the English and Korean splits, highlighting linguistic diversity and slot annotation structure.

A sample recording interface is depicted in Figure 3. All utterances underwent correction for accurate slot alignment—incorrect or misaligned samples were excluded, resulting in high-quality, error-analyzed evaluation data.

Figure 3: Data collection interface for producing natural English drink order utterances during Prolific annotation.

The annotation schema encodes 15 slot types, covering core properties and fine-grained customization (e.g., adding shots, sweetener levels, multi-entity grouping). This semantic granularity enables precise evaluation of slot filling and robustness to natural variation. Table and figure references in the source paper detail full slot/type coverage.

Evaluation Scenarios and Baseline Experiments

The StarDrinks test set enables three core SLU scenarios:

• ASR Evaluation: Speech-to-text recognition robustness.
• NLU Evaluation: Text-to-slot mapping performance from gold transcriptions.
• SLU Evaluation: Speech-to-slot-pair transformation, involving ASR as an intermediate step.

Prompts for task-oriented slot extraction using LLMs can be flexibly constructed (Figure 4), supporting both zero-shot and few-shot paradigms.

Figure 4: Illustrative 3-shot English NLU prompt incorporating the StarDrinks schema for GPT-4o slot filling.

Baseline performance, using whisper-large-v3 for ASR and GPT-4o for NLU/SLU, demonstrates the challenge of the test set: even advanced models exhibit notable degradation, especially with automatic (ASR-derived) input. Whisper-large-v3 achieves a WER of 9.2% (English) and 22.9% (Korean), with qualitative output analyses exposing consistent struggles with named entities and menu-specific terminology, reinforcing the test set’s value for adaptation research.

In the slot filling pipeline, 3-shot prompting substantially boosts accuracy (English UEM from 71.76% to 87.06%), with similar trends in Korean. However, ASR noise significantly reduces exact-match rates (84.31% SLU UEM vs. 87.06% NLU in English 3-shot), and zero-shot settings underperform across the board. Slot-level F1 metrics approach 98% in best-case 3-shot NLU; this relative slotwise robustness contrasts with utterance-level fragility due to compounding slot errors.

Theoretical and Practical Implications

StarDrinks surfaces critical generalization failures in current ASR and slot filling models. The primary challenge is the effective handling of OOV named entities and the combinatorial diversity of menu items not seen at training time. The dataset’s design, derived from authentic orders and annotated for spontaneous disfluencies, allows for examination of model robustness to lexical, syntactic, and prosodic variability not systematically present in prior corpora (e.g., SNIPS, SLURP, MASSIVE).

For practical SLU agents (e.g., automated ordering kiosks), near-perfect slot accuracy is a deployment requirement. The results on StarDrinks, with strict UEM in the high 80s at best, reveal significant headroom for improvement. The strong negative impact of noisy ASR, especially for Korean and in cases with rare entities, highlights the paramount need for research on context-biasing, test-time adaptation, and multimodal integration. Recent work on memory-augmented ASR adaptation strategies and continual test-time learning offers promising directions but requires rigorous benchmarking, which StarDrinks is positioned to catalyze.

Additionally, the inclusion of both English and Korean facilitates meaningful cross-lingual SLU evaluation and pushes toward more robust, language-agnostic model architectures.

Future Directions

Several research avenues become salient with the introduction of StarDrinks:

• Domain Adaptation and Test-Time Biasing: Leveraging menu context and entity dictionaries for in-domain adaptation.
• Few-shot/Fine-tuning Paradigms: Investigating efficient schema generalization in multilingual SLU/ASR.
• Robustness to Spontaneous Speech: Probing models’ ability to handle naturally occurring speech phenomena, moving beyond scripted data.
• Cross-lingual Transfer: Exploring transfer learning and joint training for hard-to-resource language pairs.

The dataset also invites extensions—additional domains (e.g., full-service restaurants), richer conversational multi-turn interactions, and injected adversarial phenomena for stress-testing real-world SLU systems.

Conclusion

StarDrinks provides a comprehensive, multilingual, and realistic test set for evaluating SLU robustness in a widely deployed dialog setting, filling a critical void left by prior resources. Its unique structure and linguistic diversity serve both as a challenging benchmark and as a foundation for developing more adaptable, resilient speech understanding models. The evidence from baseline experiments solidifies the dataset’s utility for both empirical benchmarking and as a springboard for further research into NLU/SLU adaptability, context integration, and multilingual robustness (2604.26500).