TaigiSpeech: Taiwanese Hokkien Intent Dataset
- TaigiSpeech is a spoken intent dataset in Taiwanese Taigi, designed for low-resource, elderly-oriented applications such as healthcare and home assistance.
- The corpus includes 3,079 utterances from 21 elderly speakers, capturing natural, scenario-driven speech with intents like emergency alerts and functional commands.
- The study explores scalable mining strategies, using keyword matching with LLM pseudo labeling and an audio-visual framework to manage minimal textual supervision.
Searching arXiv for TaigiSpeech and closely related low-resource speech papers to ground the article. TaigiSpeech is a real-world spoken intent dataset in Taiwanese Taigi, also identified as Taiwanese Hokkien or Southern Min, introduced for spoken language understanding under low-resource and partly unwritten conditions. It is designed for intent detection in practical elderly-oriented scenarios, especially healthcare and home assistant use, and is collected from older adults speaking in scenario-driven settings rather than from read speech. The dataset comprises 21 speakers and 3,079 utterances, and the accompanying study examines two scalable mining strategies for enlarging low-resource intent data: keyword match data mining with LLM pseudo labeling via an intermediate language, and an audio-visual framework with minimal textual supervision. The resource is positioned as the first Taiwanese Hokkien dataset specifically designed for spoken intent recognition and is scheduled for release under the CC BY 4.0 license (Chang et al., 23 Mar 2026).
1. Linguistic, technological, and application setting
TaigiSpeech is situated at the intersection of low-resource speech technology, spoken language understanding, and elderly-oriented assistive computing. The motivating language is Taiwanese Taigi, which the paper characterizes as low-resource, primarily spoken, and partly unwritten. The stated obstacles are limited labeled resources, the absence of a standardized writing system, weak or unavailable ASR, and the concentration of existing SLU benchmarks in high-resource languages such as English, Mandarin, and French. Within that framing, TaigiSpeech addresses a domain in which speech interfaces are not merely convenience tools but a plausible access channel for older adults who may need hands-free communication for emergencies or home automation (Chang et al., 23 Mar 2026).
The application scope is deliberately narrow and operational. The target scenarios include emergency help detection, smart-home command recognition, elderly care or healthcare voice interfaces, emergency notification or triage, and hands-free assistance in low-resource languages. The paper emphasizes Taiwan’s aging population and notes that many older adults in Taiwan primarily speak Taiwanese. The resulting design choice is to focus on utterances relevant to falls, breathing distress, pain, calling for help, calling a contact, and controlling lights. This use-case specificity distinguishes TaigiSpeech from generic command corpora and from read-speech collections assembled primarily for ASR.
A recurrent misconception in low-resource speech research is that “unwritten” necessarily implies the absence of exploitable text. TaigiSpeech does not adopt that position. Instead, it treats weak or indirect textual supervision as usable when available, while also exploring a pathway with minimal text dependence. That distinction is central to the paper’s contribution: the objective is not only to provide a benchmark, but also to examine scalable acquisition strategies for intent data when conventional transcription pipelines are inadequate.
2. Corpus composition and collection protocol
The corpus consists of 21 elderly speakers: 13 female and 8 male. The age range is 54–78, with a mean age of 67.9. The total size is 3,079 utterances. The paper reports total duration as about 6.1 hours in Table 3 and about 6.3 hours in the text, with an average utterance duration of 7.15 seconds. Audio is mono at 48 kHz, and the average number of utterances per speaker is 146.6 (Chang et al., 23 Mar 2026).
| Item | Value |
|---|---|
| Speakers | 21 elderly speakers |
| Gender | 13 female, 8 male |
| Age | 54–78; mean 67.9 |
| Utterances | 3,079 |
| Duration | about 6.1 hours in Table 3; about 6.3 hours in the text |
| Average utterance duration | 7.15 seconds |
| Audio format | mono, 48 kHz |
| Average utterances per speaker | 146.6 |
Collection was conducted in quiet rooms under IRB-approved protocols, with at least one trained moderator present during each session. Participants could use a mobile phone, laptop, desktop, or tablet, with either built-in or external microphones. The recording system supported registration, recording, playback, re-recording, reordering, and submission. Before each recording, a 1-second silent segment was used to measure ambient noise level, and this metadata was logged for each file. These design details matter because the dataset is intended for practical deployment conditions while still maintaining controlled metadata about recording quality.
Participant metadata included age, gender, education level, native language, Taiwanese Hokkien fluency, and hometown. The paper states that hometown was collected to support future study of accent variation. This suggests that TaigiSpeech is not only a task benchmark but also a seed resource for variation-aware research on Taiwanese speech.
3. Scenario elicitation and intent schema
The speech in TaigiSpeech is scenario-driven expressive speech rather than read speech. Participants were given imagined scenarios and encouraged to speak freely, with the explicit goal of eliciting natural, spontaneous speech. Each intent had 20 distinct scenarios, for a total prompt inventory of 160 prompts. Google Gemini Pro 2.5 was used to help generate prompts, and researchers refined them. Google Veo 3 was used to generate short silent or lightly guided scenario videos to help participants imagine the situation (Chang et al., 23 Mar 2026).
The dataset contains eight intents, divided conceptually into four emergency intents and four non-emergency functional commands.
| Category | Intent |
|---|---|
| Emergency | SOS_CALL |
| Emergency | FALL_HELP |
| Emergency | BREATH_EMERG |
| Emergency | PAIN_GENERAL |
| Functional | CALL_CONTACT |
| Functional | LIGHT_ON |
| Functional | LIGHT_OFF |
| Functional | CANCEL_ALERT |
The paper notes that the dataset is balanced conceptually between emergency and non-emergency needs. It also reports per-intent coverage across all eight classes, with the total remaining 3,079 utterances. The collection protocol explicitly allowed direct commands, fragmented utterances, repetitions, and urgent vocalizations. This is significant because it shifts the benchmark away from canonical phrase classification and toward speech that more closely resembles the variability of emergency or home-assistant interaction.
A plausible implication is that TaigiSpeech tests robustness to disfluency and urgency markers as much as lexical recognition. The reported confusion between semantically or lexically adjacent intents later in the paper is consistent with that interpretation, but the dataset design itself already signals that the benchmark is intended to capture more than neat command templates.
4. Scalable data mining strategies
A major contribution of the work is the investigation of two levels of supervision for scalable intent-data mining from weakly structured online video sources. The stronger-supervision route is keyword match data mining with LLM pseudo labeling via an intermediate language; the weaker-supervision route is audio-visual mining with minimal textual supervision. Both are designed for low-resource or unwritten spoken languages, but they make very different assumptions about available side information (Chang et al., 23 Mar 2026).
In the keyword-match pipeline, Mandarin subtitles serve as an intermediate pivot language because Taiwanese Hokkien lacks a widely adopted written standard. The source pool is the Taiwanese Drama dataset, comprising about 7,000 hours of video. For each intent , the authors manually construct a set of Mandarin keywords
A subtitle sentence is retrieved if it contains at least one keyword from . The retrieved segment is then expanded with local context: the subtitle sentence containing the keyword, plus 5 preceding and 5 following subtitle sentences. An LLM decides whether the contextualized segment semantically matches the target intent; if the judgment is positive, the aligned speech segment is extracted. The pipeline also mines roughly 100 daily-life keywords to construct an OTHERS class. The stated advantage is scalability through a high-resource pivot language, while the stated limitation is dependence on subtitle availability, cross-lingual pivoting, and LLM capability in the intermediate language.
In the audio-visual pipeline, the unlabeled pool is represented as
where each clip is
with the audio stream and the visual frames. Intent descriptions are written in a source language such as Mandarin:
A pretrained cross-modal encoder, PE-AV, provides an audio-visual encoder and a text encoder:
0
Similarity is computed by cosine similarity,
1
and the top-2 clips for each intent description are retrieved as pseudo-labeled training data. The paper reports an important simplification: fine-grained emergency distinction was difficult, so the four emergency intents were aggregated for coarse-grained emergency versus non-emergency retrieval. This clarifies that the audio-visual approach is not evaluated under the same level of granularity as the keyword-based pipeline.
5. Experimental protocol, data splits, and model families
The evaluation uses independent-speaker splits. The Taigi test set reserves 6 speakers, with 3 male and 3 female, each contributing 160 utterances, for a total of 960 utterances. For Taigi-adapt experiments, 10 speakers are used, with 1,600 utterances for training, 119 for validation, and 960 for test. The intent is to test on unseen speakers. The mining data sizes reported are 11.3k train, 1.4k validation, and 1.4k test for Keyword Mining, and 3.6k train for Audio-Visual Mining. The drama source pool used for mining included about 28k video clips after keyword filtering; from that pool, the audio-visual setup selected the top 2,000 as emergency data and the last 2,000 as non-emergency data (Chang et al., 23 Mar 2026).
The model suite covers lightweight baselines and SSL-based encoders. The lightweight models are MatchboxNet-S, MatchboxNet-M, and MatchboxNet-L. The SSL models are HuBERT-base, HuBERT-large, WavLM-base, WavLM-base-plus, and WavLM-large. Metrics include accuracy and confidence intervals via bootstrapping; in some confusion-matrix reports, macro F1 is also shown and is close to accuracy because the test set is class-balanced.
The study also includes a cascade baseline in which ASR first transcribes speech and an LLM then infers intent from the transcript. The ASR components include Whisper base, small, medium, and large-v3, along with Qwen3-ASR 0.6B and 1.7B; the LLM is Qwen3-8B. This baseline is important because it tests whether a conventional transcription-first pipeline is competitive in a low-resource, partly unwritten setting where orthography and ASR quality are both problematic.
6. Preliminary results, transfer behavior, and failure modes
Under Keyword Match Mining in the drama-domain 5-class setting, the best results are high: WavLM-large reaches 92.36%, HuBERT-large 90.59%, HuBERT-base 89.60%, and WavLM-base-plus 89.67%. On real TaigiSpeech before adaptation, performance drops markedly, indicating strong domain mismatch between mined drama speech and real elderly speech: WavLM-base-plus reaches 73.23%, HuBERT-base 67.92%, and WavLM-large 70.00%, while MatchboxNet models fall to roughly the 40–50% range. After in-domain fine-tuning in the 8-class Taigi-adapt setting, performance recovers substantially: WavLM-base-plus reaches 90.21%, HuBERT-base 90.10%, and WavLM-base 88.96% (Chang et al., 23 Mar 2026).
Under Audio-Visual Mining, the pattern is weaker throughout. In drama-domain binary emergency classification, SSL models obtain moderate results: WavLM-base-plus 80.55%, HuBERT-base 79.49%, and WavLM-large 81.75%. On TaigiSpeech binary classification, performance is nearly random at around 52–55% for SSL models, with lightweight models worse. After Taigi-adapt fine-tuning, accuracy improves but remains below the keyword-based route: HuBERT-base reaches 86.77%, WavLM-base-plus 87.19%, and WavLM-large 77.08%. The paper interprets this as evidence that audio-visual retrieval signals do not transfer well to real elderly Taigi speech, that binary training may encourage superficial cues, and that cross-lingual and domain mismatch are substantial.
The cascade ASR+LLM baseline underperforms direct end-to-end speech classification. The best cascade combination, Qwen3-ASR 1.7B + Qwen3-8B, reaches 74.48%, well below HuBERT-base at 90.10% and WavLM-base-plus at 90.21% in the adapted end-to-end setting. The paper attributes significance to this contrast because the cascade requires much larger models, more compute, and privacy-sensitive transcription, yet still performs worse.
The confusion analysis identifies two recurring patterns. First, emergency classes are often confused with each other because of overlapping acoustic and lexical cues. Second, LIGHT_ON and LIGHT_OFF are frequently confused because they differ by only one word. These findings align with the dataset design: the benchmark intentionally includes semantically close and pragmatically urgent utterances rather than easily separable keyword commands.
7. Position within low-resource speech research
TaigiSpeech belongs to a broader wave of work on under-resourced and dialectally variable speech technology, but its task is spoken intent recognition rather than speech synthesis. In Taiwanese Hakka, VoxHakka develops a dialectally diverse multi-speaker TTS system using a web scraping pipeline, ASR-based data cleaning, forced alignment, and a YourTTS-based architecture; it supports six dialects and outperforms existing publicly available Hakka TTS systems in naturalness and tone correctness, while pronunciation accuracy remains mixed (Chen et al., 2024). In Tibetan speech synthesis, Tibetan-TTS addresses limited resources, dialectal diversity, and difficult text-to-pronunciation mapping by adapting a large speech synthesis backbone with data quality enhancement, Tibetan-oriented text representation and tokenizer adaptation, and cross-lingual adaptive training; the reported subjective results are MOS 4.28 and 4.35 with pronunciation accuracies 97.6% and 96.6% for the syllable-level and BPE-based systems, respectively (He et al., 4 May 2026).
These adjacent studies are relevant because they show parallel methodological pressures in low-resource speech: noisy or fragmented data sources, dialectal variation, and the need for adaptation rather than task-specific redesign. TaigiSpeech differs in its primary objective, but it shares with these systems a concern for realistic data construction under resource scarcity. This suggests a broader pattern across low-resource speech research: performance depends not only on model family, but also on how successfully data collection, weak supervision, and domain adaptation are matched to the linguistic and social setting.
Within that landscape, TaigiSpeech’s distinctive contribution is the pairing of a real-world elderly Taigi intent corpus with an explicit test of scalable in-the-wild mining. The study’s main conclusion is not that weak supervision alone solves low-resource SLU, but that source-domain mined data are insufficient without in-domain adaptation. For Taiwanese Taigi and other low-resource spoken languages, that result makes the dataset valuable both as a benchmark and as evidence about where transfer learning breaks down.