YT-THDC: Taiwanese Hokkien Drama Corpus

• YT-THDC is a bilingual corpus of 30 hours of Taiwanese Hokkien drama speech paired with loosely aligned Mandarin subtitles and manually verified transcriptions.
• The dataset supports low-resource ASR research by leveraging translation-guided learning to reduce character error rates through auxiliary text inputs.
• It features defined train and test splits, utilizes PGCA in a Whisper-based ASR model, and highlights the challenges of domain-specific, acoustically diverse drama speech.

Searching arXiv for the primary YT-THDC paper and closely related context. Searching for arXiv id (Yang et al., 25 Feb 2026). YT-THDC denotes the YouTube-sourced Taiwanese Hokkien Drama Corpus, a corpus introduced for low-resource Taiwanese Hokkien automatic speech recognition (ASR) in the context of translation-guided learning (Yang et al., 25 Feb 2026). It was created to address a specific resource mismatch: Taiwanese Hokkien speech is abundant in media such as TV dramas and online videos, but transcriptions in Taiwanese Hokkien are scarce, while accompanying subtitles are usually in Mandarin and are only loosely aligned to the spoken content (Yang et al., 25 Feb 2026). YT-THDC is therefore structured as a paired resource in which Taiwanese Hokkien drama speech is associated with both Mandarin subtitles and manually verified Taiwanese Hokkien transcriptions, making it simultaneously a supervised ASR dataset and a benchmark for exploiting subtitle-rich, transcription-poor media through translation-guided learning (Yang et al., 25 Feb 2026).

1. Definition and research setting

YT-THDC is presented as a corpus tailored to the practical conditions of Taiwanese Hokkien ASR, where publicly available spoken material exists in quantity but exact target-language transcripts do not (Yang et al., 25 Feb 2026). The corpus is derived from publicly available YouTube videos, specifically Taiwanese Hokkien drama series, and the authors state in the ethics section that the material comes from official broadcasting channels (Yang et al., 25 Feb 2026). Its source domain is therefore not read speech, broadcast news, or open-ended conversation, but drama speech from online video (Yang et al., 25 Feb 2026).

The corpus is designed around a bilingual supervision structure. Each speech segment is associated with existing Mandarin subtitle text and with a Taiwanese Hokkien transcription that was first bootstrapped by ASR and then corrected by experts (Yang et al., 25 Feb 2026). This arrangement is central to the paper’s translation-guided ASR framework, because the Mandarin subtitle stream provides naturally occurring auxiliary text that can be used directly or translated into additional languages for decoder-side guidance (Yang et al., 25 Feb 2026).

A plausible implication is that YT-THDC is intended not only as a static dataset contribution but also as an experimental model of a broader class of low-resource settings in which target-language speech is available from media archives while aligned annotations are present primarily in a higher-resource contact language.

2. Corpus composition and source characteristics

The paper reports that YT-THDC contains approximately 30 hours of Taiwanese Hokkien drama speech, with the detailed corpus statistics given as follows (Yang et al., 25 Feb 2026).

Split	Duration	Utterances
Train	27.51 hours	50,984
Test	2.79 hours	4,859
Total	30.30 hours	55,843

A development split is not reported in the corpus table or elsewhere (Yang et al., 25 Feb 2026). The paper explicitly reports only train and test (Yang et al., 25 Feb 2026).

The recordings are described as naturally containing multiple speakers, background music, ambient noise, and varied scenes and recording environments (Yang et al., 25 Feb 2026). The corpus is therefore acoustically heterogeneous and linguistically variable, and the paper presents this variability as making the resource useful for studying noise-robust, context-aware ASR (Yang et al., 25 Feb 2026). At the same time, the domain remains narrow: it is limited to drama speech, which the paper identifies as a source of domain bias and a limit on generalization (Yang et al., 25 Feb 2026).

The subtitle stream is integral to the corpus but is not treated as a verbatim transcript of the speech. The paper states that the Mandarin subtitles are “loosely aligned references rather than exact transcriptions of the spoken content” (Yang et al., 25 Feb 2026). This distinction is methodologically important: “Mandarin (GT)” in the experiments denotes the actual subtitle text present in the corpus, not a ground-truth transcription of the spoken Taiwanese Hokkien (Yang et al., 25 Feb 2026).

3. Data creation and annotation process

The paper gives a high-level description of corpus construction. Taiwanese Hokkien speech was initially transcribed using a pre-trained ASR model, and those transcriptions were then manually verified and refined by linguistic experts (Yang et al., 25 Feb 2026). The resulting Hokkien text is therefore neither raw subtitle text nor uncorrected machine output; it is expert-corrected supervision derived from an ASR-assisted annotation workflow (Yang et al., 25 Feb 2026).

The Mandarin subtitle text was already present in the source YouTube videos and is treated as aligned auxiliary text, but the exact alignment mechanism is not specified (Yang et al., 25 Feb 2026). The paper does not state whether the alignment was inherited directly from subtitle timing, improved through ASR/subtitle matching, or refined manually (Yang et al., 25 Feb 2026). Likewise, it does not specify:

• the identity or architecture of the pre-trained ASR model used for initial transcription,
• whether that model was trained on Taiwanese Hokkien, Mandarin, or multilingual data,
• how utterance segmentation was created,
• whether segmentation was subtitle-based, VAD-based, or manually defined,
• whether experts corrected only text or also segment boundaries,
• speaker counts or demographics,
• annotation guidelines,
• inter-annotator agreement,
• or total human correction effort (Yang et al., 25 Feb 2026).

This leaves the corpus well characterized at the level of purpose and aggregate composition, but only partially specified at the level of detailed annotation protocol. A plausible implication is that reproducibility of corpus construction depends on information not fully exposed in the paper, even though the dataset is clearly defined as speech segments paired with Mandarin subtitle text and manually verified Taiwanese Hokkien transcriptions (Yang et al., 25 Feb 2026).

4. Text supervision, representation, and alignment assumptions

The corpus’s most distinctive feature is the coexistence of two textual channels: Mandarin subtitles and Taiwanese Hokkien transcriptions (Yang et al., 25 Feb 2026). This makes YT-THDC suitable for a research design in which direct target-language supervision is supplemented by semantically related text in another language (Yang et al., 25 Feb 2026).

In the experiments, the Mandarin subtitles serve as the corpus-native auxiliary language, denoted “Mandarin (GT)” (Yang et al., 25 Feb 2026). The paper also uses SeamlessM4T to translate these subtitles into other languages, including English, Hindi, Spanish, and French, after which the translated texts are encoded by mBERT Base and supplied to the ASR decoder (Yang et al., 25 Feb 2026). The corpus thus functions not merely as a speech-transcription dataset but as a bilingual anchor for constructing multilingual auxiliary supervision.

Text normalization and orthographic conventions are only partially specified. The paper evaluates using character error rate (CER) and motivates that choice by noting that word boundaries can be ambiguous and tokenization standards may vary in Taiwanese Hokkien (Yang et al., 25 Feb 2026). This implies that target-side evaluation is performed at the character level (Yang et al., 25 Feb 2026). However, the paper does not specify the script standard used for Taiwanese Hokkien, whether romanization appears, whether punctuation is retained, how numerals are normalized, or whether any traditional/simplified normalization is applied (Yang et al., 25 Feb 2026). It also does not formally define the corpus orthography, though example tokens in the attention analysis suggest representation in Chinese characters (Yang et al., 25 Feb 2026).

A few audio-processing constraints are described in the experimental setup rather than the corpus section. Audio inputs are constrained to 10 seconds, the maximum raw audio length is 160,000 samples, and acoustic features use 80 mel-frequency bins (Yang et al., 25 Feb 2026). From 160,000 samples over 10 seconds, one can infer a 16 kHz sampling rate in the processing pipeline, but the paper does not explicitly state the native corpus sample rate or whether the audio was resampled (Yang et al., 25 Feb 2026).

5. Role in translation-guided ASR

YT-THDC is the sole benchmark used to evaluate the paper’s TG-ASR framework, which is described as a translation-guided ASR method for low-resource Taiwanese Hokkien drama speech recognition (Yang et al., 25 Feb 2026). The model backbone is Whisper Small, trained in two stages (Yang et al., 25 Feb 2026). In Stage 1, the full Whisper model is fine-tuned on Taiwanese Hokkien speech with batch size 4, learning rate $1.25 \times 10^{-5}$, 80,000 steps, and 8,000 warm-up steps (Yang et al., 25 Feb 2026). In Stage 2, training resumes from the best Stage 1 checkpoint, the proposed PGCA layers are added, the Whisper encoder and prior decoder parameters are frozen, and only the PGCA layers are updated with batch size 8, learning rate $5.0 \times 10^{-5}$, 180,000 steps, and 30,000 warm-up steps (Yang et al., 25 Feb 2026). Optimization uses AdamW, weight decay 0.01, and FP16 mixed precision (Yang et al., 25 Feb 2026).

Auxiliary texts are encoded by mBERT Base with 12 layers, 768 hidden units, and 12 attention heads, and mBERT remains frozen (Yang et al., 25 Feb 2026). The main multilingual configuration uses $L=5$ auxiliary languages: Mandarin, English, Hindi, Spanish, and French (Yang et al., 25 Feb 2026). Mandarin comes directly from the corpus subtitles, while the other languages are generated from those subtitles by machine translation (Yang et al., 25 Feb 2026).

The paper’s core architectural mechanism is parallel gated cross-attention (PGCA), which injects auxiliary translation embeddings into the Whisper decoder through multiple language-specific attention branches with learnable gates (Yang et al., 25 Feb 2026). The fusion is written as

$$\mathbf{Y}' = \mathbf{Y} + \sum_{l=1}^{L} \tanh(\alpha_{\operatorname{attn}^{(l)}})\times \operatorname{attn}(\mathbf{Y}, \mathbf{E}_l, \mathbf{E}_l),$$

$$\mathbf{Z} = \mathbf{Y}' + \tanh(\alpha_{\operatorname{FNN}})\times \operatorname{FNN}(\mathbf{Y}').$$

Here, $\mathbf{Y} \in \mathbb{R}^{T_y \times d}$ denotes decoder input states, $\mathbf{E}_l \in \mathbb{R}^{T_l \times d}$ the embedding sequence for auxiliary language $l$, and $L$ the number of auxiliary languages (Yang et al., 25 Feb 2026). The gating parameters are initialized to zero so that the new path begins with minimal interference, which the paper presents as a stabilization measure (Yang et al., 25 Feb 2026). The PGCA modules are inserted at the outset of each Whisper decoder block (Yang et al., 25 Feb 2026).

Within this setup, YT-THDC is not incidental. Its subtitle/transcript pairing is the enabling condition that allows the model to receive speech plus auxiliary textual meaning cues derived from subtitles while still being trained against manually verified Taiwanese Hokkien targets (Yang et al., 25 Feb 2026).

6. Experimental results and benchmark function

The paper reports character error rate (CER) as the main evaluation metric and notes that the reported CER values are computed under teacher-forcing decoding, so the measurements isolate the effect of translation guidance rather than fully autoregressive free-running behavior (Yang et al., 25 Feb 2026).

On YT-THDC, the baseline ASR model trained only on Taiwanese Hokkien transcripts achieves 13.40% CER (Yang et al., 25 Feb 2026). Adding auxiliary text sources gives the following results (Yang et al., 25 Feb 2026).

System	CER
A0 Baseline	13.40%
A1 Mandarin (GT)	11.87%
A2 Hindi	13.17%
A3 English	13.10%
A4 French	12.98%
A5 Spanish	12.84%
A6 Mandarin (GT) + Spanish	11.42%

The paper reports the relative reductions over the no-auxiliary baseline as 11.42% for Mandarin (GT), 1.72% for Hindi, 2.24% for English, 3.13% for French, 4.18% for Spanish, and 14.77% for Mandarin (GT) + Spanish (Yang et al., 25 Feb 2026). The headline experimental result is therefore a 14.77% relative reduction in CER when combining Mandarin subtitle guidance with Spanish translation guidance (Yang et al., 25 Feb 2026).

The paper interprets these findings as showing that Mandarin is the most effective single auxiliary language because it is closest semantically and is directly available in the source media, while machine-translated auxiliary languages can still help despite translation noise (Yang et al., 25 Feb 2026). It further reports that Spanish is the most effective among the translated languages and that combining languages can outperform Mandarin alone, suggesting complementary semantic cues (Yang et al., 25 Feb 2026).

Ablation results on YT-THDC are used to validate the PGCA design (Yang et al., 25 Feb 2026). The reported CER values are 11.42 for Full PGCA, 11.46 w/o tanh gating, 11.60 for Sequential Attention, 12.00 for Shared Attention, 27.68 for Addition, and 24.09 for Concatenation (Yang et al., 25 Feb 2026). These comparisons are used in the paper to argue that gating helps, parallel attention is preferable to sequential attention, language-specific independent attention branches are preferable to shared weights, and naive fusion performs poorly (Yang et al., 25 Feb 2026).

The paper also compares translation sources for the best two-language setting and reports 11.42 CER using SeamlessM4T versus 11.52 CER using NLLB (Yang et al., 25 Feb 2026).

7. Limitations, release conditions, and scientific significance

The paper explicitly identifies several limitations of YT-THDC (Yang et al., 25 Feb 2026). First, the corpus is small, at about 30 hours, and restricted to the drama domain (Yang et al., 25 Feb 2026). Second, drama speech introduces acoustic/domain bias, including scripted language, music, and dramatic prosody, rather than ordinary spontaneous speech (Yang et al., 25 Feb 2026). Third, the Mandarin subtitles are not exact transcripts of the spoken Taiwanese Hokkien and may include paraphrases, omissions, or semantic simplifications (Yang et al., 25 Feb 2026). Fourth, auxiliary-language texts created by machine translation may introduce translation noise and biases or misalignments affecting model behavior (Yang et al., 25 Feb 2026). Fifth, the experimental conclusions are confined to Taiwanese Hokkien, and transferability to other low-resource languages is not established (Yang et al., 25 Feb 2026).

The dataset is explicitly described as released, available for research use only, and restricted to non-commercial research and educational purposes (Yang et al., 25 Feb 2026). The paper does not provide a URL, repository, formal license name, or access procedure in the provided description (Yang et al., 25 Feb 2026).

In scientific terms, YT-THDC functions as both a resource contribution and an enabling corpus (Yang et al., 25 Feb 2026). It is a resource because it supplies 30.30 hours of Taiwanese Hokkien drama speech paired with Mandarin subtitles and manually verified Taiwanese Hokkien transcriptions (Yang et al., 25 Feb 2026). It is enabling because it operationalizes a realistic low-resource condition in which speech is abundant, exact target-language transcripts are scarce, and another-language subtitle stream is available for exploitation through translation-guided learning (Yang et al., 25 Feb 2026). This suggests that the corpus is significant less as a generic speech benchmark than as a concrete experimental substrate for studying subtitle-driven multilingual supervision in under-resourced ASR.