---
title: 'YT-THDC: Taiwanese Hokkien Drama Corpus'
url: https://www.emergentmind.com/topics/yt-thdc
type: topic
---

# YT-THDC: Taiwanese Hokkien Drama Corpus

Searching arXiv for the primary YT-THDC paper and closely related context.
Searching for arXiv id 2602.22039.
YT-THDC denotes the **YouTube-sourced Taiwanese Hokkien Drama Corpus**, a corpus introduced for **low-resource Taiwanese Hokkien automatic speech recognition (ASR)** in the context of translation-guided learning [2602.22039]. It was created to address a specific resource mismatch: Taiwanese Hokkien speech is abundant in media such as TV dramas and online videos, but **transcriptions in Taiwanese Hokkien are scarce**, while accompanying subtitles are usually in **Mandarin** and are only loosely aligned to the spoken content [2602.22039]. YT-THDC is therefore structured as a paired resource in which Taiwanese Hokkien drama speech is associated with both **Mandarin subtitles** and **manually verified Taiwanese Hokkien transcriptions**, making it simultaneously a supervised ASR dataset and a benchmark for exploiting subtitle-rich, transcription-poor media through **translation-guided learning** [2602.22039].

## 1. Definition and research setting

YT-THDC is presented as a corpus tailored to the practical conditions of Taiwanese Hokkien ASR, where publicly available spoken material exists in quantity but exact target-language transcripts do not [2602.22039]. The corpus is derived from **publicly available YouTube videos**, specifically **Taiwanese Hokkien drama series**, and the authors state in the ethics section that the material comes from **official broadcasting channels** [2602.22039]. Its source domain is therefore not read speech, broadcast news, or open-ended conversation, but **drama speech from online video** [2602.22039].

The corpus is designed around a bilingual supervision structure. Each speech segment is associated with existing **Mandarin subtitle text** and with a **Taiwanese Hokkien transcription** that was first bootstrapped by ASR and then corrected by experts [2602.22039]. This arrangement is central to the paper’s translation-guided ASR framework, because the Mandarin subtitle stream provides naturally occurring auxiliary text that can be used directly or translated into additional languages for decoder-side guidance [2602.22039].

A plausible implication is that YT-THDC is intended not only as a static dataset contribution but also as an experimental model of a broader class of low-resource settings in which target-language speech is available from media archives while aligned annotations are present primarily in a higher-resource contact language.

## 2. Corpus composition and source characteristics

The paper reports that YT-THDC contains approximately **30 hours** of Taiwanese Hokkien drama speech, with the detailed corpus statistics given as follows [2602.22039].

| Split | Duration | Utterances |
|---|---:|---:|
| Train | 27.51 hours | 50,984 |
| Test | 2.79 hours | 4,859 |
| Total | 30.30 hours | 55,843 |

A **development split is not reported** in the corpus table or elsewhere [2602.22039]. The paper explicitly reports only **train** and **test** [2602.22039].

The recordings are described as naturally containing **multiple speakers**, **background music**, **ambient noise**, and varied scenes and recording environments [2602.22039]. The corpus is therefore acoustically heterogeneous and linguistically variable, and the paper presents this variability as making the resource useful for studying **noise-robust, context-aware ASR** [2602.22039]. At the same time, the domain remains narrow: it is limited to **drama speech**, which the paper identifies as a source of domain bias and a limit on generalization [2602.22039].

The subtitle stream is integral to the corpus but is not treated as a verbatim transcript of the speech. The paper states that the Mandarin subtitles are **“loosely aligned references rather than exact transcriptions of the spoken content”** [2602.22039]. This distinction is methodologically important: “Mandarin (GT)” in the experiments denotes the actual subtitle text present in the corpus, not a ground-truth transcription of the spoken Taiwanese Hokkien [2602.22039].

## 3. Data creation and annotation process

The paper gives a high-level description of corpus construction. Taiwanese Hokkien speech was **initially transcribed using a pre-trained ASR model**, and those transcriptions were then **manually verified and refined by linguistic experts** [2602.22039]. The resulting Hokkien text is therefore neither raw subtitle text nor uncorrected machine output; it is expert-corrected supervision derived from an ASR-assisted annotation workflow [2602.22039].

The Mandarin subtitle text was already present in the source YouTube videos and is treated as aligned auxiliary text, but the exact alignment mechanism is **not specified** [2602.22039]. The paper does not state whether the alignment was inherited directly from subtitle timing, improved through ASR/subtitle matching, or refined manually [2602.22039]. Likewise, it does not specify:

- the identity or architecture of the pre-trained ASR model used for initial transcription,
- whether that model was trained on Taiwanese Hokkien, Mandarin, or multilingual data,
- how utterance segmentation was created,
- whether segmentation was subtitle-based, VAD-based, or manually defined,
- whether experts corrected only text or also segment boundaries,
- speaker counts or demographics,
- annotation guidelines,
- inter-annotator agreement,
- or total human correction effort [2602.22039].

This leaves the corpus well characterized at the level of purpose and aggregate composition, but only partially specified at the level of detailed annotation protocol. A plausible implication is that reproducibility of corpus construction depends on information not fully exposed in the paper, even though the dataset is clearly defined as speech segments paired with Mandarin subtitle text and manually verified Taiwanese Hokkien transcriptions [2602.22039].

## 4. Text supervision, representation, and alignment assumptions

The corpus’s most distinctive feature is the coexistence of two textual channels: **Mandarin subtitles** and **Taiwanese Hokkien transcriptions** [2602.22039]. This makes YT-THDC suitable for a research design in which direct target-language supervision is supplemented by semantically related text in another language [2602.22039].

In the experiments, the Mandarin subtitles serve as the corpus-native auxiliary language, denoted **“Mandarin (GT)”** [2602.22039]. The paper also uses **SeamlessM4T** to translate these subtitles into other languages, including **English, Hindi, Spanish, and French**, after which the translated texts are encoded by **mBERT Base** and supplied to the ASR decoder [2602.22039]. The corpus thus functions not merely as a speech-transcription dataset but as a bilingual anchor for constructing multilingual auxiliary supervision.

Text normalization and orthographic conventions are only partially specified. The paper evaluates using **character error rate (CER)** and motivates that choice by noting that word boundaries can be ambiguous and tokenization standards may vary in Taiwanese Hokkien [2602.22039]. This implies that target-side evaluation is performed at the **character level** [2602.22039]. However, the paper does **not** specify the script standard used for Taiwanese Hokkien, whether romanization appears, whether punctuation is retained, how numerals are normalized, or whether any traditional/simplified normalization is applied [2602.22039]. It also does not formally define the corpus orthography, though example tokens in the attention analysis suggest representation in **Chinese characters** [2602.22039].

A few audio-processing constraints are described in the experimental setup rather than the corpus section. Audio inputs are constrained to **10 seconds**, the maximum raw audio length is **160,000 samples**, and acoustic features use **80 mel-frequency bins** [2602.22039]. From 160,000 samples over 10 seconds, one can infer a **16 kHz sampling rate** in the processing pipeline, but the paper does not explicitly state the native corpus sample rate or whether the audio was resampled [2602.22039].

## 5. Role in translation-guided ASR

YT-THDC is the sole benchmark used to evaluate the paper’s **TG-ASR** framework, which is described as a translation-guided ASR method for low-resource Taiwanese Hokkien drama speech recognition [2602.22039]. The model backbone is **Whisper Small**, trained in two stages [2602.22039]. In **Stage 1**, the full Whisper model is fine-tuned on Taiwanese Hokkien speech with batch size **4**, learning rate **\(1.25 \times 10^{-5}\)**, **80,000** steps, and **8,000** warm-up steps [2602.22039]. In **Stage 2**, training resumes from the best Stage 1 checkpoint, the proposed **PGCA** layers are added, the Whisper encoder and prior decoder parameters are **frozen**, and only the **PGCA layers** are updated with batch size **8**, learning rate **\(5.0 \times 10^{-5}\)**, **180,000** steps, and **30,000** warm-up steps [2602.22039]. Optimization uses **AdamW**, weight decay **0.01**, and **FP16** mixed precision [2602.22039].

Auxiliary texts are encoded by **mBERT Base** with **12 layers**, **768 hidden units**, and **12 attention heads**, and mBERT remains **frozen** [2602.22039]. The main multilingual configuration uses **\(L=5\)** auxiliary languages: **Mandarin, English, Hindi, Spanish, and French** [2602.22039]. Mandarin comes directly from the corpus subtitles, while the other languages are generated from those subtitles by machine translation [2602.22039].

The paper’s core architectural mechanism is **parallel gated cross-attention (PGCA)**, which injects auxiliary translation embeddings into the Whisper decoder through multiple language-specific attention branches with learnable gates [2602.22039]. The fusion is written as

\[
\mathbf{Y}' = \mathbf{Y} + \sum_{l=1}^{L} \tanh(\alpha_{\operatorname{attn}^{(l)}})\times \operatorname{attn}(\mathbf{Y}, \mathbf{E}_l, \mathbf{E}_l),
\]

\[
\mathbf{Z} = \mathbf{Y}' + \tanh(\alpha_{\operatorname{FNN}})\times \operatorname{FNN}(\mathbf{Y}').
\]

Here, \(\mathbf{Y} \in \mathbb{R}^{T_y \times d}\) denotes decoder input states, \(\mathbf{E}_l \in \mathbb{R}^{T_l \times d}\) the embedding sequence for auxiliary language \(l\), and \(L\) the number of auxiliary languages [2602.22039]. The gating parameters are initialized to zero so that the new path begins with minimal interference, which the paper presents as a stabilization measure [2602.22039]. The PGCA modules are inserted **at the outset of each Whisper decoder block** [2602.22039].

Within this setup, YT-THDC is not incidental. Its subtitle/transcript pairing is the enabling condition that allows the model to receive **speech plus auxiliary textual meaning cues** derived from subtitles while still being trained against manually verified Taiwanese Hokkien targets [2602.22039].

## 6. Experimental results and benchmark function

The paper reports **character error rate (CER)** as the main evaluation metric and notes that the reported CER values are computed under **teacher-forcing decoding**, so the measurements isolate the effect of translation guidance rather than fully autoregressive free-running behavior [2602.22039].

On YT-THDC, the baseline ASR model trained only on Taiwanese Hokkien transcripts achieves **13.40% CER** [2602.22039]. Adding auxiliary text sources gives the following results [2602.22039].

| System | CER |
|---|---:|
| A0 Baseline | 13.40% |
| A1 Mandarin (GT) | 11.87% |
| A2 Hindi | 13.17% |
| A3 English | 13.10% |
| A4 French | 12.98% |
| A5 Spanish | 12.84% |
| A6 Mandarin (GT) + Spanish | 11.42% |

The paper reports the relative reductions over the no-auxiliary baseline as **11.42%** for **Mandarin (GT)**, **1.72%** for **Hindi**, **2.24%** for **English**, **3.13%** for **French**, **4.18%** for **Spanish**, and **14.77%** for **Mandarin (GT) + Spanish** [2602.22039]. The headline experimental result is therefore a **14.77% relative reduction in CER** when combining Mandarin subtitle guidance with Spanish translation guidance [2602.22039].

The paper interprets these findings as showing that Mandarin is the most effective single auxiliary language because it is closest semantically and is directly available in the source media, while machine-translated auxiliary languages can still help despite translation noise [2602.22039]. It further reports that **Spanish** is the most effective among the translated languages and that combining languages can outperform Mandarin alone, suggesting complementary semantic cues [2602.22039].

Ablation results on YT-THDC are used to validate the PGCA design [2602.22039]. The reported CER values are **11.42** for **Full PGCA**, **11.46** **w/o tanh gating**, **11.60** for **Sequential Attention**, **12.00** for **Shared Attention**, **27.68** for **Addition**, and **24.09** for **Concatenation** [2602.22039]. These comparisons are used in the paper to argue that gating helps, parallel attention is preferable to sequential attention, language-specific independent attention branches are preferable to shared weights, and naive fusion performs poorly [2602.22039].

The paper also compares translation sources for the best two-language setting and reports **11.42 CER** using **SeamlessM4T** versus **11.52 CER** using **NLLB** [2602.22039].

## 7. Limitations, release conditions, and scientific significance

The paper explicitly identifies several limitations of YT-THDC [2602.22039]. First, the corpus is small, at about **30 hours**, and restricted to the **drama** domain [2602.22039]. Second, drama speech introduces **acoustic/domain bias**, including scripted language, music, and dramatic prosody, rather than ordinary spontaneous speech [2602.22039]. Third, the Mandarin subtitles are not exact transcripts of the spoken Taiwanese Hokkien and may include paraphrases, omissions, or semantic simplifications [2602.22039]. Fourth, auxiliary-language texts created by machine translation may introduce **translation noise** and **biases or misalignments** affecting model behavior [2602.22039]. Fifth, the experimental conclusions are confined to **Taiwanese Hokkien**, and transferability to other low-resource languages is not established [2602.22039].

The dataset is explicitly described as **released**, available for **research use only**, and restricted to **non-commercial research and educational purposes** [2602.22039]. The paper does not provide a URL, repository, formal license name, or access procedure in the provided description [2602.22039].

In scientific terms, YT-THDC functions as both a **resource contribution** and an **enabling corpus** [2602.22039]. It is a resource because it supplies **30.30 hours** of Taiwanese Hokkien drama speech paired with **Mandarin subtitles** and **manually verified Taiwanese Hokkien transcriptions** [2602.22039]. It is enabling because it operationalizes a realistic low-resource condition in which speech is abundant, exact target-language transcripts are scarce, and another-language subtitle stream is available for exploitation through translation-guided learning [2602.22039]. This suggests that the corpus is significant less as a generic speech benchmark than as a concrete experimental substrate for studying subtitle-driven multilingual supervision in under-resourced ASR.

Source: https://www.emergentmind.com/topics/yt-thdc