TEEMI Corpus: ASR Test Set
- TEEMI Corpus is a specialized test set featuring noisy L2 business presentation audio paired with company-related texts for ASR evaluation.
- The corpus comprises 39 short recordings from 61 speakers, including manual transcriptions and additional slide decks and web pages for domain adaptation experiments.
- Baseline evaluations using systems like JRTk, Google, and Kaldi reveal significant challenges such as non-native accents, live noisy conditions, and recognition of firm-specific entities.
Searching arXiv for the specified paper to ground the article in the source record. The resource referred to here as the TEEMI Corpus is a small, purpose-built test corpus for automatic speech recognition (ASR) that combines audio recordings and transcriptions of student business presentations with associated slide decks and company web pages. It was introduced by Dominik Macháček, Jonáš Kratochvíl, Tereza Vojtěchová, and Ondřej Bojar in "A Speech Test Set of Practice Business Presentations with Additional Relevant Texts" (Macháček et al., 2019). In the source text, the corpus is not explicitly named the TEEMI Corpus, and no expansion of the acronym is given. Its defining characteristic is the pairing of noisy English L2 speech from European high school students with firm-specific additional texts, enabling evaluation of both blind ASR and ASR with domain adaptation.
1. Origin and intended evaluation setting
The corpus was created to evaluate ASR systems on challenging, realistic speech from European high school students presenting "mock companies" at a trade fair. English functions as a lingua franca and is a second language for the speakers. The recordings therefore combine diverse L1 accents with domain-specific vocabulary and named entities, including entities from non-English locales pronounced in L2-accented ways.
A central feature of the evaluation scenario is the prior availability of in-domain text. Each presentation is linked to the corresponding company’s slide deck and web pages, even though these materials were not shown during the recorded "elevator pitch" presentations. This supports experiments in which ASR is evaluated both without adaptation and with adaptation based on firm-specific texts. The intended use includes expanding lexicons, handling out-of-vocabulary items, injecting named entities, and adapting LLMs using the available slides and web pages.
This design targets a practically important ASR setting: events such as conferences and trade fairs often provide preparatory materials before the talks. The corpus operationalizes that condition in a controlled test-only resource rather than a full train/dev/test benchmark.
2. Corpus composition and domain profile
The test set comprises 39 English presentations, each capped at 90 seconds, totaling 58 minutes 28 seconds of audio. It is split between 17 single-speaker recordings and 22 two-speaker recordings, with 61 distinct speakers overall. Students are European high schoolers using English as a second language.
| Item | Count |
|---|---|
| Presentations | 39 |
| Total audio | 58 minutes 28 seconds |
| Single-speaker recordings | 17 |
| Two-speaker recordings | 22 |
| Distinct speakers | 61 |
| Transcript words | 6,613 |
The speaker L1 distribution is specified separately for the single-speaker and two-speaker subsets. Among single-speaker recordings, L1s include cs (9), es (1), ro (3), sk (1), hu (1). Among two-speaker recordings, L1s include cs (18), de (4), it (16), es (2), nl (3), fi (1), totaling 44 speakers. The firms originate from Czech Republic (18), Italy (8), Romania (4), Slovakia (3), Austria (2), Spain (2), and Belgium (2).
Transcripts contain 6,613 words, with 2,891 words in single-speaker recordings and 3,722 words in two-speaker recordings. Presentation topics span travel agencies (7), food and beverage producers (4), beauty and health (3), clothes and shoes (3), household equipment (3), online promotion (2), accessories (2), logistics (2), and a broader "others" category (13). The domain is intentionally narrow and business-oriented, with localized named entities, product names, locations, and contact details relevant to each mock company.
A common misconception would be to treat the resource as a general-purpose speech corpus. Its scope is much narrower: short business presentations by European L2 English speakers in a live-fair environment.
3. Modalities, transcription policy, and metadata
For each recorded presentation, the corpus provides one audio recording, manual transcriptions, and additional texts associated with the firm. The transcriptions were initially produced by the student presenters themselves, then reviewed and lightly edited by the authors. The additional texts consist of slide decks used at the trade fair and company web pages, where available.
The text materials are distributed in multiple representations. The corpus includes original slide files in Microsoft Office formats and original web content in HTML and images, as well as XLIFF generated via MateCat Filters and derived plaintext extracted from XLIFF for ease of use. Language tags are provided on all documents. The link between audio and texts is at the company level rather than at sentence level or through time alignment.
To comply with GDPR, personal data such as student names and photos are removed from recordings, transcripts, and texts. Company-related facts, including email addresses, phone numbers, websites, and locations, are preserved. The paper provides firm counts and language tags but does not enumerate detailed speaker metadata fields such as IDs or proficiency ratings.
The recordings were made during a live fair with headset microphones to improve signal quality, but the environment remains acoustically difficult. Background noise includes live music, stage announcements, crowd conversations in multiple languages, and ambient activity such as people entering and exiting. The paper does not specify audio format details such as sampling rate, bit depth, channels, or file formats.
Transcription conventions are deliberately pragmatic. The authors corrected missing parts, normalized punctuation, and fixed obvious misspellings, while preserving authentic learner grammar and vocabulary. Examples explicitly mentioned include "massageses" and "botel." For evaluation, casing and punctuation are disregarded. Non-speech events, fillers, and pronunciation variants are not explicitly cataloged.
4. Additional texts and adaptation workflows
Additional texts were collected for 36 of the 39 firms: 20 firms provide both slides and a website, 12 provide slides only, 4 provide a website only, and 3 provide neither. These materials amount to roughly 97,000 words with a vocabulary of about 15,000 word types.
| Additional text availability | Firms |
|---|---|
| Both slides and website | 20 |
| Slides only | 12 |
| Website only | 4 |
| Neither | 3 |
The slide sets total 32 and are in Czech (14), English (15), Slovak (1), and some mixed-language pairs such as cs/en and sk/en. The websites total 23 and are mostly in national languages, with counts including Czech (14), English (2), German (2), Italian (2), Spanish (1), Romanian (1), Dutch (3 combined via firms), Finnish (1 via firms), plus one multilingual site (it/en/es/de). Although these materials were not used during the recorded "elevator pitch," they are suitable for building company-specific vocabularies and named-entity lists for ASR adaptation (Macháček et al., 2019).
The paper describes several practical procedures for exploiting these texts. Text preprocessing may convert originals to XLIFF via MateCat Filters and then to plaintext, normalize whitespace, language-tag documents, and optionally perform sentence segmentation. Vocabulary and named-entity harvesting may extract product names, company names, local place names, email domains, phone numbers, URLs, and domain-specific terms to build a per-firm lexicon and phrase list. Where websites are non-English, named entities and numeric strings can be used directly, and machine translation may be considered if English LM adaptation is desired.
For hybrid systems such as Kaldi, lexicon augmentation may add harvested words to the pronunciation lexicon and generate accent-aware pronunciations using G2P models and manual rules reflecting L2 pronunciations. For cloud systems that support phrase hints or custom class boosting, such as Google, the same terms and entities can be injected to bias decoding. LM adaptation may train or fine-tune a small n-gram LM or neural LM on per-firm text and then rescore N-best hypotheses or lattices to promote in-domain sequences. OOV handling should ensure that added words and entities are recognized as tokens rather than split or ignored, with numeric normalization and URL or email tokenization made consistent with transcript style.
These procedures are methodological guidance rather than reported improvement results. No perplexity or LM-adaptation metrics are presented.
5. Baseline ASR systems and reported performance
Three baseline ASR systems were evaluated: Janus Recognition Toolkit (JRTk) classic ASR; Google Cloud Speech-to-Text (English, United States); and a Kaldi-based hybrid system trained on the Multi-Genre Broadcast (MGB) Challenge data, described as roughly 1,600 hours of BBC TV broadcast audio and hundreds of millions of subtitle words for its LM, hence optimized for native British English. Microsoft’s cloud ASR was also tried and failed on all recordings.
WER is computed conventionally as
where denotes substitutions, deletions, insertions, and the number of words in the human reference transcript. Casing and punctuation are ignored. The authors report two evaluation scenarios: all recordings, and the subset for which all systems produced non-empty output.
| System | Recognized-by-all mean WER | All 39 recordings mean WER |
|---|---|---|
| 73.59% | 89.32% | |
| Kaldi BBC | 87.55% | 87.47% |
| JRTk | 45.21% | 45.63% |
On the recognized-by-all subset, Google has min 20.90, max 98.31, median 87.50, stddev 27.87; Kaldi BBC has min 83.96, max 91.03, median 87.59, stddev 2.29; and JRTk has min 25.00, max 74.08, median 43.41, stddev 15.28. On all 39 recordings, including empty outputs, Google has min 20.90, max 100.00, median 100.00, stddev 21.82; Kaldi BBC has min 83.96, max 91.03, median 87.04, stddev 1.92; and JRTk has min 25.00, max 99.58, median 46.31, stddev 15.23 (Macháček et al., 2019).
The reported error rates are attributed chiefly to strong L2 accents, suboptimal microphone placement, unclear articulation, and the live, noisy environment. Named entities and domain terms are highlighted as a common source of misrecognition because of accent-driven pronunciations and lack of coverage in out-of-domain training data. The results therefore document the difficulty of the task rather than demonstrating successful adaptation.
6. Positioning, access, limitations, and research use
Relative to corpora such as TED-LIUM and LibriSpeech, which are dominated by L1 English speech in controlled or clean conditions, this resource is small, at about 1 hour, but specifically focused on L2 English, real-world noise, and business-presentation content. Common Voice offers speaker diversity, but typically lacks firm-specific associated texts per recording. The distinctive contribution here is the pairing of each noisy L2 recording with additional, curated, in-domain materials intended to drive lexicon and LM adaptation. This suggests a niche role as a stress-test for adaptation-sensitive ASR rather than a broad-coverage benchmark.
The corpus is publicly released at the persistent identifier http://hdl.handle.net/11234/1-3023. License terms and any usage restrictions are provided on the landing page; the paper itself does not state the license. The released materials include audio recordings, transcripts, and additional texts in original formats, XLIFF, and plaintext, all with language tags. Citation of the resource follows the paper title and author list.
Its limitations are explicit. It is intentionally small and test-only, with 39 short recordings and 61 speakers limited to European high school students. L1 coverage, accents, and noise conditions are varied but not exhaustive. Recording quality varies due to live conditions. Transcripts preserve learner grammar rather than enforcing a normalized style beyond punctuation and obvious misspelling fixes. Alignment between audio and additional texts is at the firm level, not sentence-level or time-aligned. Some company websites are non-English, which makes direct English LM training less straightforward.
Two misunderstandings are particularly important to avoid. First, the resource is not a training corpus: no training or development split is included. Second, the presence of slides and web pages does not mean that those materials were part of the spoken input during the recordings; they are accompanying documents intended for adaptation experiments. The paper does not outline future expansions, but natural enhancements would include more speakers across additional L1s, cleaner and more detailed metadata, richer annotation of disfluencies and named entities, and clearer protocols and baselines for LM or lexicon adaptation.