High-Resource Methodological Bias in Low-Resource Investigations (2211.07534v1)

Abstract: The central bottleneck for low-resource NLP is typically regarded to be the quantity of accessible data, overlooking the contribution of data quality. This is particularly seen in the development and evaluation of low-resource systems via down sampling of high-resource language data. In this work we investigate the validity of this approach, and we specifically focus on two well-known NLP tasks for our empirical investigations: POS-tagging and machine translation. We show that down sampling from a high-resource language results in datasets with different properties than the low-resource datasets, impacting the model performance for both POS-tagging and machine translation. Based on these results we conclude that naive down sampling of datasets results in a biased view of how well these systems work in a low-resource scenario.

• The paper reveals that simulating low-resource conditions by downsampling high-resource data produces datasets with properties distinct from genuine low-resource data.
• These differences in data properties lead to biased and unreliable model performance evaluations when tested on genuine low-resource languages.
• The study highlights that this methodological bias can misdirect research and advocates for prioritizing evaluation on genuine low-resource datasets.

The paper investigates the common practice in NLP research of simulating low-resource conditions by downsampling high-resource language datasets. It questions the validity of this approach, arguing that the quantity of data is often prioritized over its quality, leading to potential biases in evaluating systems intended for genuine low-resource scenarios. The central hypothesis is that datasets created via downsampling high-resource data exhibit different characteristics compared to naturally occurring low-resource datasets, which in turn affects model performance and evaluation validity.

Methodology and Experimental Setup

The research empirically examines this hypothesis using two standard NLP tasks: Part-of-Speech (POS) tagging and Machine Translation (MT).

For POS tagging, the investigation utilizes the Universal Dependencies (UD) treebanks. High-resource languages (e.g., English, French, German) are downsampled to various sizes (1k, 5k, 10k tokens) to mimic low-resource settings. These downsampled datasets are then compared against genuine low-resource language datasets from UD (e.g., Bambara, Buryat, Naija) of comparable sizes. The comparison focuses on intrinsic data properties and the performance of a standard BiLSTM-CRF POS tagger trained on these datasets.

For MT, the paper uses datasets from the Workshop on Machine Translation (WMT) and Opus. High-resource language pairs (e.g., English-German, English-French) are downsampled to create simulated low-resource parallel corpora (e.g., 10k, 50k, 100k sentence pairs). These are compared with genuine low-resource pairs (e.g., English-Hausa, English-Zulu) from sources like Opus and the Low Resource MT Benchmark. Comparisons involve analyzing corpus statistics and training Transformer-based NMT models on both types of datasets to evaluate performance differences using BLEU scores.

The core methodology involves:

1. Data Sampling: Implementing naive random downsampling from high-resource corpora to create simulated low-resource datasets.
2. Data Property Analysis: Computing various statistics for both simulated and genuine low-resource datasets. These include:
   • Type-Token Ratio (TTR)
   • Average sentence length
   • POS tag distribution (for POS tagging)
   • Word alignment statistics (for MT, where applicable)
   • Subword vocabulary overlap and frequency distribution (using BPE)
3. Model Training and Evaluation: Training standard models (BiLSTM-CRF for POS, Transformer for MT) on both types of datasets.
4. Performance Comparison: Comparing model performance (accuracy/F1 for POS, BLEU for MT) achieved on genuine low-resource test sets when trained on simulated versus genuine low-resource training data of matched sizes.

Findings on Data Property Divergence

The empirical analysis reveals significant differences in properties between downsampled high-resource datasets and genuine low-resource datasets, even when controlling for size (token count or sentence pair count).

• Lexical Diversity: Genuine low-resource datasets often exhibit higher Type-Token Ratios (TTR) compared to size-matched downsampled high-resource datasets. This suggests that a smaller corpus in a low-resource language might cover a relatively broader vocabulary space proportionally than a similarly sized subset of a high-resource language corpus. Downsampling high-resource data tends to retain the lower TTR characteristic of the parent corpus.
• Sentence Length Distribution: Differences were observed in average sentence lengths and their distributions. Downsampled high-resource data often inherits the sentence length patterns of the original corpus, which may not align with the typical sentence structures found in genuine low-resource language data.
• Label/Tag Distribution (POS): The frequency distribution of POS tags in downsampled datasets often mirrors the distribution in the high-resource source, which can differ substantially from the tag distribution observed in genuine low-resource languages. For instance, specific grammatical constructions or phenomena might be more or less frequent, leading to different tag distributions.
• Subword Characteristics (MT): When applying techniques like BPE, the resulting subword vocabularies and frequency distributions differ. Downsampled data tends to have subword units representative of the high-resource language pair, which may not effectively cover the morphological richness or different linguistic structures present in genuine low-resource pairs.

These divergences indicate that downsampled data does not accurately replicate the intrinsic characteristics of naturally occurring low-resource data.

Impact on Model Performance and Evaluation

The observed differences in data properties translate directly into discrepancies in model performance, challenging the reliability of evaluations based on downsampled data.

• POS Tagging: Models trained on downsampled high-resource data often perform differently (sometimes better, sometimes worse) than models trained on genuine low-resource data of the same size when evaluated on a genuine low-resource test set. Specifically, the BiLSTM-CRF tagger showed sensitivity to the differing tag distributions and lexical properties. Using downsampled data could lead to overly optimistic or pessimistic conclusions about a model's suitability for actual low-resource deployment, depending on the specific language pair and evaluation setup. The paper reported significant performance gaps, highlighting that models optimized on downsampled data might not generalize well to the target low-resource environment.
• Machine Translation: Similar effects were observed for NMT. Transformer models trained on downsampled high-resource parallel corpora yielded BLEU scores that did not consistently correlate with performance achieved when training on genuine low-resource corpora of equivalent size. The differences in vocabulary coverage, sentence complexity, and domain relevance between simulated and real low-resource data impacted translation quality. For instance, a model might perform well translating downsampled news articles but struggle with the different domains or linguistic phenomena prevalent in a genuine low-resource dataset (e.g., religious texts, conversational data). The results indicated that performance metrics obtained via downsampling do not reliably predict performance in authentic low-resource scenarios.

The core finding is that the evaluation methodology itself (using downsampled data) introduces a bias. Performance gains or model rankings observed in simulated low-resource settings may not hold true when applied to genuine low-resource languages and datasets.

Conclusion and Implications

The paper concludes that naive downsampling of high-resource datasets is an inadequate proxy for genuine low-resource conditions in NLP. The resulting datasets possess distinct statistical properties compared to true low-resource data, leading to potentially misleading model evaluations and biased conclusions about system performance. This high-resource methodological bias can misdirect research efforts, potentially favoring models or techniques that perform well on the cleaner, more homogenous downsampled data but fail in real-world low-resource settings characterized by greater linguistic diversity, different data domains, and unique quality challenges. Researchers should exercise caution when interpreting results obtained from downsampled datasets and prioritize evaluation on genuine low-resource data whenever possible to ensure the ecological validity of their findings and the practical applicability of their proposed methods. The paper advocates for a greater focus on curating and utilizing authentic low-resource datasets and developing evaluation methodologies that better reflect the complexities of low-resource scenarios.