BT-TAPT: Back-Translation Pretraining

• BT-TAPT is a task-adaptive pretraining framework that augments limited in-domain text via neural back-translation to enhance semantic diversity.
• It employs a two-stage process—initial TAPT followed by augmented MLM pretraining with 20 stochastic paraphrases per sentence—improving accuracy and robustness.
• Empirical evaluations on text classification tasks show BT-TAPT mitigates underfitting and achieves consistent gains in low-resource and noisy environments.

Back-Translation-Based Augmentation (BT-TAPT) denotes a task-adaptive pretraining framework for LLMs wherein the task-specific corpus is augmented using neural back-translation. The method first generates stochastic paraphrases of in-domain unlabeled data through round-trip machine translation (English→German→English), then employs these paraphrases for continued masked language modeling (MLM) pretraining, before fine-tuning on the supervised task. BT-TAPT aims to address the underfitting that arises when the distribution of task-relevant data is narrowly represented or limited in quantity, enhancing both in-domain accuracy and robustness to distributional noise (Lee et al., 2021).

1. Motivation and Problem Formulation

The canonical approach to fine-tuning pretrained LMs such as BERT and RoBERTa consists of initializing from pretrained parameters $\theta_{pre}$ (typically learned over a large, generic corpus) and then fine-tuning on a labeled downstream dataset. Task-Adaptive Pretraining (TAPT) augments this by first conducting additional MLM pretraining on a small in-domain, unlabeled corpus $D_{task}$, yielding adaptation to the downstream data distribution. The effectiveness of TAPT is constrained by the typically small size of $D_{task}$, resulting in underfitting by the LLM: it is insufficiently exposed to the relevant lexical and discourse variety of the target domain.

Formally:

• Let $\theta_{pre} \in \mathbb{R}^d$ be the pretrained LM parameters.
• $D_{task} = \{x^{(1)}, ..., x^{(N)}\}$ is the set of unlabeled in-domain sentences.
• The aim is to expand $D_{task}$ into a larger $D_{aug}$ for further MLM pretraining.

BT-TAPT defines the augmentation of $D_{task}$ through stochastic paraphrasing by neural back-translation, increasing coverage of the task domain and mitigating data scarcity effects (Lee et al., 2021).

2. Back-Translation Augmentation Pipeline

BT-TAPT leverages neural back-translation by subjecting each original sentence $x \in D_{task}$ to an English→German→English translation loop:

• The translation model is Facebook WMT’19 “transformer-big” with German as the pivot language.
• Nucleus (top-p) sampling is applied ($p=0.95$) at decoding to ensure diversity across paraphrases.
• For each $D_{task}$0, $D_{task}$1 back-translated variants are generated, so $D_{task}$2.
• Each back-translation $D_{task}$3 is treated as an independent sample from a paraphrase distribution $D_{task}$4, where the stochasticity is induced by both the neural MT model and the sampling method.

Semantically, this procedure augments the input distribution with in-domain paraphrastic diversity, expanding the effective context available to the LLM during re-pretraining (Lee et al., 2021).

3. BT-TAPT Algorithm: Stages and Objective

The BT-TAPT training pipeline is as follows:

1. Initialize from $D_{task}$5.
2. TAPT Stage: $D_{task}$6.
3. Augmentation: For each $D_{task}$7, generate $D_{task}$8 stochastic back-translations and aggregate into $D_{task}$9.
4. BT-TAPT Stage: $D_{task}$0.
5. Fine-tune $D_{task}$1 on $D_{task}$2 for the target classification task.

The MLM pretraining objective is

$D_{task}$3

Where, for BT-TAPT, $D_{task}$4 in step 4 and no non-uniform weighting is introduced.

The TAPT→BT ordering, i.e., TAPT followed by pretraining on back-translated paraphrases, provides the highest mean downstream accuracy and lowest variance relative to alternative orders, such as BT→TAPT or joint mixing (Lee et al., 2021).

4. Training and Hyperparameter Regimen

Key practical details for BT-TAPT include:

• Base models: BERT-base and RoBERTa-base (HuggingFace “transformers” implementation).
• Re-pretraining: For low-resource settings ($D_{task}$5 small), both TAPT and BT-TAPT receive $D_{task}$6K steps; for high-resource, $D_{task}$7K. Batch size is 64, maximum input length 512.
• Optimization: AdamW optimizer (learning rate $D_{task}$8), linear cooldown with $D_{task}$9 warm-up steps.
• Fine-tuning: Hyperparameters (epoch count, batch size, learning rate, weight decay) are grid searched with early stopping based on dev set loss. Each experiment average is over five random seeds.

The critical augmentation hyperparameter is $\theta_{pre} \in \mathbb{R}^d$0, the number of paraphrases per input; ablation finds that accuracy plateaus for $\theta_{pre} \in \mathbb{R}^d$1, justifying this choice (Lee et al., 2021).

5. Empirical Evaluation and Robustness

Experiments employ a variety of standard text classification datasets, with both high- and low-resource configurations:

• Sentiment: IMDB (low: 2.5K/class), MR, SST-2, Amazon (downsampled for low-resource)
• Question Classification: TREC6
• Topic Classification: AGNews (low: 10K/class)

Representative low-resource accuracies (mean $\theta_{pre} \in \mathbb{R}^d$2 std):

• IMDB: BERT-base: 92.2±0.3 (base), 93.0±0.2 (+TAPT), 93.3±0.2 (+BT-TAPT)
• Amazon macro-F$\theta_{pre} \in \mathbb{R}^d$3: 60.8±2.3 (base), 67.0±0.8 (+TAPT), 67.3±0.9 (+BT-TAPT)
• Similar patterns for SST-2, MR, TREC, AGNews.

In high-resource settings, BT-TAPT outperforms TAPT by +0.2 to +3.2 percentage points on IMDB and Amazon.

Noise robustness is assessed through five perturbation types: Synonym (WordNet), BT-beam, BT-top-p, CharSwap, and InvTest. BT-TAPT consistently yields higher accuracy gains over TAPT after noise injection, frequently converting TAPT’s negative robustness into positive robustness (Lee et al., 2021).

6. Ablation Studies and Analysis

Comparisons with alternative augmentation approaches reveal:

• Methods such as EDA, embedding-swap, and TF-IDF swap do not improve or destabilize accuracy, while BT-TAPT adds +0.5 points with low variance in IMDB low-resource setup.
• Ordering effects are pronounced: TAPT→BT order achieves optimal performance.
• Increasing $\theta_{pre} \in \mathbb{R}^d$4 above 20 does not appreciably improve accuracy, implying diminishing returns beyond this margin.

This suggests that the benefit of back-translation stems from enriching the paraphrastic coverage of the in-domain distribution, and that the role of the augmentation is not merely increasing data size, but the semantic diversity induced by stochastic neural paraphrasing (Lee et al., 2021).

7. Significance, Limitations, and Future Directions

BT-TAPT theoretically enhances domain generalization by expanding the overlap between the pretraining domain and the downstream task distribution via paraphrase diversity. This is particularly consequential when $\theta_{pre} \in \mathbb{R}^d$5 is small, mitigating underfitting.

Practical trade-offs include computational cost, dominated by the generation of $\theta_{pre} \in \mathbb{R}^d$6 paraphrases via high-capacity MT models and the necessity of a redundant MLM pretraining round. The method is currently validated for English→German→English, and exclusively on text classification.

The efficacy of back-translation in other tasks (e.g., question answering, summarization), alternate pivot languages, or more efficient augmentation regimens remains untested. Proposed extensions include dynamic weighting schemes, multilingual pipelines, and application to sequence generation objectives (Lee et al., 2021).