Text-Only Decoder Pre-training

• Text-only decoder pre-training is a strategy that pre-trains decoders exclusively on unpaired text data to improve downstream tasks.
• It reduces architectural complexity and training cost by substituting paired data with large, readily available text corpora.
• Empirical results demonstrate enhanced performance in image captioning, speech recognition, and dense retrieval via noise injection and pseudo-conditioning.

Text-only decoder pre-training refers to the strategy of pre-training the decoder component of a neural model using exclusively unpaired text data. This paradigm omits paired input modalities (e.g., images for captioning, speech for ASR, real images for scene text recognition, or documents for dense retrieval) in the early stage, focusing on enabling the decoder to effectively model the generation of target text conditioned on representations mimicking the missing modalities. Deployment of this approach is grounded in the observation that large-scale, high-quality unpaired text is easy to source and that aligning decoder initialization to textual distribution enhances downstream modeling, especially for tasks or domains where paired data is scarce or costly to obtain. Multiple recent frameworks leverage text-only decoder pre-training across modalities, with significant methodological and architectural innovation observed in image captioning, speech processing, scene text recognition, and dense passage retrieval.

1. Motivations and Conceptual Foundations

The availability of extensive unpaired text and the scarcity of paired modality–text data (e.g., images with captions, speech with transcripts) drive the adoption of text-only decoder pre-training. The methodology is rooted in several observations and objectives:

• Statistical Overlap in Embedding Space: Vision–LLMs like CLIP demonstrate that text and image embeddings coexist within overlapping but non-identical representational manifolds. This allows for "pseudo-conditioning" of the decoder on text-side surrogates during pre-training, with inference conditioned on the true (e.g., visual or acoustic) side at deployment (Nukrai et al., 2022, Li et al., 2023, Zhao et al., 2024).
• Enhanced Generalization: For modalities with scarce paired supervision, such as low-resource ASR or dense retrieval in niche domains, pre-training the decoder on generic language data can provide robust linguistic priors and prevent overfitting to small supervised corpora (Wiesner et al., 2018, Zhang et al., 2022, Tsunoo et al., 2023).
• Reduced Architectural Complexity and Training Cost: Some approaches use only text and pre-computed embeddings, avoiding large-scale joint optimization or expensive negative sampling as found in many multi-modal setups. This can streamline compute requirements and enable rapid iteration, especially for lightweight decoder designs (Li et al., 2023, Zhao et al., 2024).

A plausible implication is that text-only decoder pre-training is particularly beneficial for rapid domain adaptation, transfer to novel styles, and applications with large text corpora but limited paired data.

2. Architectural Strategies for Text-Only Decoder Pre-Training

Several architectures instantiate text-only decoder pre-training, tuned for either generative (captioning, translation) or discriminative (retrieval) downstream use:

• Frozen Encoder + Trainable Decoder: In settings such as CLIP-based captioning or scene text recognition, the CLIP text encoder is frozen; its output is treated as a surrogate "visual" embedding to condition a transformer-based decoder—either a lightweight LLM (e.g., 4–8 transformer layers) or a large pre-trained LM head (GPT-2) (Nukrai et al., 2022, Li et al., 2023, Zhao et al., 2024).
• Decoder-Only LLMs for Modal Tasks: In speech recognition, the decoder-only transformer (no source–target attention) is trained as a conditional LM, augmented with prompts derived from CTC-compressed encoder features or direct text embeddings (Tsunoo et al., 2023).
• Two-Stage Enc–Dec for Unit-to-Text: In SpeechUT, a unit embedding from a shared encoder is fed into a transformer decoder, trained on large text corpora converted into discrete units; this aligns both speech and text via a unified unit representation (Zhang et al., 2022).
• Masked Autoencoding for Encoder-Pretraining: In dense retrieval, an encoder processes heavily masked inputs, and the (weak) decoder reconstructs masked spans from the encoder and bottleneck. The decoder's reconstruction capacity is modulated (e.g., via importance-aware masking) to demand richer representations from the encoder (Li et al., 2023).

Essentially, these designs leverage either pre-computed or synthetic pseudo-modal representations, processed by the decoder under the same architectural and tokenization regime as in full multimodal training or inference.

3. Conditioning, Noise, and Bridging Modality Gaps

A major challenge in text-only decoder pre-training for conditional generation or retrieval is the non-identical distribution between (e.g.) image and text embeddings. Multiple strategies bridge this modality gap:

• Noise Injection: The CapDec system injects Gaussian noise into CLIP text-embeddings during decoder pre-training. The noise scale is set to match the typical offset between paired image and text embeddings under the ℓ∞–norm, forming a robust decoding region—a small ball around each text embedding—so the decoder learns invariance to text–image embedding shifts (Nukrai et al., 2022). This corrects mismatch at inference, allowing the decoder to generate captions from genuine image embeddings.
• Projection and Memory-Augmented Bridging: DeCap constructs a large memory bank of text-side embeddings, then creates a projection of the visual embedding by weighted averaging, using cosine similarity as weights, to move the image embedding into the textual manifold before it conditions the decoder (Li et al., 2023). This is entirely training-free at inference and eliminates the need for additional paired supervision.
• Offline Randomized Perturbation (ORP): In scene text recognition, DPTR forms perturbed "pseudo-visual embeddings" by mixing small amounts of real image embedding noise (from the CLIP image encoder) into CLIP text embeddings during pre-training. This forces the decoder to generalize over a small ball of possible textual–visual embeddings, mitigating overfitting to the text manifold and further encouraging alignment (Zhao et al., 2024).

In the context of dense retrieval via masked autoencoders (CDMAE), the "challenge" is modulated in the masking strategy: masking tokens of higher pointwise mutual information pushes the encoder–decoder system to better preserve and recover salient text features (Li et al., 2023).

4. Objectives, Training Protocols, and Data

Training objectives are typically standard cross-entropy losses for autoregressive generation, sometimes combined with CTC losses or special perturbation regimes:

• Prefix LM and Autoencoding: For captioning and scene text, the pre-training objective is to predict each token given all previous tokens and a fixed or perturbed embedding (from text or noise-injected text), minimized over the corpus:

$$L = -\frac{1}{T} \sum_{i=1}^T \log P(w_i | w_{<i}, \text{embedding})$$

(Nukrai et al., 2022, Li et al., 2023, Zhao et al., 2024).

• Multitask Losses: For decoder-only ASR, multitask batches combine paired ASR tasks (CTC and attention loss) and pure text-LM tasks, the latter via both ordinary and pseudo-prompt language modeling (Tsunoo et al., 2023). Text-only batches use either plain LM loss or feed gold embeddings as prompts to mimic CTC conditioning.
• Unit-to-Text and Joint Multi-Tasking: In SpeechUT, unit sequences (from T2U generators) serve as inputs, with autoregressive cross-entropy and auxiliary CTC losses on the decoder output. The U2T loss is integrated with S2U and masked-unit-modeling objectives in a single multitask schedule (Zhang et al., 2022).

Training is performed on large text corpora—Wikipedia, BookCorpus, filtered web corpora, or language modeling datasets—as well as custom collections (e.g., COCO captions, CC3M, synthetic labels). For models requiring synthetic input conditioning, components such as T2U models, G2P, or precomputed embedding banks are constructed in advance. Hyperparameters are detailed per model, with batch sizes ranging from hundreds to thousands, optimizers (Adam/AdamW), and custom learning rate schedules.

5. Integration with Downstream Multimodal Pipelines

Post pre-training, decoders are integrated into either multi-modal (e.g., image captioning, ASR) or monomodal (dense retrieval) pipelines:

• Zero-Shot and Style-Guided Captioning: The CapDec and DeCap frameworks demonstrate that a text-only pre-trained decoder can generate meaningful captions for images at zero-shot, matching or outperforming prior captioning systems that rely on paired image-caption data. CapDec also supports style transfer by fine-tuning the decoder on unpaired stylistic corpora, with guided caption generation at inference (Nukrai et al., 2022, Li et al., 2023).
• Scene Text Recognition: DPTR, after decoder-only pre-training, plugs the decoder into an image encoder + Feature Merge Unit (FMU) for STR. Fine-tuning proceeds as in standard pipelines. Ablations confirm near-SOTA performance, with clear improvements over synthetic-image pretraining, particularly on real image domains (Zhao et al., 2024).
• Speech Recognition: Decoder-only ASR models mix CTC-based pseudo-prompting and text LM objectives, achieving word error rates below strong encoder–decoder baselines, especially with text augmentation (Tsunoo et al., 2023). For low-resource setups (MMDA, PSDA), text-only decoder pre-training reduces character error rates by 20–30% relative, even in extremely data-scarce regimes (Wiesner et al., 2018).
• Dense Retrieval: In CDMAE, the (challenging) decoder is discarded after pre-training; the encoder+projection constitutes the retriever. Importance-aware masking injects more difficulty into decoding, translating to improved encoder representations and state-of-the-art BEIR zero-shot retrieval (Li et al., 2023).

A summary table of representative frameworks and their modalities/conditioning follows.

Paper (arXiv)	Modality / Task	Decoder Conditioning	Main Contribution
(Nukrai et al., 2022)	Image Captioning	CLIP text embedding + noise	Robust mapping for zero-shot/transfer
(Li et al., 2023)	Image Captioning	CLIP text emb. as prefix	Training-free memory bridging
(Zhao et al., 2024)	Scene Text Recognition	CLIP text emb. + image noise	ORP/FMU bridging real/synthetic gap
(Tsunoo et al., 2023)	Speech Recognition	CTC-compressed / text prompt	Multitask decoder-only LM
(Wiesner et al., 2018)	Low-resource ASR	Synthetic (G2P) features	Text-only pretrain + joint finetune
(Zhang et al., 2022)	Speech/ST	T2U/synthetic unit sequences	Unit-to-text transformer U2T
(Li et al., 2023)	Dense Passage Retrieval	Bottleneck+masked passage	PMI masking for stronger retrieval

6. Empirical Results and Performance Impact

A range of empirical benchmarks and ablations confirm the efficacy of text-only decoder pre-training:

• Zero-Shot Generation:
  • CapDec achieves MSCOCO (zero-shot, text-only train) CIDEr=91.8, with B@1=69.2, outperforming prior text-only methods (MAGIC: CIDEr=49.3) (Nukrai et al., 2022).
  • DeCap achieves MSCOCO CIDEr=42.1–50.6, a 27–36 point gain over ZeroCap; on NoCaps, reaches CIDEr=42.7 (Li et al., 2023).
  • DPTR improves average real STR benchmark accuracy (e.g., IIIT5k, SVT, IC13, etc.) to 96.4% (vs. base 95.5%, synthetic-image pretrain 95.8%), with +0.5% from ORP. On highly challenging real image datasets (ArT, COCO, Uber), DPTR maintains gains of 1–5% over non-pretrained baselines (Zhao et al., 2024).
• ASR and Speech Tasks:
  • Decoder-only ASR with text augmentation yields word error rate improvements of up to 3.1% absolute (clean/other), and further with external LM fusion, it outperforms encoder–decoder baselines at half the compute cost (Tsunoo et al., 2023).
  • For low-resource ASR, PSDA+Pretrain reduces character error rate by 20–26% over non-pretrained setups; more pronounced at extremely low hours of real speech (Wiesner et al., 2018).
  • SpeechUT’s U2T decoder pre-training achieves an additional 0.3–0.5 absolute WER improvement in LibriSpeech fine-tuning (Zhang et al., 2022).
• Dense Retrieval:
  • CDMAE outperforms RetroMAE and Contriever (nDCG@10=49.0 on BEIR) and sets new benchmarks on 9 out of 14 datasets (Li et al., 2023).

The consistent outcome is that text-only decoder pre-training, via projections, noise, or pseudo-conditioning, yields non-trivial improvements—often state-of-the-art or strong competitive performance—across modal, language, and domain boundaries.

7. Implications and Extensions

The methodology described above underscores the potential for text-only decoder pre-training to:

• Enable rapid scaling to new modalities or tasks with scarce paired data, by leveraging vast unpaired textual corpora.
• Reduce data collection costs and architectural complexity while maintaining, or even surpassing, the downstream performance of conventional paired-data approaches.
• Offer robust invariance to input domain shifts (e.g., synthetic-to-real gap in STR, cross-domain ASR).
• Suggest that more general projections, perturbations, and memory-augmented modules could further enhance transfer and alignment in future multi-modal, multilingual, and cross-domain neural architectures.

A plausible implication is that as embedding alignment and decoder generalization continue to improve—through better noise modeling, more informative pseudo-conditioning, or advanced memory-based bridging—text-only decoder pre-training will become standard for a variety of conditional generation and retrieval applications.

---

References: (Nukrai et al., 2022, Li et al., 2023, Zhao et al., 2024, Tsunoo et al., 2023, Wiesner et al., 2018, Zhang et al., 2022, Li et al., 2023).