--- title: Text-Only Decoder Pre-training url: https://www.emergentmind.com/topics/text-only-decoder-pre-training type: topic --- # Text-Only Decoder Pre-training Text-only decoder pre-training refers to the strategy of pre-training the decoder component of a neural model using exclusively unpaired text data. This paradigm omits paired input modalities (e.g., images for captioning, speech for ASR, real images for scene text recognition, or documents for dense retrieval) in the early stage, focusing on enabling the decoder to effectively model the generation of target text conditioned on representations mimicking the missing modalities. Deployment of this approach is grounded in the observation that large-scale, high-quality unpaired text is easy to source and that aligning decoder initialization to textual distribution enhances downstream modeling, especially for tasks or domains where paired data is scarce or costly to obtain. Multiple recent frameworks leverage text-only decoder pre-training across modalities, with significant methodological and architectural innovation observed in image captioning, speech processing, scene text recognition, and dense passage retrieval. ## 1. Motivations and Conceptual Foundations The availability of extensive unpaired text and the scarcity of paired modality–text data (e.g., images with captions, speech with transcripts) drive the adoption of text-only decoder pre-training. The methodology is rooted in several observations and objectives: - **Statistical Overlap in Embedding Space**: Vision–language models like CLIP demonstrate that text and image embeddings coexist within overlapping but non-identical representational manifolds. This allows for "pseudo-conditioning" of the decoder on text-side surrogates during pre-training, with inference conditioned on the true (e.g., visual or acoustic) side at deployment [2211.00575, 2303.03032, 2408.05706]. - **Enhanced Generalization**: For modalities with scarce paired supervision, such as low-resource ASR or dense retrieval in niche domains, pre-training the decoder on generic language data can provide robust linguistic priors and prevent overfitting to small supervised corpora [1812.03919, 2210.03730, 2309.08876]. - **Reduced Architectural Complexity and Training Cost**: Some approaches use only text and pre-computed embeddings, avoiding large-scale joint optimization or expensive negative sampling as found in many multi-modal setups. This can streamline compute requirements and enable rapid iteration, especially for lightweight decoder designs [2303.03032, 2408.05706]. A plausible implication is that text-only decoder pre-training is particularly beneficial for rapid domain adaptation, transfer to novel styles, and applications with large text corpora but limited paired data. ## 2. Architectural Strategies for Text-Only Decoder Pre-Training Several architectures instantiate text-only decoder pre-training, tuned for either generative (captioning, translation) or discriminative (retrieval) downstream use: - **Frozen Encoder + Trainable Decoder**: In settings such as CLIP-based captioning or scene text recognition, the CLIP text encoder is frozen; its output is treated as a surrogate "visual" embedding to condition a transformer-based decoder—either a lightweight language model (e.g., 4–8 transformer layers) or a large pre-trained LM head (GPT-2) [2211.00575, 2303.03032, 2408.05706]. - **Decoder-Only Language Models for Modal Tasks**: In speech recognition, the decoder-only transformer (no source–target attention) is trained as a conditional LM, augmented with prompts derived from CTC-compressed encoder features or direct text embeddings [2309.08876]. - **Two-Stage Enc–Dec for Unit-to-Text**: In SpeechUT, a unit embedding from a shared encoder is fed into a transformer decoder, trained on large text corpora converted into discrete units; this aligns both speech and text via a unified unit representation [2210.03730]. - **Masked Autoencoding for Encoder-Pretraining**: In dense retrieval, an encoder processes heavily masked inputs, and the (weak) decoder reconstructs masked spans from the encoder and bottleneck. The decoder's reconstruction capacity is modulated (e.g., via importance-aware masking) to demand richer representations from the encoder [2305.13197]. Essentially, these designs leverage either pre-computed or synthetic pseudo-modal representations, processed by the decoder under the same architectural and tokenization regime as in full multimodal training or inference. ## 3. Conditioning, Noise, and Bridging Modality Gaps A major challenge in text-only decoder pre-training for conditional generation or retrieval is the non-identical distribution between (e.g.) image and text embeddings. Multiple strategies bridge this modality gap: - **Noise Injection**: The CapDec system injects Gaussian noise into CLIP text-embeddings during decoder pre-training. The noise scale is set to match the typical offset between paired image and text embeddings under the ℓ∞–norm, forming a robust decoding region—a small ball around each text embedding—so the decoder learns invariance to text–image embedding shifts [2211.00575]. This corrects mismatch at inference, allowing the decoder to generate captions from genuine image embeddings. - **Projection and Memory-Augmented Bridging**: DeCap constructs a large memory bank of text-side embeddings, then creates a projection of the visual embedding by weighted averaging, using cosine similarity as weights, to move the image embedding into the textual manifold before it conditions the decoder [2303.03032]. This is entirely training-free at inference and eliminates the need for additional paired supervision. - **Offline Randomized Perturbation (ORP)**: In scene text recognition, DPTR forms perturbed "pseudo-visual embeddings" by mixing small amounts of real image embedding noise (from the CLIP image encoder) into CLIP text embeddings during pre-training. This forces the decoder to generalize over a small ball of possible textual–visual embeddings, mitigating overfitting to the text manifold and further encouraging alignment [2408.05706]. In the context of dense retrieval via masked autoencoders (CDMAE), the "challenge" is modulated in the masking strategy: masking tokens of higher pointwise mutual information pushes the encoder–decoder system to better preserve and recover salient text features [2305.13197]. ## 4. Objectives, Training Protocols, and Data Training objectives are typically standard cross-entropy losses for autoregressive generation, sometimes combined with CTC losses or special perturbation regimes: - **Prefix LM and Autoencoding**: For captioning and scene text, the pre-training objective is to predict each token given all previous tokens and a fixed or perturbed embedding (from text or noise-injected text), minimized over the corpus: $$ L = -\frac{1}{T} \sum_{i=1}^T \log P(w_i | w_{