Decoder-only Foundation Model

• Decoder-only foundation models are transformer architectures that use only decoder layers with causal autoregressive attention, eliminating the need for a separate encoder.
• They simplify design while delivering robust performance across natural language generation, translation, and multimodal applications through unified token processing.
• Innovations like dynamic layer selection and parallel decoding enhance efficiency and scalability, addressing challenges like attention degeneration and resource constraints.

A decoder-only foundation model refers to a transformer-based model architecture designed to perform a wide range of predictive, generative, or sequence modeling tasks using only a stack of decoder layers, i.e., without a separate encoder module as found in encoder-decoder (ED) architectures. These models leverage a causal autoregressive mechanism—each layer receives as input a sequence of tokens and computes representations such that each token only attends to its own and previous positions (with various adaptations for multimodal or structured data inputs). Recent advances have demonstrated decoder-only models as universal function approximators not only for natural language generation, but also for machine translation, time-series forecasting, speech-to-text, ranking, recommendation, and vision tasks.

1. Architectural Foundations and Comparison with Encoder-Decoder Models

A classical sequence-to-sequence transformer comprises an encoder for the source sequence and a decoder for the target, with two distinct attention pathways—self-attention in each module, and cross-attention from decoder to encoder outputs. By contrast, a decoder-only model ingests the source and (optionally, the previously generated target) tokens in a single sequence, employing masked self-attention throughout. This causal structure simplifies parameter sharing and reduces architectural complexity but brings crucial differences in information flow.

The regularized encoder-decoder (RED) framework (Fu et al., 2023) helps illuminate this relationship. By concatenating source and target tokens and inducing unidirectional cross-attention, RED models mimic the classical decoder-only behavior within an ED structure, enabling precise analysis of limitations such as the attention degeneration problem—where decoder-side attention focus on source tokens degrades as generation proceeds. Theoretical sensitivity analysis quantifies this effect using the Jacobian norm $S_{ij} = \|\partial z_i / \partial x_j\|$:

• Encoder attention: $$\|J_{ij}^{E}\| \leq C_3(1/N + \sqrt{\ln(1/\delta)})$$
• Decoder-only cross attention: $$\|J_{ij}^{C}\| \leq C_3(1/(N + i) + \sqrt{\ln(1/\delta)})$$

As the generation index $i$ grows, sensitivity to the source diminishes in the decoder-only structure. This behavior directly impacts tasks requiring sustained source awareness, such as machine translation and summarization.

2. Extensions to Multimodal and Structured Inputs

Beyond plain LLMing, decoder-only models have been extended through prompt design, preprocessing modules, and novel attention flows to integrate multimodal signals and complex data. Systems such as Speech-LLaMA (Wu et al., 2023), DTrOCR (Fujitake, 2023), and decoder-only ASR with CTC prompts (Tsunoo et al., 2023) demonstrate this versatility.

• Speech-LLaMA uses a Connectionist Temporal Classification (CTC) compressor to reduce the sequence length of speech frames, then a lightweight audio encoder to embed acoustic features in the LLM's semantic space. The processed speech and/or text prompt form a single sequence to be autoregressively decoded using causal attention.
• DTrOCR “patchifies” input images and feeds the patch sequence directly into a decoder-only transformer for OCR. No vision encoder or explicit cross-attention is employed; positional encodings ensure spatial order is preserved.
• ASR with CTC prompts passes audio through a conformer encoder, uses the CTC blank-removal operation to compress, and then maps the resulting frames as prompts to an autoregressive decoder. The decoder is trained both on paired audio–text and on text-only data, enhancing linguistic robustness and efficiency.

These adaptations maintain the fundamental decoder-only property (single stack, masked causal attention) but augment the tokenization, embedding, or input pipeline to support diverse modalities.

3. Efficiency, Scalability, and Dynamic Computation

Decoder-only models are amenable to efficiency optimizations and scalability enhancements. Notable advances include memory reduction, latency optimization, and dynamic inference.

• YOCO (You Only Cache Once) (Sun et al., 8 May 2024): Implements a decoder-decoder split—global key-value caches computed once by a self-decoder, then reused in subsequent cross-decoder layers via cross-attention. This reduces GPU memory requirements for long-context inference by factors proportional to the number of layers.
• Dynamic Layer Selection (Glavas et al., 26 Oct 2024): Investigates per-token or per-sequence adaptive computation. Uniform layer skipping is shown to robustly preserve hidden states and next-token prediction quality compared to early exit. Oracle controllers selecting the minimal set of layers per input can match full-model performance using as little as 23.3% of the computational cost. Skip controllers based on hidden states offer negligible improvement over token-agnostic controllers—indicating that dynamic inference gain is feasible without complex gating logic.
• Parallel Decoding and KV-Cache (Pang et al., 2 Dec 2024): In visual autoregressive models (e.g., RandAR), position instruction tokens and random order training unlock concurrent prediction of multiple image tokens, yielding 2.5× acceleration in generation latency.

These innovations address practical bottlenecks in deployment and training, enabling foundation models to operate with reduced computational and resource overhead while retaining task performance.

4. Specialized Model Designs and Applications

Recent work demonstrates that decoder-only models can serve as foundation architectures for domain-specific applications:

• Biomedical text generation in Italian (Igea) (Buonocore et al., 8 Jul 2024): Utilizes a decoder-only transformer (Minerva/Mistral backbone), continually pretrained on a heterogeneous corpus of Italian medical text. Released in 350M, 1B, and 3B parameter sizes, Igea balances efficiency and specialized terminology coverage. The model demonstrates superior performance on in-domain question answering (MedMCQA-ITA; up to 31.3% accuracy for the 3B variant) while retaining general language understanding relative to non-specialized baselines.
• Personalized ranking and recommendation (360Brew) (Firooz et al., 27 Jan 2025): A 150B parameter, decoder-only model trained/fine-tuned on LinkedIn data, unifies >30 predictive ranking tasks previously handled by bespoke models. All user data, item information, and instruction signals are verbalized as text and processed autoregressively, enabling in-context learning for new tasks and domains and obviating complex feature engineering and dependency graphs.
• Time-series forecasting (Das et al., 2023): Patched-decoder architectures process input sequences in nonoverlapping patches, using causal attention and autoregressive output patches. Zero-shot performance on public datasets is competitive with supervised state-of-the-art models.

Domain adaptation strategies include continual pretraining on specialized corpora and the use of text-based prompts to encode entity attributes, interactions, or domain knowledge.

5. Innovations in Attention Mechanisms and Inference Strategies

The decoder-only paradigm requires careful attention to monotonic alignment, source-target correlation, and order bias—addressed by new inference and training strategies.

• Partial Attention LLM (PALM) (Fu et al., 2023): Introduces a dedicated partial attention component (ATTₗᴾ) responsible for attending exclusively to source features across decoding, combined with separate positional encodings and language embeddings. PALM resolves attention degeneration and maintains source sensitivity for long generations.
• Attention-Constrained Inference (ACI) for TTS (Wang et al., 30 Apr 2024): Identifies alignment-emerged attention maps (AEAMs) in certain decoder heads, exploited during inference via constraining masks. ACI reduces word error rates in synthesized speech by up to 20.5% without altering training, maintaining naturalness and speaker similarity.
• Streaming Self-Attention (SSA) in Simultaneous Translation (Guo et al., 6 Jun 2024): DST (Decoder-only Streaming Transformer) uses SSA to allocate attention policy between source and target prefixes, enabling adaptive “read/write” decisions and high-quality, low-latency translation.

These techniques ensure decoder-only models maintain task-relevant alignment and ordering capabilities without explicit encoder modules.

6. Limitations, Trade-offs, and Open Research Questions

While decoder-only foundation models yield architectural simplicity, parameter sharing, and unification across domains, several limitations persist:

• Attention degeneration: Pure decoder-only causal attention is prone to losing source sensitivity during long generations, resulting in hallucinated or prematurely truncated outputs. Remedies such as PALM and partial attention components are effective but introduce extra design complexity.
• Bidirectional context modeling: Encoder-decoder architectures inherently capture bidirectional dependencies, beneficial in complex translation and summarization. Decoder-only models must rely on careful prompt structure, positional encoding, and attention innovations to approximate these capabilities.
• Feature granularity: Vision and multimodal tasks may suffer from limited fine-grained feature extraction when using token patch embedding without dedicated encoders. Enhanced pre-processing and improved architectural patches are active research topics.
• Scalability: Large-scale models (e.g., 150B parameters) demand substantial compute for training. Techniques such as YOCO, layer skipping, and parallel decoding alleviate but do not wholly solve resource constraints.
• Interpretability and control: Dynamic inference controllers, source-target positional schemes, and prompt engineering present active areas for interpretability, data efficiency, and output controllability.

Further, domain adaptation and low-resource settings require continual research into pretraining strategies, loss functions, and efficient architectural scaling.

7. Future Directions and Implications

Emerging trends suggest that decoder-only models will continue expanding as general-purpose, domain-transferable architectures for foundation models:

• Unified model deployment: 360Brew demonstrates consolidation of dozens of recommendation tasks into a single textual interface model, providing scalable generalization via in-context learning.
• Multimodal and structured data integration: Decoder-only models are increasingly adapted to process images, speech, time series, and structured records directly as token sequences.
• Zero-shot and prompt-based learning: Task definitions and behavioral cues are provided via natural language, reducing the dependence on hand-crafted features and facilitating rapid adaptation to new domains.
• Efficient inference: Advances in caching, layer selection, and concurrent decoding further improve latency and resource requirements, making long-context and streaming use cases practical.
• Foundation for interpretability, robustness, and domain specialization: As shown in biomedical, recommendation, and translation domains, continual adaptation of decoder-only models enables domain-specific excellence with retained generalist capability.

A plausible implication is that decoder-only architectures, when properly adapted with innovations in attention, input processing, and dynamic inference, will provide a flexible framework for the next generation of universal foundation models in AI. However, attention degeneration, interpretability, and structured context modeling remain active challenges for further refinement.