Full-bandwidth transformer

Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal access to the past, but the vertical feedback channel between decoding steps remains narrow: only the sampled token returns to the bottom of the stack, while the top-layer hidden state is discarded. We introduce the \emph{full-bandwidth transformer}, which widens this channel with \emph{latent feedback}: at each decoding step, the previous top-layer hidden state is fused with the sampled token embedding through a gated linear unit and fed back as the next input. Latent feedback lets non-verbalized computation re-enter the stack with a renewed depth budget, while preserving the standard transformer architecture, KV cache, and language-modeling objective. To train full-bandwidth transformers without losing parallel teacher forcing, we use a scheduled multi-pass objective that introduces latent feedback late in pretraining and mixes a small fraction of deeper feedback passes for stability. We train 1B-parameter full-bandwidth transformers up to 400B tokens and find that latent feedback improves validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance. With negligible per-token decoding overhead, full-bandwidth transformers match or approach standard transformers trained with roughly $1.5\times$ more tokens, and manage to produce shorter reasoning traces at equal or better accuracy.

• The paper introduces latent feedback decoding, which feeds the previous top-layer hidden state into the next token input through a gated fusion, giving shallow layers access to fully processed context without changing the Transformer stack or KV-cache structure.
• The paper uses multi-pass parallel training and finds that a schedule with 3% three-pass batches stabilizes long-horizon recurrence, while 200B-token models can approach or exceed standard models trained on two to five times more data.
• The paper reports gains in mathematical reasoning and code generation, including Math500 improvements from 0.27 to 0.37 and instruction-tuned GSM8K gains from 64.5% to 67.9%, with less than 1% added per-token decoding overhead.

Full-bandwidth transformer

Central thesis

“Full-bandwidth transformer” (2608.08888) addresses a structural asymmetry in decoder-only Transformers. Dense causal attention provides high-bandwidth horizontal communication across positions, but inter-step vertical communication is severely constrained: after producing a token, the model retains the token embedding and layer-indexed states in the KV cache, while the final-layer hidden state is not reintroduced into the bottom of the network. Consequently, computation performed at deep layers remains “depth-frozen”; it can be accessed only by subsequent layers at or above the corresponding depth.

The paper proposes latent feedback decoding, in which the previous top-layer hidden state is fused with the next token embedding and supplied as the next input to the Transformer. This widens the recurrent channel from a discrete token to a full residual-stream vector while preserving the ordinary language-modeling interface, KV-cache structure, and autoregressive output space. The claimed benefit is computational rather than informational: the hidden state is already determined by the preceding context, but it becomes available to shallow layers with a renewed depth budget.

The authors train 1B-parameter models for up to 400B tokens and report improvements in validation loss, few-shot and zero-shot evaluation, mathematical reasoning, code generation, and instruction-tuned performance. The strongest claim is that latent-feedback models trained on 200B tokens can approach or exceed standard Transformers trained on two to five times as many tokens, while adding less than 1% per-token decoding overhead.

Architectural mechanism

In standard decoding, the next input is the embedding of the sampled token. In the proposed architecture, the token embedding acts as a gate over a transformed copy of the previous top-layer state. The fusion is a gated linear unit of the form

$u_t = W_u h_{t-1}^L \odot \sigma(W_g e_t),$

where $h_{t-1}^L$ is the preceding top-layer representation and $e_t$ is the current token embedding. The asymmetric design is important. An additive fusion such as $e_t + W h_{t-1}^L$ would permit the model to suppress the latent pathway and recover ordinary token-only decoding. By making the hidden state the value pathway and the token the gate, the model is forced to process a state-dependent input.

The intervention is external to the Transformer stack. It does not add recurrent attention modules, alter the KV-cache representation, or introduce layer-specific state projections. Only two $D \times D$ projections are added. During generation, the top-layer state is already computed, so the additional cost is limited to the fusion operation. The architecture is therefore compatible with standard serving systems and can be integrated into the decoding loop without changing the asymptotic token-generation complexity.

Figure 1: Standard decoding exposes only the sampled token to the next input, whereas latent feedback reintroduces the previous top-layer state through a dimension-preserving gated fusion.

The paper formalizes the distinction through layerwise reachability. In a standard Transformer, a state at position $t$ and layer $\ell$ can access earlier positions only through representations from shallower layers. A top-layer representation at an earlier position cannot be routed back to layer zero of a later position. Latent feedback removes this restriction for the recurrent state: every new position receives a summary that has already traversed the full stack. Thus, the mechanism does not increase the number of Transformer blocks executed per token, but changes which previously computed representations are available to each layer.

This distinction also separates the method from RNNs and state-space models. The latent state is recurrently transmitted, but it is not a mutable fixed-size memory that overwrites the past. Earlier fused inputs remain in the KV cache and remain available through attention. The method combines explicit long-context storage with a recurrent pathway for the latest fully processed state.

Parallel training through multi-pass objectives

Directly training the exact latent-feedback recurrence would destroy parallel teacher forcing because the input at position $t$ depends on the completed computation at position $t-1$. The paper instead uses temporal parallelism. A first forward pass processes the sequence normally. A second pass shifts the first-pass top-layer states one position to the right, fuses them with token embeddings, and processes all positions in parallel. Additional passes repeat the procedure using the preceding pass’s states.

With $k$ passes, the model trains a feedback horizon of approximately $h_{t-1}^L$0 positions while paying sequential cost across passes rather than across sequence positions. The next-token-prediction loss is applied to every pass, and gradients are allowed to propagate through the latent states between passes. This turns later-pass losses into an auxiliary objective that encourages top-layer states to remain useful as future inputs, rather than merely serving as predictors for the immediately following token.

The training schedule is progressive. Most pretraining uses the ordinary single-pass objective; latent feedback is introduced later, initially through two-pass batches, followed by a small proportion of deeper feedback batches. A prefix-mixin procedure further reduces the distribution shift between training and inference by allowing an arbitrary plain-token prefix followed by fused positions.

A particularly notable empirical result concerns long-horizon stability. Models trained only with single- and two-pass batches become unstable when the recurrence is iterated beyond the trained horizon: validation loss rises and successive hidden-state updates oscillate. Adding only 3% three-pass batches—with a mixture of 75% single-pass, 22% two-pass, and 3% three-pass batches—produces stable behavior over substantially longer rollouts. The authors interpret this as evidence that the learned feedback map approximates a contraction toward a stable fixed point.

Figure 2: A small three-pass training fraction changes long-horizon feedback from oscillatory divergence to stable convergence of the hidden-state iterates.

The stabilization recipe also includes depth scaling, RMS normalization of fused inputs, tied input and output embeddings, and uniform jitter noise applied to the carried state. These choices address the central difficulty of recurrently reusing a representation whose distribution may drift under repeated self-composition. The reported extrapolation extends to 30 and, in an appendix experiment, 1,000 feedback passes, although these results should be interpreted as empirical stability diagnostics rather than a formal convergence guarantee.

Prefill-time computation and data efficiency

Latent feedback can be used during prompt prefilling as well as token generation. The prompt is processed once normally and then optionally reprocessed using fused states from the preceding pass. This adds parallel computation over the prompt without increasing autoregressive generation cost.

The gains are front-loaded: the first fused prefill pass produces most of the improvement, while subsequent passes provide diminishing returns. The authors report that a 100B-token full-bandwidth model with two feedback passes reaches the performance of a 200B-token standard baseline, while a 200B-token model reaches the performance of a 400B-token baseline on validation loss and average 5-shot LM evaluation. Under this accounting, one or two additional prefill passes produce approximately a twofold improvement in effective pretraining data efficiency.

Figure 3: Additional fused prefill passes improve validation perplexity and aggregate LM evaluation, with most of the gain appearing after the first recurrence step.

An important aspect of this result is that the model retains useful performance in standard mode. When evaluated without latent feedback, the recurrence-trained model incurs only a small validation-loss penalty relative to a conventional baseline and can still improve average LM evaluation accuracy. This indicates that the multi-pass objective is not merely optimizing a specialized decoding procedure; it also supplies additional supervision to intermediate representations.

The mechanism resembles test-time depth expansion, but with a different cost profile from looped Transformers. A looped Transformer explicitly reexecutes the stack and therefore increases inference cost with each recurrent iteration. Full-bandwidth feedback reuses the previous token’s already-computed top-layer state and adds only the fusion projections during generation. Extra full-stack computation is optional and concentrated in prefilling.

Generative evaluation

The paper distinguishes three inference regimes:

1. Standard: ordinary prefill and token-only generation.
2. Soft: ordinary prefill followed by latent-feedback generation.
3. Fused: one additional fused prefill pass followed by latent-feedback generation.

Soft decoding isolates the effect of recurrent hidden-state transmission during generation, while Fused decoding additionally refines the prompt representation before the first generated token.

Figure 4: Standard, Soft, and Fused decoding differ in whether latent feedback is used during generation, prefilling, or both.

On base models, Soft decoding improves over Standard decoding on every reported task at both evaluated training scales. The effect is task-dependent. On Math500, the 200B-token model improves from 0.27 to 0.37 under Soft decoding and exceeds the reported 1T-token standard baseline. On coding tasks, Fused decoding is generally strongest: at 200B tokens, HumanEval improves from 0.31 to 0.34, and MBPP from 0.38 to 0.40.

The authors further report that a 200B-token full-bandwidth model approaches or exceeds standard models trained on two to five times more tokens on GSM8K, Math500, HumanEval, and MBPP. Pass@3 improves alongside Pass@1, suggesting that the latent pathway does not simply collapse the sampling distribution or impair diversity.

Figure 5: Latent feedback changes the accuracy–reasoning-length tradeoff on Math500, with shorter solutions often achieving equal or higher accuracy.

The reported instruction-tuned results remain positive. After long-context extension and instruction tuning, Soft or Fused decoding improves GSM8K from 64.5% to 67.9% for the 200B model and HumanEval from 42.5% to 45.9%. For the 400B model, Fused decoding reaches 71.8% on GSM8K, 48.4% on Math500, 47.6% on HumanEval, and 41.7% on MBPP. These gains persist despite the post-training stages being much shorter than pretraining.

However, the paper identifies an important contradiction. Latent feedback produces markedly shorter reasoning traces in base models, but this concision disappears after instruction tuning. The proposed explanation is distributional: instruction-tuning targets were generated using conventional verbalized reasoning, so supervised fine-tuning reinstates the verbose policy even though the model possesses a wider latent channel. This suggests that post-training data and rollout policy must be co-designed if latent computation is intended to replace explicit chain-of-thought tokens.

Representation-level evidence

The authors supplement end-task evaluations with synthetic state-tracking experiments. These tasks require the model to retain binary relations, delayed memory, or the latest values of multiple registers while processing label-independent distractors. Linear probes are trained at each residual depth.

Under standard prefilling, the final token’s layer-zero representation contains little information about the global state because the prefix has only been processed to the depth available at that position. One-step recurrent prefilling exposes the preceding top-layer state at the input, allowing shallow layers to access a fully processed summary of the prefix.

Figure 6: One recurrent step makes global state nearly perfectly decodable at layer zero, whereas standard prefilling requires additional depth to reconstruct the same state.

The strongest reported results are 99.6% layer-zero probe accuracy for completion tracking and 100% for delayed memory after one recurrent step. Multi-register experiments show that recurrent prefilling improves shallow accessibility, while full recurrence is most effective when multiple registers are repeatedly overwritten. These experiments directly support the paper’s reachability argument, but the authors appropriately distinguish representation accessibility from causal utilization: a state being linearly decodable does not establish that the language-model head uses it correctly.

Relation to adjacent approaches

The proposal is related to Feedback Transformer, latent recurrent Transformers, temporal middle-layer recurrence, continuous latent reasoning, and looped Transformers. Its principal distinction is the location and implementation of recurrence. Rather than modifying attention or adding layerwise recurrent modules, it injects the previous top-layer state at the model input through a small gated fusion module. This yields a parameter overhead of only two $h_{t-1}^L$1 matrices, compared with substantially larger layerwise or MLP-based recurrence mechanisms.

Relative to continuous reasoning methods that replace discrete tokens with latent states, full-bandwidth feedback retains ordinary tokens and uses the latent state as an auxiliary channel. This simplifies standard language-model supervision and preserves text generation, but it may be less token-efficient than approaches that eliminate discrete intermediate outputs entirely. Relative to PonderLM-style interleaving, the method avoids doubling sequence length and KV-cache size.

The theoretical framing is also distinct from standard recurrent compression. Since all previous fused representations remain in the attention cache, the method does not impose a fixed-size bottleneck on history. Its main contribution is therefore vertical accessibility, not horizontal memory compression.

Limitations and future directions

The principal empirical limitation is scale. All experiments use approximately 1B-parameter models, despite claims that the method may become more useful as model depth and representational capacity increase. It remains uncertain whether the observed data-efficiency gains persist at 7B, 70B, or larger scales, where optimization dynamics, activation statistics, serving costs, and recurrence stability may differ materially.

The feedback-pass schedule is also heuristic. The observation that 3% three-pass batches stabilize long-horizon iteration is compelling but does not establish a general rule. A more principled approach could optimize spectral properties of the feedback Jacobian, use adaptive pass allocation, or directly regularize contraction behavior. Fixed-point diagnostics and implicit differentiation may provide a way to train stable feedback maps without repeatedly testing arbitrary rollout horizons.

Several additional research directions follow naturally. Latent feedback could be combined with next-latent prediction, multi-token prediction, verifier-guided reasoning, or reinforcement learning whose rollouts are generated under the latent-feedback policy. Post-training on-policy data may recover the shorter reasoning traces observed in base models. The fusion operator could also be made adaptive across layers, tokens, or task types, potentially allowing the model to decide when continuous state transmission is useful and when ordinary token embeddings suffice.

Finally, the paper’s effective-token claims require careful compute accounting. Latent-feedback training uses additional forward passes, and fused prefilling consumes extra inference FLOPs. The comparison with models trained on more tokens is therefore most meaningful under a deployment setting where training compute, unique-data availability, and prompt-processing latency are jointly constrained. Future evaluations should report full training FLOPs, wall-clock cost, memory traffic, prompt length, generation length, and batch-level serving throughput.

Conclusion

“Full-bandwidth transformer” (2608.08888) reframes autoregressive decoding as a computation graph with an underutilized vertical communication channel. By feeding the previous top-layer state back through a gated token-conditioned fusion, it enables shallow layers to access fully processed historical information without changing the Transformer stack or imposing substantial per-token serving overhead. The multi-pass training procedure makes this recurrence compatible with parallel teacher forcing, while a small fraction of deeper-pass batches provides empirical long-horizon stability.

The reported improvements are substantial: approximately twofold pretraining data-efficiency gains in several settings, stronger math and coding performance, and shorter base-model reasoning traces at comparable or higher accuracy. The central unresolved question is scalability. If the feedback map remains stable and useful in substantially larger models, latent feedback could become a practical axis for reallocating computation from unique training tokens toward representation reuse and test-time computation.