Full-Bandwidth Transformer

• Full-Bandwidth Transformer is an autoregressive decoder architecture that fuses each sampled token embedding with the preceding top-layer hidden state, enabling continuous vertical feedback while preserving standard Transformer blocks, causal attention, and KV caching.
• A gated fusion mechanism adds roughly 2D² parameters and less than 1% per-token generation overhead, while multi-pass training and prefix mixing help stabilize recurrence and make deeply processed information accessible to shallow layers.
• Experiments on approximately 1B-parameter models report about 2× data efficiency in the tested regime, with improvements in validation loss, mathematical reasoning, code generation, and state-tracking accuracy, although benefits vary by task and decoding mode.

“Full-bandwidth transformer” is an autoregressive decoder architecture that augments the inter-step feedback channel with latent feedback. At each decoding step, it combines the sampled token embedding with the preceding step’s final hidden state and feeds the fused representation into the ordinary Transformer stack. The method is intended to address a structural asymmetry in decoder-only Transformers: causal attention provides broad horizontal information flow across token positions, whereas the vertical feedback channel between successive decoding steps normally transmits only a discrete sampled token. Latent feedback preserves the standard Transformer blocks, causal KV cache, and language-modeling objective while allowing deeply processed continuous representations to re-enter the stack (Wang et al., 9 Aug 2026).

1. Architectural motivation

A decoder-only Transformer computes along two principal axes: horizontally across token positions and vertically through its $L$ layers. At generation time, causal self-attention allows a new token to attend to representations associated with the entire preceding context. This provides a broad horizontal channel. The vertical channel is narrower: after a token has passed through all Transformer layers, its final hidden state is normally discarded as a recurrent input, and only the sampled token is embedded at the next decoding step.

Let $h_t^\ell\in\mathbb{R}^D$ denote the representation at position $t$ and layer $\ell$, where $D$ is the residual-stream width. Under standard decoding, the next input is

$e_{t+1}=e(x_{t+1}),$

where $e(\cdot)$ is the token-embedding function and $x_{t+1}$ is sampled from the language-model distribution. The previous top-layer state $h_t^L$ remains represented indirectly through the KV cache, but it is not returned to the bottom of the stack for another complete depth traversal.

The paper characterizes the standard reachable set at position $t$ and layer $h_t^\ell\in\mathbb{R}^D$0 as

$h_t^\ell\in\mathbb{R}^D$1

whose size is $h_t^\ell\in\mathbb{R}^D$2 for sequence length $h_t^\ell\in\mathbb{R}^D$3. A shallow layer processing a new token cannot directly access a deeply processed representation from an earlier position. Earlier states are therefore depth-frozen: later layers can use them through attention, but they cannot be returned to layer $h_t^\ell\in\mathbb{R}^D$4 for renewed processing.

The sampled token provides a discrete communication channel with at most $h_t^\ell\in\mathbb{R}^D$5 bits of symbol identity, where $h_t^\ell\in\mathbb{R}^D$6 is the vocabulary size. This does not imply that all prior computation is lost, because the KV cache retains lower-layer keys and values. Rather, it means that the top-layer representation is not directly available as a new bottom-layer input. Full-bandwidth feedback introduces a continuous state pathway in addition to token transmission.

2. Latent-feedback recurrence

The ordinary Transformer stack is represented as

$h_t^\ell\in\mathbb{R}^D$7

where $h_t^\ell\in\mathbb{R}^D$8 is the causal context, implemented during autoregressive decoding by the KV cache. The language-model head produces logits

$h_t^\ell\in\mathbb{R}^D$9

and the next token is sampled according to

$t$0

In the full-bandwidth architecture, the next input is a fusion of the sampled token embedding and the previous final hidden state:

$t$1

The subsequent Transformer evaluation is

$t$2

The first position uses an ordinary embedding,

$t$3

while later positions use

$t$4

Consequently, the input sequence has the form

$t$5

The hidden state is folded into the input vector of the next position rather than represented as an additional token. Sequence length and the causal attention pattern are therefore unchanged.

Gated fusion

The principal fusion operator is a dimension-preserving gated linear unit:

$t$6

where $t$7, $t$8 is the elementwise sigmoid, and $t$9 denotes elementwise multiplication. The hidden state travels through the value pathway, while the token embedding controls a multiplicative gate.

This asymmetric construction differs from additive fusion such as

$\ell$0

Additive fusion provides a token-only bypass through which the model could suppress the latent pathway and recover ordinary token-only processing. In the gated formulation, the hidden state constitutes the principal value pathway; token identity is retained through the gate. The architecture introduces two $\ell$1 projections, corresponding to approximately $\ell$2 additional parameters, while leaving the Transformer stack otherwise unchanged.

The latent state may be written as $\ell$3, with sampled token $\ell$4:

$\ell$5

$\ell$6

where $\ell$7. The output head and sampling process remain standard.

3. Attention, KV caching, and computational cost

Latent feedback does not replace causal self-attention. The fused vector is used as the current position’s input embedding, after which the ordinary attention, residual, normalization, and MLP operations are applied.

The KV cache stores keys and values generated from the actual inputs

$\ell$8

which may now be fused representations rather than plain token embeddings. Earlier latent states do not need to be stored separately because their influence has already been incorporated into the cached keys and values. The only additional recurrent state maintained outside the conventional cache is the most recent top-layer state $\ell$9.

The reachable set under latent feedback is described as

$D$0

with size $D$1. The asymptotic sequence complexity is not changed. The architectural difference is that shallow layers can receive information that has already traversed the full depth of earlier positions.

The additional per-token computation consists of two $D$2 projections, a sigmoid, and an elementwise product:

$D$3

The paper reports an added cost below $D$4 per generated token. The Transformer is not evaluated an additional time for each token. The preceding top-layer state is already available from the previous decoding step, so latent feedback reuses an existing computation.

Three decoding regimes are distinguished:

• Standard decoding: ordinary prefill and token-only generation.
• Soft decoding: ordinary prefill followed by latent feedback during generation.
• Fused decoding: an additional fused prefill pass followed by latent-feedback generation.

Fused decoding doubles prompt-prefill computation when the extra pass is used, but the additional prefill remains parallel over prompt positions. During generation, only one ordinary Transformer evaluation is performed per token.

The method was implemented in vLLM by retaining the latest trunk hidden state for each request in a dedicated buffer. The scheme is compatible with CUDA graphs and serving patterns such as those used by EAGLE/MTP, although the serving loop must be modified to save and retrieve the additional state. The KV-cache layout and asymptotic size remain unchanged; only one additional $D$5-dimensional state buffer is required per active request.

4. Scheduled multi-pass training

Parallelizing the recurrence

Directly unrolling latent feedback during training would make position $D$6 depend sequentially on the completed Transformer computation at position $D$7, eliminating parallel teacher forcing. The training method instead parallelizes recurrence across passes.

Let $D$8 denote the top-layer state at position $D$9 on feedback pass $e_{t+1}=e(x_{t+1}),$0. The first pass is ordinary teacher forcing:

$e_{t+1}=e(x_{t+1}),$1

The second pass uses the preceding pass’s hidden state shifted one position to the right:

$e_{t+1}=e(x_{t+1}),$2

In general,

$e_{t+1}=e(x_{t+1}),$3

where all states required by pass $e_{t+1}=e(x_{t+1}),$4 were computed in pass $e_{t+1}=e(x_{t+1}),$5. Every pass is therefore parallel over sequence positions. After $e_{t+1}=e(x_{t+1}),$6 passes, information can propagate approximately $e_{t+1}=e(x_{t+1}),$7 token steps through the latent-feedback recurrence.

Multi-pass objective

The model applies next-token prediction to every pass:

$e_{t+1}=e(x_{t+1}),$8

with $e_{t+1}=e(x_{t+1}),$9. The first-pass loss preserves ordinary no-feedback behavior, which is important for prompt processing. Later-pass losses train the model to consume latent-feedback inputs.

Gradients are not detached between passes. Consequently, later-pass losses can backpropagate through the fused states into earlier passes. The training signal therefore encourages top-layer states to remain reusable as future inputs rather than merely predictive of the immediately following token.

Training schedule

The reported training mixtures are:

Run	Pass mixture	Tokens	Token-equivalent compute
10B	100% three-pass	10B	40B
100B	75% one-pass, 25% three-pass	100B	150B
200B	75% one-pass, 22% two-pass, 3% three-pass	200B	256B
400B	75% one-pass, 22% two-pass, 3% three-pass	400B	512B

A model trained with 75% one-pass and 25% two-pass batches performed well within its training horizon but became unstable when repeatedly iterated beyond that horizon. Adding only 3% three-pass batches made the feedback map stable for at least 30 repeated passes in the reported diagnostic and stable in tests extending to 1,000 feedback passes.

The authors interpret the result as evidence that the learned recurrence becomes approximately contractive toward a fixed point, with

$e(\cdot)$0

decaying toward a small plateau. Ordinary training dominates initially, feedback passes are introduced during pretraining, and deeper-pass batches are added later. The paper reports the mixture proportions and 200 warm-up optimization steps but does not provide a complete token-by-token transition schedule.

Prefix mixin and stabilization

At inference, prompt positions generally use plain embeddings, whereas generated positions use fused inputs. Prefix mixin reproduces this boundary during training:

$e(\cdot)$1

where $e(\cdot)$2 is a randomly selected prefix length.

The reported stabilization mechanisms are depth scaling, RMSNorm on the fused input, tied embedding and output weights, and jitter noise on the carried state:

$e(\cdot)$3

with $e(\cdot)$4.

5. Experimental configuration and empirical findings

The experiments use approximately 1B-parameter decoder-only models with 24 Transformer layers, hidden dimension $e(\cdot)$5, 6,656-dimensional SiLU-GLU feed-forward blocks, vocabulary size 100,352, tied embedding and output weights, grouped-query attention, 16 query heads, 8 shared key/value heads, QK RMS normalization, rotary positional embeddings, context length 8192, mostly 2048-token sliding-window attention, full attention every sixth layer, and RMS normalization around residual blocks and at the output.

Optimization uses NorMuon for matrix parameters with learning rate $e(\cdot)$6 and matrix weight decay $e(\cdot)$75\times10^{-4}e(⋅)e(\cdot)810^{-5}e(⋅)e(\cdot)9\sigma=0.02.</sup></sup>Theusualglobalbatchsizeis300Ktokens.The1T−tokenno−feedbackbaselineuses1.2Mtokensperglobalbatch.</p><p>Evaluationincludesvalidationlossandperplexity;5−shotLMEvalacrossRTE,TruthfulQA−MC2,ARC−Easy,<ahref="https://www.emergentmind.com/topics/arc−challenge"title=""rel="nofollow"data−turbo="false"class="assistant−link"x−datax−tooltip.raw="">ARC−Challenge</a>,BoolQ,PIQA,WinoGrande,OpenBookQA,COPA,andMMLU;0−shotcomparisons;<ahref="https://www.emergentmind.com/topics/arithmetic−reasoning−gsm8k"title=""rel="nofollow"data−turbo="false"class="assistant−link"x−datax−tooltip.raw="">GSM8K</a>andMATH−500;<ahref="https://www.emergentmind.com/topics/humaneval"title=""rel="nofollow"data−turbo="false"class="assistant−link"x−datax−tooltip.raw="">HumanEval</a>and<ahref="https://www.emergentmind.com/topics/mostly−basic−programming−problems−mbpp"title=""rel="nofollow"data−turbo="false"class="assistant−link"x−datax−tooltip.raw="">MBPP</a>;instruction−tunedmodels;andsyntheticstate−trackingandprobingexperiments.</p><h3class=′paper−heading′id=′validation−loss−and−data−efficiency′>Validationlossanddataefficiency</h3><p>Themainvalidation−lossresultsshowthatfusedprefillimprovesvalidationloss,withmostoftheimprovementarisingafterthefirstfeedbackpassanddiminishingreturnsthereafter.A100B−tokenfull−bandwidthmodelwithtwofeedbackpassesreachestheperformanceofa200B−tokenstandardTransformer,whilea200B−tokenfull−bandwidthmodelwithtwofeedbackpassesreachestheperformanceofa400B−tokenstandardTransformer.Thepapersummarizesthisdisplayedregimeasapproximately.</sup></sup> The usual global batch size is 300K tokens. The 1T-token no-feedback baseline uses 1.2M tokens per global batch.</p> <p>Evaluation includes validation loss and perplexity; 5-shot LM Eval across RTE, TruthfulQA-MC2, ARC-Easy, <a href="https://www.emergentmind.com/topics/arc-challenge" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">ARC-Challenge</a>, BoolQ, PIQA, WinoGrande, OpenBookQA, COPA, and MMLU; 0-shot comparisons; <a href="https://www.emergentmind.com/topics/arithmetic-reasoning-gsm8k" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">GSM8K</a> and MATH-500; <a href="https://www.emergentmind.com/topics/humaneval" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">HumanEval</a> and <a href="https://www.emergentmind.com/topics/mostly-basic-programming-problems-mbpp" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">MBPP</a>; instruction-tuned models; and synthetic state-tracking and probing experiments.</p> <h3 class='paper-heading' id='validation-loss-and-data-efficiency'>Validation loss and data efficiency</h3> <p>The main validation-loss results show that fused prefill improves validation loss, with most of the improvement arising after the first feedback pass and diminishing returns thereafter. A 100B-token full-bandwidth model with two feedback passes reaches the performance of a 200B-token standard Transformer, while a 200B-token full-bandwidth model with two feedback passes reaches the performance of a 400B-token standard Transformer. The paper summarizes this displayed regime as approximately x_{t+1}$0 pretraining data efficiency.

A reported 0-shot comparison gives an average score of 52.66 for a 200B-token full-bandwidth model without feedback passes and 53.58 with one feedback pass. For example, the one-pass condition improves WinoGrande from 60.46 to 62.59, PIQA from 71.11 to 71.49, OpenBookQA from 34.60 to 35.00, ARC-Easy from 62.42 to 63.43, and ARC-Challenge from 34.73 to 35.41.

Base-model generation

The 200B-token model obtains:

• MATH-500: Standard decoding 0.27; Soft decoding 0.37.
• HumanEval: Standard decoding 0.31; Fused decoding 0.34.
• MBPP: Standard decoding 0.38; Fused decoding 0.40.

Soft decoding improves over Standard decoding on every reported task at both model scales. The preferred regime is task-dependent: mathematical generation benefits particularly from latent state carried during generation, whereas coding benefits particularly from fused prompt prefilling.

The paper also reports that 200B-token recurrent models approach or exceed standard models trained with two to five times more tokens on some tasks, including performance near the 1T-token baseline on GSM8K and HumanEval. This is an empirical, task-dependent comparison rather than an exact equivalence between latent feedback and additional training data.

Instruction-tuned results

After long-context extension from 8K to 32K and six billion instruction-tuning tokens, the reported results include:

Task	Full-bandwidth 200B Standard	Full-bandwidth 200B Soft	Full-bandwidth 200B Fused	Full-bandwidth 400B Standard	Full-bandwidth 400B Soft	Full-bandwidth 400B Fused
GSM8K	64.52	67.93	67.55	67.90	71.00	71.80
MATH-500	43.80	45.60	45.60	46.00	45.40	48.40
HumanEval	42.54	45.06	45.92	46.50	47.20	47.60
MBPP	38.39	39.80	41.22	40.50	40.60	41.70

For comparison, standard 200B, 400B, and 1T-token models obtain GSM8K scores of 62.93, 68.39, and 70.13; MATH-500 scores of 42.40, 46.40, and 47.40; HumanEval scores of 37.16, 44.85, and 50.01; and MBPP scores of 38.61, 40.28, and 41.93, respectively.

The results show task-dependent benefits, but the supplied results do not include formal significance tests or confidence intervals.

Reasoning-trace length

On the base 200B-token model, Soft decoding often produces shorter reasoning traces than Standard decoding while preserving or improving MATH-500 accuracy. The reported comparison uses median lengths to reduce the effect of outliers. For the question concerning the last nonzero decimal digit of $x_{t+1}$1, Standard decoding produces a long division explanation, whereas Soft decoding gives $x_{t+1}$2 and immediately answers $x_{t+1}$3.

The authors interpret this as evidence that intermediate computation can be carried in the continuous latent state rather than verbalized token by token. The effect disappears after instruction tuning, which uses ordinary verbose token-level reasoning traces. On-policy post-training using latent-feedback rollouts is proposed as a future direction rather than an evaluated method.

State-tracking probes

Synthetic experiments test whether latent feedback makes globally processed information accessible to shallow layers. With one recurrent step, layer-0 linear-probe accuracy reaches 99.6% on completion tracking and 100% on delayed-memory tracking. Under standard prefilling, layer-0 accuracy is near chance because the final token’s input has not yet received the prefix’s deep processing.

In multi-register latest-write tracking, one recurrent step improves shallow accessibility, while full recurrence performs best when there are more overwrites. Linear decodability demonstrates that information is present in the representation, but does not establish that the language-model head uses it optimally.

6. Interpretation, limitations, and relation to Transformer efficiency

The full-bandwidth Transformer should not be confused with a model that has a larger vocabulary, longer context, mutable global memory, or asymptotically greater attention bandwidth. Its defining change is the transmission of a $x_{t+1}$4-dimensional continuous top-layer state between decoding steps:

$x_{t+1}$5

The token controls the gate, while the previous top-layer representation supplies the value. This lets deeply processed information return to the input layer and receive another depth budget. The method therefore combines information transport, recurrent computation, and additional training supervision.

The evidence supports three related effects:

1. Information transport: state-tracking probes show that deeply processed information becomes accessible at shallow layers.
2. Effective depth expansion: additional fused prefill passes improve validation loss and downstream evaluation, with diminishing returns after the first pass.
3. Empirical data efficiency: models trained on 100B and 200B tokens with feedback reportedly match standard models trained on approximately 200B and 400B tokens in the displayed regime.

These effects are not independent in the reported experiments. Additional feedback passes provide more computation, the recurrent pathway transports latent information, and later-pass losses provide an auxiliary training signal. The method is consequently not merely a larger memory mechanism: its distinctive feature is the restoration of a top-to-bottom vertical communication path across decoding steps.

Important limitations remain. Experiments are limited to approximately 1B parameters, and the feedback-pass schedule is heuristic. Multi-pass training increases token-equivalent compute and activation memory because feedback states are not detached. Fused decoding adds a prompt-prefill pass, while generation requires an additional pair of projections and gating operations. Long-horizon recurrence can become unstable without scheduling, normalization, state scaling, and jitter. Serving systems require a separate latest-hidden-state buffer and modifications to the decoding loop.

The benefits are task-dependent. Mathematical generation benefits particularly from Soft decoding, coding benefits particularly from Fused decoding, and shorter reasoning traces disappear after instruction tuning under the reported off-policy setup. The architecture also retains discrete tokens and uses them as gates; it is therefore not a purely latent-reasoning or token-free system.

In broader terms, the full-bandwidth Transformer expands the inter-step communication capacity of an autoregressive model without replacing its Transformer blocks or KV-cache representation. Its central design claim is that a sampled token is an unnecessarily narrow interface between successive decoding computations when a continuous top-layer state is already available. The reported results indicate that reusing this state can improve representation accessibility, validation loss, downstream generation, and some measures of training-data efficiency, while preserving the standard language-modeling framework.