Full-Bandwidth Transformer: Feeding Hidden States Back Through the Model
This presentation explores a structural redesign of decoder-only Transformers that addresses a hidden asymmetry in how models process information. While standard Transformers allow rich horizontal communication across tokens through attention, they severely restrict vertical communication between processing steps. The proposed latent feedback mechanism feeds each position's final hidden state back into the next token's input, widening the recurrent channel from a single discrete token to a full residual-stream vector. Through multi-pass training and careful architectural choices, this approach delivers substantial gains in data efficiency, mathematical reasoning, and code generation while adding less than 1% decoding overhead.Script
Standard Transformers have a hidden asymmetry. Dense attention connects positions horizontally with high bandwidth, but once a token is processed to the top layer, only a discrete token embedding gets fed back to the next position. Everything computed at deep layers stays depth-frozen, accessible only to subsequent layers at or above the same depth.
Latent feedback decoding widens that recurrent channel. Instead of feeding only the sampled token into the next position, the model fuses the previous top-layer hidden state with the token embedding through a gated linear unit. The token acts as a gate, and the hidden state becomes the value pathway, forcing the model to process a state-dependent input that has already traversed the full stack.
Training this recurrence requires temporal parallelism. A first pass processes the sequence normally. A second pass shifts those top-layer states one position right, fuses them with token embeddings, and processes all positions in parallel. Gradients flow through the latent pathway, turning later-pass losses into an auxiliary objective that trains hidden states to remain useful as future inputs. Most pretraining uses single-pass batches, but adding just 3 percent three-pass batches stabilizes the feedback map over arbitrarily long rollouts.
The data efficiency gains are striking. A 200 billion token full-bandwidth model with two feedback passes matches the validation loss of a 400 billion token standard baseline. On mathematical reasoning, the effect is even stronger, with models trained on 200 billion tokens exceeding conventional models trained on up to 5 times more data. The first recurrent prefill pass delivers most of the gain, with subsequent passes showing diminishing returns.
Base models using latent feedback produce markedly shorter reasoning traces at equal or higher accuracy on Math500, suggesting the latent channel can carry computational work that would otherwise require explicit token-level reasoning. But instruction tuning erases this concision. The authors attribute this to a distribution mismatch, supervised fine-tuning reinstates verbose policies because the training targets were generated with conventional verbalized reasoning, even though the model now possesses a wider computational channel.
Full-bandwidth feedback reframes autoregressive decoding as a graph with an underutilized vertical channel. By reintroducing the previous top-layer state at the input, it gives shallow layers a renewed depth budget to process fully refined historical information. The mechanism adds only two projection matrices and less than 1 percent per-token overhead, but the central question remains scalability. All experiments use 1 billion parameter models, and whether these gains persist at 7, 70, or 700 billion parameters will determine if latent feedback becomes a practical axis for reallocating computation from unique training tokens toward representation reuse. You can explore this paper and generate your own video explanations at EmergentMind.com.
