Papers
Topics
Authors
Recent
Search
2000 character limit reached

Low-Rank Attention Residuals

Published 19 Jun 2026 in cs.LG and cs.CL | (2607.09694v1)

Abstract: Attention Residuals replace the fixed residual sum with depthwise attention over previous sub-layer outputs in LLMs, but use each output as both a full-dimensional key and value. This couples routing with representation and makes depth-routing scores scale with the hidden width dd. We propose Low-Rank Attention Residuals (LR-AttnRes), which keep full-dimensional residual values while using rr-dimensional keys, with r≪dr \ll d, for routing. Projected LR-AttnRes emits learned low-rank keys from existing output projections, decoupling routing from residual content and achieving the best validation loss among the variants tested. Sliced LR-AttnRes uses the last rr dimensions of each value as the routing key, removing the auxiliary key-projection path and reducing residual-side FLOPs while still improving performance. Comprehensive sweeps show that depthwise routing can be effective with far fewer dimensions than the model width. We release code and models to facilitate future research.

Authors (1)

Summary

No one has generated a summary of this paper yet.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Continue Learning

We haven't generated follow-up questions for this paper yet.

Tweets

Sign up for free to view the 2 tweets with 145 likes about this paper.