Reduced Transformer: Compression Strategies

Updated 14 July 2026

Reduced Transformer (RT) is an umbrella term for methods that compress transformer models by reducing parameters, attention span, or latent dimensions.
Key techniques include cross-layer weight sharing, recurrent memory tokens, and dynamic routing that lower computational load and, in some cases, reduce parameters by over 60% with marginal performance loss.
These strategies are applied across domains such as speech recognition, object detection, and robotics to enhance efficiency while maintaining high-quality outcomes.

“Reduced Transformer (RT)” is not a single stabilized model name in recent arXiv usage. The surveyed literature instead uses the label “RT” for several distinct model families, including Robotics Transformer, Real-Time Detection Transformer, and Recurrent Transformer, while genuinely reduction-oriented proposals pursue compression through cross-layer weight sharing, recurrent memory, dynamic routing, factorized weights, or dimensionality-reduction bottlenecks. The surveyed papers therefore suggest that “Reduced Transformer” is best understood as an umbrella description for multiple transformer-reduction strategies rather than as one canonical architecture (Brohan et al., 2022, Mucllari et al., 2 May 2025, Wang et al., 2023, Lv et al., 2024).

1. Terminological scope and recurrent sources of ambiguity

A persistent source of confusion is that several influential papers whose names begin with “RT” do not define RT as “Reduced Transformer.” In the robotics literature, RT-1 and RT-X use RT to mean Robotics Transformer; in detection, RT-DETR uses RT to mean Real-Time; and in sequence modeling, Recurrent Transformer is also abbreviated RT in discussion (Brohan et al., 2022, Collaboration et al., 2023, Oncescu et al., 23 Apr 2026, Lv et al., 2024).

Paper	Meaning of “RT” or related term	Relation to “Reduced Transformer”
"RT-1: Robotics Transformer for Real-World Control at Scale" (Brohan et al., 2022)	Robotics Transformer	Explicitly not “Reduced Transformer”
"Open X-Embodiment: Robotic Learning Datasets and RT-X Models" (Collaboration et al., 2023)	Robotics Transformer family	Explicitly not “Reduced Transformer”
"RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer" (Lv et al., 2024)	Real-Time Detection Transformer	RT means real-time, not reduced
"Compact Recurrent Transformer with Persistent Memory" (Mucllari et al., 2 May 2025)	CRT	Reasonably interpretable as a compact/reduced transformer variant
"ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers" (Wang et al., 2023)	ResidualTransformer	Directly compression-oriented
"The Recurrent Transformer: Greater Effective Depth and Efficient Decoding" (Oncescu et al., 23 Apr 2026)	Recurrent Transformer	Reduced chiefly in effective depth/inference footprint, not by name

The main misconception is therefore lexical rather than architectural: RT often names a task domain or deployment goal, while the actual reduction mechanism may be orthogonal. A second misconception is that “reduction” always means shrinking the transformer itself. "Transformer-based dimensionality reduction" (Ran et al., 2022), for example, reduces the data representation, not the transformer architecture.

One direct “reduced transformer” interpretation is parameter reduction by structural reparameterization. "ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers" (Wang et al., 2023) compresses Transformer encoders for streaming speech systems by sharing full-rank matrices across adjacent layers and restoring layer specificity with low-rank-plus-diagonal residuals. Its defining parameterization is

$W_l^{\text{eff}} = U_{\lceil l/K \rceil} + A_l B_l + D_l,$

where $U_{\lceil l/K \rceil}$ is a shared block weight, $A_lB_l$ is a layer-specific low-rank residual, and $D_l$ is a diagonal augmentation. The method is applied to all encoder projection matrices—Q, K, V, attention output, and FFN projections. On 10k-hour ASR/ST, the baseline Transformer layers contain 56.7M parameters, while the highlighted setting $K=3, R=2$ reduces this to 19.3M, i.e. about 34.0% of baseline size, with 13.53 WER versus 13.28 for the baseline ASR model and similarly small BLEU degradation in ST (Wang et al., 2023).

A second reduction strategy is replacing long-range attention memory with a compact recurrent state. "Compact Recurrent Transformer with Persistent Memory" (Mucllari et al., 2 May 2025) processes long sequences as short local segments, concatenates a single memory token to each segment, and compresses the segment outputs into one persistent memory vector using a GRU or NCGRU. This reduces the attention window to the current segment plus one memory token, rather than maintaining full token-level access to prior segments. The paper gives explicit FLOPs expressions for standard Transformer, Transformer-XL, and CRT-GRU, and emphasizes that CRT uses one vector for memory where Transformer-XL uses $n$ memory vectors from a previous segment. On WikiText-103, a 3-layer CRT with memory/segment 1/150 attains 31.8 perplexity versus 39.1 for a segmented Transformer and 32.6 for Transformer-XL; on Word PTB, a 3-layer CRT with 1/70 attains 58.3 versus 67.0 and 65.1, respectively (Mucllari et al., 2 May 2025).

A third interpretation is reducing required depth at fixed parameter count by adding recurrence inside layers. "The Recurrent Transformer: Greater Effective Depth and Efficient Decoding" (Oncescu et al., 23 Apr 2026) computes persistent key/value pairs from the layer output $z_i$ rather than the layer input $x_i$ , so later positions attend to earlier positions already updated by the same layer. The paper’s practical claim is not that RT means “Reduced Transformer,” but that recurrence can trade depth for width and thereby reduce KV-cache size and decode-time latency. In the 300M setting, the 6-layer Recurrent Transformer with width 2048 achieves validation cross-entropy 2.860, outperforming 6-layer, 12-layer, and 24-layer conventional Transformers at the same parameter budget; the paper states that if quality-equivalent depth is reduced by a factor $\alpha$ , KV cache size decreases by a factor of $\sqrt{\alpha}$ (Oncescu et al., 23 Apr 2026).

Taken together, these works define a coherent structural theme: reduction can mean fewer independent layer weights, fewer explicit memory vectors, or fewer layers for the same quality. This suggests that “reduced transformer” is better viewed as a family of resource-allocation strategies than as a single topology.

3. Post-training and systems-level reduction

A different line of work reduces active computation after pretraining. "ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation" (Liu et al., 2024) adds tiny routing modules to a frozen pretrained backbone and learns, per input, which tokens and which sub-parameters should be active. The method distinguishes Input Subset Selection, which keeps only $U_{\lceil l/K \rceil}$ 0 of $U_{\lceil l/K \rceil}$ 1 tokens for a module, from Parameter Subset Selection, which keeps all inputs but activates only subsets of heads or MLP experts. The token router uses

$U_{\lceil l/K \rceil}$ 2

while the parameter router uses

$U_{\lceil l/K \rceil}$ 3

Routers are trained by self-distillation, with forward top-50 KL for language and visual-language tasks, cosine distance for ViT-MAE, plus load-balancing and top-k auxiliary losses. The headline result is that 38% active MHA parameters and 56% active MLP parameters suffice to match the base pretrained model in language, while about 20% of tokens can be dropped from MLP processing; with rank-1 LoRA on Q/V projections, about 20% of tokens can also be dropped from MHA processing while retaining teacher-level performance (Liu et al., 2024).

At the hardware-software co-design level, "T-REX: A 68-567 μs/token, 0.41-3.95 μJ/token Transformer Accelerator with Reduced External Memory Access and Enhanced Hardware Utilization in 16nm FinFET" (Moon et al., 1 Mar 2025) reduces inference cost by factorizing each weight matrix into a dense shared matrix and a layer-distinct sparse matrix, executed as

$U_{\lceil l/K \rceil}$ 4

The shared factor is preloaded once and reused across layers; the sparse factor is stored as compressed $U_{\lceil l/K \rceil}$ 5 pairs. The paper combines this with 4b non-uniform quantization of $U_{\lceil l/K \rceil}$ 6, 6b uniform quantization of sparse values, 5b delta encoding of sparse indices, dynamic batching, and a two-direction accessible register file. Reported effects include 8.5–10.7× EMA reduction from factorizing training, an additional 2.1–2.9× EMA reduction from compression, and total parameter-size reduction of 15.9–25.5×. Measured system-level latency spans 68–567 $U_{\lceil l/K \rceil}$ 7s/token and energy 0.41–3.95 $U_{\lceil l/K \rceil}$ 8J/token across ViT-B, RD-NMT, S2T-Medium, and BERT-Large (Moon et al., 1 Mar 2025).

These works make explicit that reduction need not be static. A plausible implication is that the modern reduced-transformer problem has split into at least two regimes: dynamic activation reduction at the model level and memory-traffic reduction at the system level.

4. Robotics uses of “RT”: not a reduced transformer, but often reduction-aware

In robotics, “RT” most often means Robotics Transformer, not “Reduced Transformer.” "RT-1: Robotics Transformer for Real-World Control at Scale" (Brohan et al., 2022) is explicit on this point. RT-1 is a language-conditioned visuomotor policy that maps a history of 6 RGB images at $U_{\lceil l/K \rceil}$ 9 and a natural-language instruction to discretized robot actions. Its pipeline uses a USE-conditioned, FiLM-modulated EfficientNet-B3 visual encoder, followed by TokenLearner compression from 81 visual tokens per image to 8 tokens per image, after which a decoder-only transformer with 8 self-attention layers and about 19M parameters predicts 11 discretized action dimensions. The full model has about 35M parameters. The reduction mechanisms here are instrumental rather than nominative: TokenLearner gives a reported 2.4x speedup, token reuse 1.7x, and the deployed controller runs at 3 Hz. RT-1 also removes autoregressive action-token generation in the final system because it slows inference from 15 ms to 36 ms without sufficient benefit (Brohan et al., 2022).

"Open X-Embodiment: Robotic Learning Datasets and RT-X Models" (Collaboration et al., 2023) preserves this meaning of RT as Robotics Transformer while expanding training to a multi-robot mixture. RT-X standardizes heterogeneous datasets into image history plus language and an 8-dimensional tokenized action space with 256 uniform bins per dimension, of which 7 dimensions are end-effector movement and 1 is episode termination. The paper states that its aim is not to innovate in architectures, but to show that RT-1 and RT-2 style models can absorb cross-embodiment data. Thus, RT-X is better described as a scaling regime for Robotics Transformers than as a reduced architecture (Collaboration et al., 2023).

A more direct robotics-reduction paper is "SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention" (Leal et al., 2023). SARA-RT converts quadratic-attention robotic policies, including RT-2-style VLAs and Point Cloud Transformers, into linear-attention counterparts via up-training. The central linearized action form is

$A_lB_l$ 0

with SARA feature maps

$A_lB_l$ 1

For a PaLI-X 5B RT-2 variant, replacing the ViT encoder attention with SARA yields a reported forward-pass reduction from 53.2 ms to 45.7 ms on TPU, i.e. a 14% speedup, while maintaining similar mean task accuracy in the direct RT-2 comparison (Leal et al., 2023).

The robotics literature therefore clarifies an important point: RT there is usually Robotics Transformer, yet much of its technical development is explicitly about making such transformers smaller, faster, or more deployable.

5. Real-time detection transformers and training-time reduction

In detection, RT again does not mean “Reduced Transformer”; it means Real-Time Detection Transformer. "RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer" (Lv et al., 2024) keeps the original RT-DETR framework and modifies the decoder’s deformable attention to use a distinct number of sampling points for different scales, while optionally replacing grid_sample with a discrete_sample operator for deployment. The paper gives the total number of sampling points as

$A_lB_l$ 2

With grid_sample, reducing total points from 86,400 to 43,200 lowers AP only from 47.9 to 47.7 on RT-DETRv2-S, suggesting that selective sampling can reduce decoder work with minor loss. Across model scales, RT-DETRv2 preserves FPS while improving AP over RT-DETR, e.g. 46.5 $A_lB_l$ 3 47.9 AP at 217 FPS for the S model and 48.9 $A_lB_l$ 4 49.9 at 161 FPS for the M model (Lv et al., 2024).

"RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision" (Wang et al., 2024) keeps the same inference architecture as RT-DETR/RT-DETRv2 but densifies supervision during training with a CNN auxiliary branch, self-attention perturbation across query groups, and a shared-weight one-to-many decoder branch. Its total loss is

$A_lB_l$ 5

with default $A_lB_l$ 6. All added branches are training-only. RT-DETRv3-R18 reaches 48.1% AP, improving over RT-DETR-R18 by +1.6% while maintaining the same latency; RT-DETRv3-R101 reaches 54.6% AP (Wang et al., 2024).

A query-focused reduction refinement appears in "Pattern-Enhanced RT-DETR for Multi-Class Battery Detection" (Zhong et al., 13 May 2026). PaQ-RT-DETR replaces raw top- $A_lB_l$ 7 encoder-token content queries with convex combinations of a learnable pattern bank: $A_lB_l$ 8 With $A_lB_l$ 9 and $D_l$ 0, this is presented as alleviating query activation imbalance with negligible computational overhead. On a six-class battery dataset of about 8,591 images, PaQ-RT-DETR-X improves mAP@50 from 0.754 to 0.782, with +0.6M parameters, +1 GFLOP, and FPS changing from $D_l$ 1 to $D_l$ 2 (Zhong et al., 13 May 2026).

A final variant, "RT-DATR:Real-time Unsupervised Domain Adaptive Detection Transformer with Adversarial Feature Learning" (Lv et al., 12 Apr 2025), adds only training-time domain-adaptation modules to RT-DETR: local object-level feature alignment on backbone features, scene semantic feature alignment on encoder outputs, and instance feature alignment via a decoupled domain query

$D_l$ 3

The total objective is

$D_l$ 4

with $D_l$ 5 and the others set to 1. Because these modules are described as training-only, the paper reports unchanged real-time inference, including 6.3 ms latency for RT-DATR-R34 on T4 TensorRT FP16 (Lv et al., 12 Apr 2025).

This body of work suggests a distinct detection-specific interpretation of reduction: maintain the inference graph, but reduce supervision sparsity, sampling overhead, or domain mismatch during training.

6. Representation reduction rather than architecture reduction

A final meaning of “reduced transformer” concerns the representation being reduced, not the transformer’s own compute graph. "Transformer-based dimensionality reduction" (Ran et al., 2022) proposes Transformer-DR, a ViT-like encoder-decoder that maps images to a lower-dimensional latent code and reconstructs them with an MSE objective,

$D_l$ 6

Dimensionality reduction occurs in the FFN of each encoder block,

$D_l$ 7

with $D_l$ 8. The model is thus “reduced” in the sense of learned low-dimensional representation, not in the sense of attention pruning or encoder compression. On ImageNet reconstruction, the paper reports a final loss around 0.011 for Transformer-DR versus around 0.017 for an autoencoder after 20 epochs, and in face recognition it reduces features from 37632 to 12544 dimensions while retaining strong verification performance (Ran et al., 2022).

This distinction matters because it separates two research questions that are often conflated. One asks how to compress the transformer—as in ResidualTransformer, CRT, ElastiFormer, T-REX, SARA-RT, and parts of RT-1. The other asks how to use a transformer to compress the input representation, as in Transformer-DR. The surveyed literature therefore supports a careful taxonomy: “Reduced Transformer” may refer to reduced parameters, reduced attention span, reduced active computation, reduced memory traffic, reduced required depth, or reduced latent dimensionality—but not all of these are the same problem, and several influential “RT” papers are not using that name at all (Wang et al., 2023, Mucllari et al., 2 May 2025, Liu et al., 2024, Moon et al., 1 Mar 2025, Ran et al., 2022).