Asymmetric Transformer Decoder Designs

• Asymmetric Transformer Decoder is a design principle that deliberately breaks standard encoder-decoder symmetry to optimize computation for specific bottlenecks.
• It leverages techniques such as unequal layer depths, fused sub-layers, and dimensional asymmetry to enhance efficiency in tasks like translation, OCR, and dense prediction.
• Empirical studies indicate notable latency improvements with minimal accuracy trade-offs, often further balanced through methods like sequence-level knowledge distillation.

Asymmetric Transformer Decoder denotes a class of decoder designs that intentionally break one or more symmetries of the canonical Transformer. In the standard encoder–decoder formulation, a decoder layer stacks masked self-attention, encoder–decoder attention, and a position-wise feed-forward network; in Pre-Norm form this is written as $Y_1 = X + \mathrm{MHSA}(\mathrm{LN}(X))$, $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$, and $Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$ (Li et al., 2021). The literature uses “asymmetric” for several distinct departures from that template: unequal encoder/decoder depth, compressed decoder sub-layers, heterogeneous encoder and decoder paradigms, unequal query and key/value dimensionalities, segment-specific masking and positional schemes in decoder-only streams, and multi-path or top-down decoders that do not mirror their encoders (Wang et al., 2019, Brattoli et al., 4 Aug 2025, Guo et al., 2024, Shen et al., 2023).

1. Canonical baseline and principal forms of asymmetry

The standard decoder is symmetric in two senses. First, it is architecturally paired with a structurally similar encoder: both are deep stacks of attention and FFN sub-layers. Second, within the decoder itself, masked self-attention, cross-attention, and FFN are treated as distinct serial components. Asymmetric decoders relax one or both assumptions. In some cases the asymmetry is algebraic, as when multiple sub-layers are fused into one. In others it is dimensional, as when queries remain low-dimensional while keys and values stay in the original feature space. In still others it is directional, with source and target tokens following different masking, positional, or communication rules.

Form of asymmetry	Mechanism	Representative instances
Depth and sub-layer asymmetry	Deep encoder/shallow decoder; one compressed decoder sub-layer replaces three canonical sub-layers	CAN for machine translation (Li et al., 2021)
Heterogeneous encoder/decoder paradigm	Transformer encoder paired with a single-layer GRU decoder	Hybrid NMT decoder (Wang et al., 2019)
Dimensional asymmetry	Queries in $d_q=64$; keys/values in $d_{kv}=1536$ or $768$	AsymTransDec for NSCLC MIL (Brattoli et al., 4 Aug 2025)
Stream and mask asymmetry	One decoder stream over source and target with separate positions; or visual prefix plus textual suffix	DST (Guo et al., 2024), DTrOCR (Fujitake, 2023)
Multi-path and multi-scale asymmetry	Bilateral, pyramid, top-down, or shared-weight decoders distinct from encoder topology	ABiU-Net (Qiu et al., 2021), CFPFormer (Cai et al., 2024), FPPformer (Shen et al., 2023), SepReformer (Shin et al., 2024)

This variety means that the term does not refer to a single decoder block. It refers to a design principle: the decoder is deliberately made non-mirroring, either relative to the encoder or relative to the standard Transformer decoder itself.

2. Sequence-to-sequence asymmetry in machine translation

A direct architectural instance appears in the Compressed Attention Network. Its decoder layer replaces masked self-attention, cross-attention, and FFN with a single compressed sub-layer. Using Pre-Norm notation, the layer computes

$$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$

then

$$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$

with residual update $X_{\text{next}} = X + Y$ and $\widetilde W_{v1} = W_{v1} W_1$, $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$0. The compression rests on three conditions: the adjacent decoder inputs satisfy the approximation $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$1; empirical cosine similarity between inputs of self-attention and cross-attention is high, reported as greater than $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$2 on the diagonal; and the attention weighted sum is linear, so the first FFN projection can be pushed through the attention operation. The dominant complexity remains $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$3, but the critical path shrinks from three sequential sub-layers to one, increasing parallelism and reducing kernel launches (Li et al., 2021).

The empirical consequences are latency-oriented rather than asymptotic. On 14 WMT machine translation tasks, CAN is reported as approximately $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$4–$Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$5 faster than the strong $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$6 deep-encoder/shallow-decoder baseline while losing less than $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$7 BLEU on average, and up to approximately $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$8 faster than the standard $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$9 baseline. The paper’s WMT14 En$Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$0De ablation isolates the gain sources: compress only attentions gives BLEU $Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$1 with $Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$2 speed, compress only cross+FFN gives BLEU $Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$3 with $Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$4 speed, and compress all gives BLEU $Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$5 with $Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$6 speed. Sequence-level knowledge distillation can erase the small BLEU deficit and, in the reported WMT14 En$Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$7De case, raises CAN from $Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$8 to $Y_3 = Y_2 + \mathrm{FFN}(\mathrm{LN}(Y_2))$9 (Li et al., 2021).

A more radical asymmetry replaces the decoder paradigm altogether. In the hybrid model of Chen et al., the encoder remains a standard Transformer, while the decoder becomes a single-layer GRU with additive, dot-product, or multi-head attention to encoder outputs. Their time breakdown on 1000 pseudo sentences with beam size $d_q=64$0 on a Tesla K40 GPU attributes Transformer inference latency primarily to decoder-side self-attention, cross-attention, and FFN: the base Transformer totals $d_q=64$1, of which decoding accounts for $d_q=64$2, whereas a 1-layer GRU RNMT totals $d_q=64$3, with decoding at $d_q=64$4. With sequence-level knowledge distillation, the hybrid network can decode 4-times faster than the Transformer and achieves comparable translation quality to the original Transformer; the reported end-to-end speedups are approximately $d_q=64$5–$d_q=64$6 on Chinese–English NIST and WMT17 (Wang et al., 2019).

3. Dimensional, segmental, and decoder-only asymmetry

In computational pathology, the Asymmetric Transformer Decoder is realized as dimensional asymmetry between a small query set and a large feature bank. For NSCLC mutation prediction with multiple-instance learning, learned queries live in $d_q=64$7, while patch embeddings remain in $d_q=64$8 for the ViT-based foundation model or $d_q=64$9 for the CNN-based foundation model. Attention is defined as

$d_{kv}=1536$0

The MLP and residual path stay in the small query space, while keys and values preserve the original feature dimensionality. Tissue type is integrated directly by adding learned embeddings $d_{kv}=1536$1 to each patch feature, $d_{kv}=1536$2, with $d_{kv}=1536$3. Using ViT features, the reported internal average AUROC is $d_{kv}=1536$4, compared with $d_{kv}=1536$5 for ABMIL, $d_{kv}=1536$6 for TransMIL, and $d_{kv}=1536$7 for MambaMIL; the abstract summarizes the gain as an average of $d_{kv}=1536$8, and over $d_{kv}=1536$9 for rare mutations such as ERBB2 and BRAF (Brattoli et al., 4 Aug 2025).

In simultaneous translation, asymmetry is organized over a single decoder-only stream. The Decoder-only Streaming Transformer concatenates source and target prefixes in one sequence, but assigns positions separately: source positions follow $768$0, target positions follow $768$1, so the target prefix is unaffected by expansion of the source prefix. Its Streaming Self-Attention introduces allocation masses $768$2 over source prefixes and defines a sufficiency score $768$3 for READ/WRITE decisions. Training uses the composite loss

$768$4

combining translation, summation, latency, and cross-layer consistency constraints. On WMT15 De$768$5En with $768$6, DST reports AL $768$7 and BLEU $768$8; the efficiency table lists $768$9M parameters for DST versus $$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$0M for Wait-k and HMT, with inference speed-up $$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$1 and training speed-up $$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$2 relative to HMT (Guo et al., 2024).

A different decoder-only asymmetry appears in OCR. DTrOCR removes the vision encoder and cross-attention entirely, placing image patches and text tokens in a single autoregressive sequence of the form $$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$3. Visual tokens come from non-overlapping $$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$4 patches on a $$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$5 image, giving $$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$6 patches, and are fed directly into a GPT-2 style decoder with $$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$7 layers, $$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$8 attention heads, and hidden size $$A = \mathrm{SoftMax}\!\left(\frac{Z W_q [Z W_{k1}, H W_{k2}]^\top + M_{\text{causal}}}{\sqrt d}\right),$$9. Because all visual tokens precede all text tokens, a text token can attend to the full visual prefix and the previous text history under a causal mask, but not to future text. On the Chinese Text Recognition benchmark, the reported DTrOCR system has $$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$0M parameters and $$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$1 FPS, with accuracies of $$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$2 on scene text, $$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$3 on web, $$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$4 on document, and $$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$5 on handwriting (Fujitake, 2023).

4. Multi-scale, bilateral, and reconstruction-oriented decoders

In dense prediction, asymmetry often appears as a decoder topology that differs qualitatively from the encoder. ABiU-Net uses an asymmetric bilateral decoder with two paths. The Transformer Decoder Path operates over four stages from stride $$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$6 to $$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$7 and decodes coarse salient object locations from transformer-encoded features; the Hybrid Decoder Path operates over six stages from stride $$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$8 to $$Y = \mathrm{ReLU}(Z W_1 + A [Z \widetilde W_{v1}, H \widetilde W_{v2}] + b_1) W_2 + b_2,$$9 and refines boundaries and details using CNN features. Communication is unidirectional: TDecPath outputs feed into HDecPath, but not conversely. The decoder itself is convolutional with channel attention rather than token self-attention. The reported DUTS-test scores are $X_{\text{next}} = X + Y$0, $X_{\text{next}} = X + Y$1, weighted $X_{\text{next}} = X + Y$2, and $X_{\text{next}} = X + Y$3, and the ablations show that TED+HED+DS, the full asymmetric bilateral decoder with deep supervision, yields the best results (Qiu et al., 2021).

CFPFormer centers asymmetry on the decoder side through four Cross Feature Pyramid blocks. At each level, queries come from the upsampled decoder feature, while keys and values concatenate that feature with encoder features and then undergo Feature Re-encoding through patch embedding. Attention is decomposed axially and modulated by a Gaussian decay mask

$X_{\text{next}} = X + Y$4

The decoder ascends from coarse to fine resolution and is explicitly distinct from the encoder backbone, which may be VGG-16, ResNet-50, or PVT. On ACDC with VGG-16, bilinear interpolation yields $X_{\text{next}} = X + Y$5M parameters and DSC Avg. $X_{\text{next}} = X + Y$6, compared with $X_{\text{next}} = X + Y$7M and DSC Avg. $X_{\text{next}} = X + Y$8 for transpose convolution. Reported ACDC averages are $X_{\text{next}} = X + Y$9 for the ResNet-50 variant and $\widetilde W_{v1} = W_{v1} W_1$0 for the PVT variant; for VOC 2007 detection, CFPFormer-T with ResNet-50 reports AP50 $\widetilde W_{v1} = W_{v1} W_1$1 (Cai et al., 2024).

For time-series forecasting, FPPformer defines asymmetry as a bottom-up encoder and a top-down decoder. The encoder refines element-wise features within patches and then performs patch-wise self-attention; the decoder reverses this order by applying patch-wise cross-attention first and element-wise self-attention second. Decoder stages fuse lateral encoder features through

$\widetilde W_{v1} = W_{v1} W_1$2

and the model uses $\widetilde W_{v1} = W_{v1} W_1$3 encoder and decoder stages. In the reported prediction-length-$\widetilde W_{v1} = W_{v1} W_1$4 ablation averaged across eight multivariate datasets, the complete model achieves average MSE $\widetilde W_{v1} = W_{v1} W_1$5, compared with $\widetilde W_{v1} = W_{v1} W_1$6 for a bottom-up replica, $\widetilde W_{v1} = W_{v1} W_1$7 for replacing the decoder with a linear projection, and $\widetilde W_{v1} = W_{v1} W_1$8 for patch-only attention (Shen et al., 2023).

Speech separation supplies a further reconstruction-oriented interpretation. “Separate and Reconstruct” introduces an asymmetric encoder–decoder in which the encoder analyzes a single latent sequence, a split module expands it into $\widetilde W_{v1} = W_{v1} W_1$9 speaker streams, and a weight-shared Transformer decoder reconstructs each stream while optionally performing cross-speaker attention. The early split plus shared decoder is called ESSD. On WSJ0-2Mix, the decoder design ablation reports $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$00 dB for late split plus original decoder, $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$01 dB for early split plus multiple decoders, $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$02 dB for ESSD, and $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$03 dB for ESSD plus cross-speaker processing; the large model SepReformer-L reaches $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$04 dB SI-SNRi (Shin et al., 2024).

5. Causal geometry, anisotropy, and depth-wise asymmetry

A theoretical account of decoder asymmetry treats the causal mask as a graph constraint. In a decoder-only transformer, self-attention with a lower-triangular mask induces a directed acyclic graph over token positions: token $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$05 can aggregate only from $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$06. The resulting geometry produces asymmetric receptive fields, high effective influence of early tokens, and no reverse flow from later to earlier positions in the same layer. The graph-theoretic analysis associates this with attention sinks, a runway problem for late tokens, and last-token representation collapse, because the last token is both the generator state and the point at which the largest receptive field must be compressed. Proposed mitigations include dedicated sink tokens, pause tokens, differential attention, more heads and width, late global mixing, and structured sparse connectors (Lee, 9 Dec 2025).

A distinct but related depth-wise asymmetry appears in embedding geometry. The anisotropy study defines the spectral score

$Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$07

where $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$08 is the empirical covariance. Across GPT-2, OPT, LLaMA-2, Falcon, GPT-J, Bloom, and Pythia decoders, $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$09 follows a bell-shaped profile: early and late layers are typically in the range $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$10–$Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$11, while middle-layer peaks are typically $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$12–$Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$13. Encoder profiles in BERT, RoBERTa, and ALBERT are flatter, typically $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$14–$Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$15. The same study reports a two-phase intrinsic-dimension trajectory—initial expansion and later compression—and proposes asymmetric capacity allocation across depth, such as more heads, wider MLPs, and more MoE capacity in the middle third of the decoder (Razzhigaev et al., 2023). This suggests that asymmetry need not only be between encoder and decoder; it can also be internal to decoder depth.

6. Performance profile, misconceptions, and limitations

Across the surveyed literature, asymmetry serves different operational goals. In translation it mainly targets autoregressive latency and critical-path reduction; in computational pathology it constrains capacity while preserving high-fidelity patch features; in simultaneous translation it couples policy learning with decoder-only generation; in OCR it collapses vision and language into a single causal stream; in dense prediction and forecasting it organizes coarse-to-fine reconstruction; in speech separation it turns the decoder into a discriminative reconstructor (Li et al., 2021, Brattoli et al., 4 Aug 2025, Guo et al., 2024, Fujitake, 2023, Shen et al., 2023, Shin et al., 2024).

The surveyed models also show that asymmetry is not synonymous with changing attention asymptotics. CAN preserves the dominant $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$16 term while shortening the serial path (Li et al., 2021). DTrOCR remains quadratic in the total visual-plus-text sequence length because all tokens share masked self-attention (Fujitake, 2023). AsymTransDec avoids quadratic $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$17 self-attention among patches, but its cross-attention still scales as $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$18 (Brattoli et al., 4 Aug 2025). ABiU-Net’s decoder, by contrast, has no quadratic token attention and scales with convolutions, channel attention, and interpolation (Qiu et al., 2021). The efficiency gain therefore depends on which symmetry is broken.

Accuracy trade-offs are likewise heterogeneous. CAN reports small BLEU drops without knowledge distillation and recovers them with sequence-level KD (Li et al., 2021). The hybrid Transformer–GRU decoder reaches comparable BLEU only after KD (Wang et al., 2019). AsymTransDec, DTrOCR, DST, CFPFormer, FPPformer, and SepReformer all report improvements over strong baselines in their respective tasks rather than speed-only compromises (Brattoli et al., 4 Aug 2025, Fujitake, 2023, Guo et al., 2024, Cai et al., 2024, Shen et al., 2023, Shin et al., 2024). This suggests that “asymmetric” is better understood as a biasing strategy for decoder computation than as a uniform compression recipe.

The limitations are correspondingly domain-specific. CAN’s identity-input approximation is weaker in lower layers, and the paper explicitly links that to small BLEU drops when all decoder layers are compressed (Li et al., 2021). AsymTransDec is trained from slide-level weak labels and remains exposed to class imbalance, domain shift, and limited interpretability (Brattoli et al., 4 Aug 2025). DTrOCR inherits a practical limit from sequence length and quadratic attention, with a maximum token length of $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$19 (Fujitake, 2023). ABiU-Net may fail on tiny or weakly conspicuous objects and highly confusing backgrounds (Qiu et al., 2021). The speech-separation decoder assumes a known number of speakers $Y_2 = Y_1 + \mathrm{MHA}(\mathrm{LN}(Y_1), E)$20 (Shin et al., 2024). At the theoretical end, directed-graph measures for oversquashing in causal decoders remain incomplete, and the GNN-based account explicitly notes that effective resistance and commute time are not directly defined for causal DAGs (Lee, 9 Dec 2025).

Taken together, these works establish asymmetric decoder design as a recurring research pattern rather than a single architecture. Its central claim is architectural: decoder computation need not preserve the same depth, sub-layer factorization, dimensionality, or information topology as the rest of the model. The specific asymmetry varies, but the underlying objective is consistent—to concentrate decoder capacity and latency where the task’s causal, multimodal, or multiscale bottlenecks actually lie.