Attention-Based Transform Coding

• Attention-based transform coding is a method that uses adaptive attention mechanisms, such as Transformer architectures, to learn data-dependent transforms for compression.
• It employs strategies like DCT-based initialization, windowed/global attention, and linear attention variants to enhance decorrelation and optimize entropy modeling.
• Empirical studies show significant improvements in rate-distortion performance and computational efficiency compared to classical and CNN-based codecs.

Attention-based transform coding refers to a class of methods that integrate attention mechanisms—primarily those based on the Transformer architecture—into the transform coding pipelines of image and signal compression, or into the internal decorrelation/compression of representations within deep learning models. These approaches leverage the adaptability and long-range modeling capacity of attention to improve decorrelation, compression efficiency, and/or model optimization, frequently yielding substantial advances over classical hand-crafted transforms or purely CNN-based neural models. Attention-based transform coding encompasses frequency-domain initialization and compression of self-attention (e.g., via DCT), advanced windowed or global attention modules within learned codecs, spatio-channel attention for improved entropy modeling, and linear-complexity attention variants specifically designed for efficient transform coding.

1. Principles of Attention-Based Transform Coding

Attention-based transform coding stems from the observation that attention mechanisms can extract characteristic structured dependencies—both local and long-range—in data, thus serving as trainable, content-adaptive transforms in the analysis and synthesis stages of learned compression. Unlike classical transforms (e.g., DCT, wavelets) that apply fixed linear operations to decorrelate data, attention mechanisms implement data-dependent, often nonlinear projections and context modeling via adaptive weighting of input components.

In the context of Vision Transformers (ViTs) and image codecs, attention-based transform coding can be instantiated at several levels:

• Spectral/Orthogonal Initialization: Using fixed, orthogonal transforms such as the discrete cosine transform (DCT) to initialize or structure attention projections (Pan et al., 2024).
• Window-based and Global Attention: Employing fixed or adaptive attention blocks—windowed (local) or global—within the transform network to decorrelate spatial and cross-channel dependencies (Mudgal et al., 2024, Soltani et al., 2024).
• Context Modeling: Using transformers with causal/self-attention to model and compress the spatial and channel-wise context during entropy estimation for latent codes (Koyuncu et al., 2022, Feng et al., 9 Feb 2025).
• Efficient Linear Attention: Replacing quadratic-cost attention with linear-complexity variants tailored to dense 2D feature maps and suitable for resource-constrained coding pipelines (Feng et al., 9 Feb 2025).

This methodology is unified by the principle of learning powerful, decorrelating transforms that yield more compressible latent codes or more stable, efficient learning dynamics.

2. Transform Coding via Frequency-Domain Attention

A prominent instance is the adoption of DCT-based projections within transformers. In "Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers," fixed DCT matrices are used to initialize or replace one of the query, key, or value (Q, K, V) projections in the attention mechanism (Pan et al., 2024). This initialization ensures that the patch embeddings are projected onto orthogonal frequency channels, closely approximating principal components for natural images and maximizing decorrelation at the outset of training.

In addition, DCT-based compression can be performed by truncating high-frequency components along the channel axis before attention projections. The procedure involves:

• Projecting input features onto a DCT basis.
• Retaining only the first τC low-frequency DCT channels (for a keep-ratio τ).
• Applying reduced-dimensional Q, K, V projections in the attention computation.
• Optionally, reconstructing the output back to the full channel dimension via inverse DCT fused with the output projection.

Empirically, retaining 50–75% of channel frequencies maintains almost full accuracy while reducing parameters and FLOPs by 10–20%, demonstrating efficient trade-offs between compression and task performance (Pan et al., 2024).

This approach directly parallels classical JPEG-style transform coding, where frequency-domain energy compaction and high-frequency truncation reduce redundancy and preserve perceptually salient information.

3. Direct Attention-Based Transforms in Learned Compression

A distinct line of work develops end-to-end learned transform coding pipelines based on pure attention (transformers) or hybrid attention-convolutional architectures.

• QPressFormer demonstrates purely attention-based analysis and synthesis transforms using sequences of multi-head self-attention and cross-attention blocks with learned queries and image patches, dispensing with convolutions entirely. The latent representation is quantized and entropy-coded using a factorized prior. While this achieves competitive perceptual metrics (LPIPS, FID), standard metrics (PSNR, MS-SSIM) lag behind optimal CNNs, and model size/compute is significantly larger (Luka et al., 2023).
• Hybrid Models such as "Bi-Level Spatial and Channel-aware Transformer" insert frequency-aware spatial self-attention (separately processing high- and low-frequencies via window-based and global pooling branches), channel-wise reweighting (squeeze-and-excite), and mixed local-global feed-forward modules. This design further improves coding efficiency by explicitly targeting both short-range and long-range correlations in both spatial and channel dimensions (Soltani et al., 2024).
• Cross-Scale Window Attention modules, as in (Mudgal et al., 2024), extend the receptive field by pairing fine-scale and coarse-scale windows, enabling localized context aggregation without incurring full quadratic cost in spatial dimensions.
• Linear Attention via Bi-RWKV blocks in LALIC reduce attention complexity to linear in sequence length by mixing spatial and channel tokens through recurrent-style kernel-value weighting and depthwise convolution shifts, thereby enabling scalable modeling of global dependencies for large and high-resolution images (Feng et al., 9 Feb 2025).

These designs have demonstrated rate-distortion gains over both classical and prior learned codecs, especially when multi-scale and multi-modal dependencies are explicitly modeled.

4. Attention in Context and Entropy Modeling

A major source of compression gain in modern learned image coding is accurate rate modeling of latent representations. Attention-based context models (e.g., transformers with autoregressive masked attention) have surpassed traditional masked convolutions in capturing complex local and cross-channel dependencies (Koyuncu et al., 2022, Feng et al., 9 Feb 2025).

• Spatio-Channel Attention (Contextformer): The context model processes the latent tensor by segmenting both spatial and channel axes, supporting mixed spatial-channel attention. Masked multi-head self-attention yields per-pixel or per-chunk content-adaptive contexts, directly reducing estimated entropy by flexibly conditioning on relevant past codes (Koyuncu et al., 2022).
• RWKV-based Channel Context: LALIC's entropy model applies RWKV attention over previously coded channel chunks for each spatial location, in combination with spatial masked convolutions and hyperprior side information (Feng et al., 9 Feb 2025). This enables efficient, global context modeling with linear complexity.

These transformer-based context models provide both content adaptivity and greater modeling power than local, location-invariant context convolutions. Ablations demonstrate BD-rate savings of up to 12% relative to VTM or strong learned baselines.

5. Rate-Distortion Performance and Efficiency Trade-offs

Attention-based transform coding models have demonstrated consistent improvements in empirical rate-distortion (RD) performance compared to both classical codecs (JPEG, JPEG2000, BPG, VTM) and prior learned methods.

Key findings include:

• DCT-initialized or compressed attentions in ViTs: Parameter/FLOP reductions of up to 28% (τ=0.25) with negligible accuracy loss; at moderate compression (τ=0.5), 19% compute savings with nearly zero impact (≤0.7% top-1 drop on ImageNet-1K) (Pan et al., 2024).
• Contextformer achieves up to 11% BD-rate savings over VVC (VTM 16.2) and outperforms previous neural and handcrafted codecs across datasets (Koyuncu et al., 2022).
• Bi-level spatial-channel transformer coding attains >0.2 dB PSNR gain over strongest learned codecs at 0.5 bpp (Soltani et al., 2024).
• Linear attention methods (LALIC) close the complexity gap with quadratic transformers while achieving ~15% BD-rate improvement over VTM-9.1 with global receptive field at linear computational cost (Feng et al., 9 Feb 2025).
• QPressFormer demonstrates that a fully attention-based pipeline is feasible, matching convolutional codecs in perceptual quality (LPIPS, FID) but at the expense of higher computational and parameter cost (Luka et al., 2023).

Table: Example Empirical Trade-offs in Attention-Based Transform Coding

Approach	Params/FLOPs Reduction	BD-rate/Accuracy Impact
DCT-compressed ViT (τ=0.5) (Pan et al., 2024)	–19% FLOPs, –18% params	Top-1 Δ ≈ –0.7% (ImageNet-1K), negligible
Contextformer (Koyuncu et al., 2022)	No increase	–7% to –12% BD-rate vs VTM
LALIC (Linear Attention) (Feng et al., 9 Feb 2025)	O(N) vs O(N²) complexity	–15% BD-rate vs VTM-9.1

A plausible implication is that, while pure attention architectures offer maximal flexibility, hybrid designs (window/cross-scale, channel-mixed, frequency-aware) achieve superior compression gains at manageable complexity.

6. Technical Considerations and Implementation Aspects

Key technical features and implementation details in current attention-based transform coding research include:

• Frequency Basis Construction: DCT (type-II) matrices are used as fixed, orthogonal projections for decorrelating natural image statistics along the channel axis, initializing or partially replacing projection weights in ViTs (Pan et al., 2024).
• Window Partitioning and Cross-Scale Matching: Multi-level windowing strategies permit localized attention with manageable computation, extending context via paired coarse-fine windows (Mudgal et al., 2024, Soltani et al., 2024).
• Linearization of Attention: Bi-RWKV and similar recurrence-inspired modules facilitate global context aggregation at linear sequence complexity, obviating the cost-tradeoff of quadratic self-attention (Feng et al., 9 Feb 2025).
• Integration with Entropy Models: Attention-derived contexts interface with hyperprior side information and Gaussian mixture parameterization to minimize bit-rate under fixed distortion constraints (Koyuncu et al., 2022, Feng et al., 9 Feb 2025).
• Efficient Implementation: Fusing linear inverse transforms (IDCT) with output projections, careful token partitioning, and merge of small convolutional kernels are common strategies for practical deployment (Pan et al., 2024, Feng et al., 9 Feb 2025).

These methodological advances underscore a trend toward content-adaptive, multiscale, and efficient modeling of transform-domain representations for compression and representation learning.

7. Relation to Prior Art and Future Trajectories

Attention-based transform coding consolidates and advances several complementary approaches in data decorrelation and model optimization. Compared to DCT-Former and DCFormer, which mix high/low frequencies or pool channels globally, the most effective recent architectures explicitly segment and discard only high-frequency components, preserving low frequencies exactly (Pan et al., 2024). Fully attention-based codecs, hitherto considered impractical, are now competitive for perceptual compression but still lag CNNs in MSE/PSNR at comparable computational budgets (Luka et al., 2023).

A likely trajectory is the continued refinement of hybrid attention modules (spatial, channel, and frequency-domain) and further optimization of linear-complexity attention for massively high-resolution data. Integration of these modules with improved entropy models, perceptual training objectives, and adaptive window or frequency structures represents open directions for both vision transformers and learned codecs.

In summary, attention-based transform coding unifies frequency-domain, spatial, channel, and context modeling innovations under the umbrella of adaptive, learnable transforms. This paradigm delivers superior decorrelation, compression efficiency, and operational flexibility in both deep learning and image coding pipelines (Pan et al., 2024, Mudgal et al., 2024, Koyuncu et al., 2022, Luka et al., 2023, Soltani et al., 2024, Feng et al., 9 Feb 2025).