Channel-Aware Transformer (CAT)

• CAT is a class of transformer designs that conditions attention mechanisms on channel structure and modality roles rather than treating tokens symmetrically.
• It is applied in diverse areas including communications, time series analysis, biosignal processing, and image compression to enhance performance and efficiency.
• Key principles include cross-modal attention, specialized pooling or gating, and context-conditioned routing that adapt to domain-specific channel priors.

Channel-Aware Transformer (CAT) denotes a class of transformer-based or transformer-like designs in which attention, pooling, gating, or routing is explicitly conditioned on channel structure, modality roles, or physically meaningful priors rather than treating all tokens symmetrically. In recent arXiv literature, the term does not identify a single canonical architecture. Instead, it names several related constructions: CAT-CENet for near-field MIMO ISAC channel estimation, where sensing-derived Queries guide attention over pilot-derived Keys and Values; CaT in multivariate time series, where time tokens and channel tokens interact through bidirectional cross-attention; a channel-aware Set Transformer for EEG seizure prediction, where channel importance is accumulated and used for patient-specific sensor selection; and a context-aware MoE transformer for beam prediction, where routing is conditioned on propagation descriptors. Related works in nano-drone depth estimation and learned image compression implement channel-aware mechanisms without adopting CAT as the paper’s primary acronym [2607.03224] [2203.04298] [2507.15364] [2605.19997] [2303.10386] [2408.03842]. This suggests that CAT is best understood as an architectural inductive bias rather than a standardized model family.

1. Nomenclature and conceptual scope

The literature uses closely related names for distinct but overlapping ideas. Some papers define a model explicitly called a Channel-Aware Transformer or an obvious derivative, whereas others deploy channel-aware attention inside a broader architecture.

Domain	Name used in paper	Channel-aware mechanism
Near-field MIMO ISAC	CAT-CENet	Sensing Queries attend to pilot Keys/Values
Multivariate time series	CaT	Cross-attention between time and channel tokens
EEG seizure prediction	channel-aware Set Transformer	Attention-based channel selection and pooling
Beam prediction	CAT-MoEformer	Scene-conditioned expert routing
Nano-drone depth estimation	CADiT	Channel correlation distillation map
Learned image compression	HSCATB with CaSA	SE-style channel gate inside a Transformer block

This nomenclature matters because “channel-aware” is not uniform across domains. In communications, it can mean guidance by channel physics or propagation context. In multivariate sequence modeling, it typically refers to explicit modeling of inter-channel interactions. In compression and distillation, it often means modulation in the feature-channel dimension. A common misconception is that CAT always denotes a full self-attention backbone over channel tokens. That is not supported by the available literature: some instantiations rely on cross-attention, some on Set Transformer pooling, some on mixture-of-experts routing, and some on lightweight channel gates rather than full QKV attention [2303.10386] [2408.03842].

A second terminological point is that not every channel-aware module should be retroactively renamed CAT. The nano-drone paper consistently uses “Channel-Aware Distillation Transformer (CADiT)” and explicitly notes that the term CAT does not appear there. Likewise, the learned image compression paper does not use the CAT acronym, although its CaSA module is presented as a concrete channel-aware transformer design [2303.10386] [2408.03842].

2. Core design principles

Across the surveyed models, the defining property of CAT is asymmetry in how channel information enters attention. Standard attention is typically written as
$$
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,
$$
but CAT variants alter the semantics of $Q$, $K$, and $V$, or replace token-wise attention by channel-wise correlation, pooling, or gating [2607.03224] [2203.04298].

In CAT-CENet, channel awareness is encoded through modality roles. The communication pilot branch generates $K$ and $V$ from the noisy least-squares estimate, while the sensing branch generates $Q$ from radar-estimated near-field array responses. For head $j$,
$$
Q_j=H_2W_j^Q,\quad K_j=H_1W_j^K,\quad V_j=H_1W_j^V,
$$
with
$$
E_j=\mathrm{Softmax}\left(\frac{Q_jK_j^\top}{\sqrt{F}}\right),\quad Z_j=E_jV_j,\quad H_3=H_1+Z.
$$
Because sensing-derived Queries are matched against pilot-derived Keys, the attention matrix peaks where radar targets overlap with communication scatterers, so the reweighted Value features enhance exactly those channel components [2607.03224].

In CaT for multivariate time series, channel awareness arises from dual tokenization and repeated cross-attention between temporal and channel representations. Time tokens query channel tokens, and channel tokens query time tokens. This yields an interactive two-tower Co-Transformer whose per-layer cost is $O(2TCD)$ rather than $O((T^2+C^2)D)$ for a non-interactive self-attentive two-tower alternative. The aggregate layer remains channel-centric: final channel queries attend over final time features, and the output representation is the concatenation of channel-wise embeddings [2203.04298].

Other variants replace full attention by lighter channel operators. CADiT builds a channel correlation map
$$
\mathrm{CCM}=\mathrm{Softmax}(F_S^\top F_T)\in\mathbb{R}^{C\times C},
$$
then reconfigures the student feature map by
$$
F'_S=F_S+F_S\,\mathrm{CCM}.
$$
CaSA in learned image compression uses global average pooling and two fully connected layers to produce a gate
$$
g=\sigma(\mathrm{ReLU}(\mathrm{GAP}(F)W_1)W_2),\qquad \mathrm{CaSA}(F)=F\odot g,
$$
so the model modulates informative channels without constructing full QKV attention across channels [2303.10386] [2408.03842].

Set-based CAT variants are channel-aware in yet another sense. In the EEG seizure-prediction model, a trainable seed attends over a set of channel features, and the resulting attention distribution is accumulated over held-out data to derive per-patient channel importance. In CAT-MoEformer, channel awareness is coupled to context awareness: a gate
$$
w=\mathrm{softmax}!\left(W_g^{(2)}\mathrm{ReLU}(W_g^{(1)}[s_T;\bar v_T])\right)\in\mathbb{R}^4
$$
routes the upper transformer layers to experts corresponding to LOS/NLOS and low/high-speed regimes. In both cases, the channel-aware mechanism changes the computational pathway rather than merely reweighting features [2507.15364] [2605.19997].

3. Communications instantiations: near-field ISAC and beam prediction

The most explicit communications realization is CAT-CENet for near-field XL-MIMO ISAC channel estimation. The setting is a TDD near-field XL-MIMO ISAC system with a ULA of $M=256$ antennas, spacing $d=\lambda/2$, and wavelength $\lambda=0.01$ m. Near-field effects follow the Rayleigh-distance approximation
$$
D_{\mathrm{Ray}}\approx \frac{2D_a^2}{\lambda}\approx \frac{1}{2}M^2\lambda,
$$
and the communication pilot observation satisfies
$$
y=\sqrt{P}\,h_cx+n,\qquad \tilde h_c=\frac{y}{\sqrt P}=h_c+\frac{n}{\sqrt P}.
$$
The near-field channel is modeled as
$$
h_c=\sqrt{\frac{M}{L}}\sum_{l=1}^L g_l a(\phi_l,r_l),
$$
with spherical-wave array response and distance-dependent phase. Radar sensing produces
$$
H_s=\sum_{k=1}^K s_ka(\theta_k,v_k)a(\theta_k,v_k)^H.
$$
The key structural premise is partial overlap between communication scatterers $C$ and sensing targets $S$, with overlap set $D=C\cap S$ and overlap proportion $\eta=X/L$ when $X=|D|$ [2607.03224].

CAT-CENet contains three modules: 2D-convolutional modality preprocessing, two encoders combining multi-head cross-attention and dual-attention, and convolutional post-processing that predicts noise and subtracts it from the LS estimate. The final estimate is
$$
\hat H_c=\tilde H_c-H_9,
$$
and training uses
$$
\mathcal L=\frac{1}{N_{\mathrm{tr}}}\sum_{i=1}^{N_{\mathrm{tr}}}\left|H_c^{(i)}-\hat H_c^{(i)}\right|_F^2.
$$
The reported interpretation is physically specific: when a sensed target is also a communication scatterer, sensing Queries align with pilot Keys, attention weights become large, and the Value features emphasize the corresponding directions and ranges. SHAP-style analysis shows near-zero contributions from non-overlapped sensing tensors and substantial contributions from overlapped ones; for $L=K=3$ and $X=1$, the overlapped sensing tensor contribution rises to $0.3748$, comparable to the pilot contribution. In simulation, CAT-CENet consistently outperforms LS, LMMSE, P-SOMP, MAT-CENet, MIA-GAN, and SBGM. For $L=K=3$ and $\mathrm{SNR}=10$ dB, CAT-CENet with one overlapped target achieves NMSE $\approx 10^{-3}$, whereas the baselines remain $\approx 10^{-2}$. Magnitude-based pruning yields PCAT-CENet variants with approximately unchanged NMSE and reduced non-zero parameters; CAT-CENet has $0.70$M parameters and $28.93$M FLOPs, while the $\kappa=0.5$ and $\kappa=0.8$ pruned versions have $0.64$M and $0.61$M parameters respectively, with the same reported FLOPs [2607.03224].

CAT-MoEformer addresses a different communications task: proactive mmWave beam prediction from compressed uplink pilot observations. The input is a sequence
$$
X\in\mathbb{R}^{T\times 2\times K\times S_w}=\mathbb{R}^{10\times 2\times 60\times 32},
$$
formed by splitting each complex pilot matrix into real and imaginary channels. A three-layer asymmetric CNN with squeeze-and-excitation extracts frequency-beam correlation features directly from pilots, without explicit channel reconstruction. A truncated pretrained GPT-2 backbone models temporal evolution, and the upper three transformer blocks replace the standard FFN by a scene-conditioned MoE-FFN with $E=4$ experts. The gate depends on LOS/NLOS scene label and normalized speed, not on latent hidden states, so routing is sample-level rather than token-level. Training follows a three-stage curriculum: hard expert assignment, isolated gate training under soft routing, and top-1 hard-routing fine-tuning [2605.19997].

On 3GPP TR 38.901 UMa simulations with $64{,}000$ user samples, CAT-MoEformer reports Top-1 beam prediction accuracy of $94.88\%$, Top-3 accuracy of $99.83\%$, beam-switching instant accuracy of $80.62\%$, and inference latency of $0.52$ ms. Relative to a CNN+GPT-2 baseline, the gains are $2.33\%$ in Top-1 accuracy and $9.55\%$ in switching-instant accuracy. Ablations attribute the gains to both the context-conditioned routing and the SE-enhanced spatial encoder: removing MoE, removing context, or removing SE all degrade performance [2605.19997].

Taken together, these communications models define CAT in two complementary ways. CAT-CENet uses explicit physical priors to guide cross-modal alignment in channel estimation, whereas CAT-MoEformer conditions expert specialization on propagation descriptors. In both cases, channel awareness is grounded in the structure of wireless propagation rather than in channel dimension alone.

4. Multivariate sequences and biosignals

In multivariate time series classification, CaT is the encoder within the CaSS self-supervised framework. An input sample is represented as $x\in\mathbb{R}^{C\times T}$, from which the model forms time tokens
$$
e_t=x^TW_t+e_{\mathrm{pos}}\in\mathbb{R}^{T\times D}
$$
and channel tokens
$$
e_c=xW_c\in\mathbb{R}^{C\times D}.
$$
Each Co-Transformer layer contains a Time Layer, where time tokens query channel tokens, and a Channel Layer, where channel tokens query time tokens. The aggregate layer again uses channel queries over final time features, and the representation is
$$
z=[a_c^1,a_c^2,\ldots,a_c^C]\in\mathbb{R}^{1\times (C\cdot D)}.
$$
The encoder is trained jointly with Next Trend Prediction and Contextual Similarity, with total loss
$$
\ell=\alpha_1\ell_{\mathrm{NTP}}+\alpha_2\ell_{\mathrm{CS}},\qquad \alpha_1=2,\ \alpha_2=1.
$$
Reported hyperparameters are $D=512$, $N=8$ layers, $8$ heads, dropout $0.2$, batch size $10$, and temperature $\tau=0.2$ [2203.04298].

Empirically, CaSS with CaT achieves state-of-the-art among the evaluated self-supervised MTS methods. On LSST, it attains $64.94\pm 0.02$ ACC versus $57.24\pm 0.24$ for the best self-supervised baseline W2V K=10, an improvement of about $+7.70\%$. The reported ACC values are $92.57\pm 0.24$ on UCI HAR, $97.07\pm 0.20$ on ArabicDigits, and $98.11\pm 0.27$ on JapaneseVowels. Ablations show that full CaT outperforms a self-aggregate design, channel-only self-attention, and a model without the aggregate layer, supporting the claim that early and repeated cross-attention between time and channel tokens is central to performance [2203.04298].

The EEG seizure-prediction model adopts a different strategy based on Set Transformer pooling. Stage 1 processes each EEG channel independently: band-power features are extracted every $2$ seconds with $50\%$ overlap over a $38$-second window, giving $T=19$ time steps and $\mathrm{dim}_{\mathrm{PSD}}=44$ per step. A temporal kernel serves as the query for attention pooling over these windowed features, producing one temporally aggregated representation per channel. Stage 2, the channel-aware component, treats the resulting $C$ channel features as an unordered set and uses a trainable channel kernel to pool them into a global representation for binary preictal/interictal classification. Channel importance is then computed by accumulating the channel-attention distributions over inference batches, applying a softmax normalization, ranking channels, selecting a patient-specific top-$k$, and retraining on the reduced sensor set [2507.15364].

On the CHB-MIT dataset with $22$ patients and $88$ merged seizures under even division, mean sensitivity before channel selection is $76.4\%$ with false prediction rate $0.09$/hour. After channel selection, dominant channels emerge in $20$ out of $22$ patients, the average number of channels falls from $18$ to $2.8$, and mean sensitivity rises to $80.1\%$ with FPR $0.11$/hour. Under seizure-independent division on seven patients, mean sensitivity is $72.6\%$ both before and after selection, while FPR increases slightly from $0.08$/hour to $0.10$/hour. The model has approximately $37.4$K parameters, about $8.23$M FLOPs per inference, and reported real-time feasibility with total processing of about $33.5$ ms per second of incoming EEG [2507.15364].

These two biosignal and sequence instantiations share a common departure from sequence-centric transformers. CaT explicitly couples time and channel abstractions, while the EEG model discards fixed ordering and treats both temporal windows and channels as sets. In both cases, channel awareness is tied to permutation structure and to the empirical observation that inter-channel dependencies are not well captured by purely temporal encoders.

5. Channel-aware modules in distillation and compression

The nano-drone depth-estimation work introduces CADiT as a training-time distillation module for a compact CNN student, not as a standalone CAT backbone. The teacher is Lite-Mono, the student is a reduced encoder-decoder CNN called DepthNet with about $310$K parameters, and PoseNet is used only during self-supervised training. CADiT reshapes teacher and student feature maps into $N\times C$ matrices, computes the channel correlation map $\mathrm{CCM}=\mathrm{Softmax}(F_S^\top F_T)$, and forms a reweighted student feature map $F'S=F_S+F_S\mathrm{CCM}$. The distillation term is
$$
\mathcal L{\mathrm{CADiT}}=|F'_S-F_T|_2,
$$
and the overall objective combines self-supervised depth loss, CADiT feature distillation, and output distillation [2303.10386].

On grayscale KITTI, the student trained without KD reports AbsRel $0.157$, SqRel $1.259$, RMSE $5.593$, RMSE(log) $0.229$, and $\delta<1.25$ of $0.796$. With KD using CADiT plus output L1, the student improves to AbsRel $0.147$, SqRel $1.149$, RMSE $5.394$, RMSE(log) $0.221$, and $\delta<1.25$ of $0.813$. The deployment emphasis is practical: after ONNX export and $8$-bit post-training quantization, weight storage shrinks from about $747.6$ KB to about $201.3$ KB, and inference on GAP8 reaches $1.24$ FPS at $128\times 160$. The paper is explicit that CADiT runs only during training and introduces no inference overhead on the drone [2303.10386].

In learned image compression, the “Bi-Level Spatial and Channel-aware Transformer” places channel awareness inside the Hybrid Spatial-Channel Attention Transformer Block. The spatial branch, SaSA, splits attention into high-frequency and low-frequency paths using window-based multi-head self-attention and pooled window tokens. The channel-aware branch, CaSA, is a squeeze-excitation-style gate:
$$
g=\sigma(\mathrm{ReLU}(\mathrm{GAP}(F)W_1)W_2),\qquad \mathrm{CaSA}(F)=F\odot g.
$$
MLGFFN then mixes local multi-scale depthwise convolutions and a global pooling branch. The full compression model follows the standard VAE-style LIC pipeline with analysis transform, quantization, hyperprior entropy model, and synthesis transform, optimized under the rate–distortion objective
$$
L=R+\lambda D
$$
for multiple $\lambda$ values [2408.03842].

The Kodak results are reported as superior rate–distortion performance relative to listed traditional and learned codecs, with the most notable gains at mid-to-low bitrates. Component ablations quantify the role of channel awareness: at $\lambda=0.0035$, the full model with HF+LF in SaSA plus CaSA achieves $31.53$ dB, whereas full SaSA without CaSA achieves $31.34$ dB. For MLGFFN at $\lambda=0.005$, the full version reaches $34.59$ dB and MS-SSIM $0.9659$, outperforming variants without the local or global branch. The paper also states that CaSA is not full QKV channel-wise self-attention; it is a lighter gating mechanism integrated into the Transformer block [2408.03842].

These two works show that CAT-like design need not imply channel tokens or cross-channel self-attention. In distillation, channel awareness can mean alignment between student and teacher feature channels. In compression, it can mean channel gating that complements spatial attention and improves decorrelation of latent representations.

6. Empirical patterns, limitations, and open directions

Across domains, CAT mechanisms are most effective when channel structure carries task-specific signal that ordinary attention would otherwise dilute. In CAT-CENet, performance gains increase with overlap proportion between sensing targets and communication scatterers, while with $\eta\approx 0$ the sensing contribution becomes negligible and the estimator reverts to pilot-dominated behavior. In the EEG model, dominant channels emerge for most patients but not all, and channel selection can fail to converge in some cases. In CAT-MoEformer, gains are especially strong at beam-switching instants and in high-mobility regimes, but the routing policy depends on reliable scene labels and speed estimates [2607.03224] [2507.15364] [2605.19997].

The literature also exposes several domain-specific limitations. CAT-CENet models small sensing errors with $\sigma_\theta=10^{-4}$ and $\sigma_v=10^{-2}$, and larger sensing errors or calibration mismatches may reduce cross-attention alignment. The CaT encoder does not use explicit channel positional encoding or channel adjacency priors, and its final representation scales with $C\cdot D$, which can become large for very high channel counts. The EEG work shows that even division can inflate performance relative to seizure-independent division because adjacent train and test segments may share local context. CADiT remains constrained by the severe capacity limit of a tiny student network and inherits known self-supervised depth-estimation failure cases such as transparent surfaces and very close walls. In the learned image compression model, CaSA is intentionally lighter than full channel-token self-attention, which keeps complexity low but limits the richness of inter-channel modeling [2203.04298] [2507.15364] [2303.10386] [2408.03842].

A further misconception is that CAT always implies more expensive computation. Several variants were introduced precisely to control complexity. CaT replaces quadratic time self-attention over $T$ tokens by two $O(TCD)$ cross-attention paths when $T\gg C$. The EEG model avoids pairwise self-attention entirely by using single-query pooling over time and channels. CADiT omits multi-heads, MLP blocks, and layer norms because its purpose is efficient distillation. CAT-MoEformer performs top-1 hard routing at inference so that each MoE layer activates one expert, even though all experts’ weights remain resident in memory [2203.04298] [2507.15364] [2303.10386] [2605.19997].

Several explicit future directions are already present in the surveyed papers. For near-field ISAC, proposed extensions include adaptive attention based on SNR, end-to-end ISAC joint training, and multi-user or multi-antenna extensions incorporating Doppler or elevation. For learned image compression, richer channel-token self-attention and learned frequency splits are identified as possible improvements. For CAT-MoEformer, future work includes latent scene descriptors, hierarchical gating for more regimes, and fusion with additional modalities such as sub-6 GHz or sensors. For nano-drone depth estimation, additional sensing such as sonar is suggested to handle glass and near-wall cases [2607.03224] [2408.03842] [2605.19997] [2303.10386].

Taken together, the literature presents CAT as a recurring strategy for injecting channel structure into Transformer-era models. What remains constant is not a fixed block diagram, but the principle that attention or related operators should respect the semantics of channels, modalities, or propagation regimes. What changes from domain to domain is the mechanism: cross-attention, set pooling, channel correlation, squeeze-excitation gating, or context-conditioned expert routing.