Channel-Wise Tokenization in AI

• Channel-wise tokenization is a strategy that treats the channel axis as explicit tokens, enabling reparameterization via invertible transforms and discrete quantization.
• It is applied in domains such as image compression, EEG, and speech to create hierarchical representations and enhance rate–distortion and reconstruction performance.
• Techniques like stochastic tail-dropping, channel dropout, and independent quantization help reduce cross-channel entanglement and enforce coarse-to-fine token hierarchies.

Searching arXiv for the cited papers and closely related work on channel-wise tokenization. Using arXiv search to verify the main papers on channel-wise tokenization across vision, compression, EEG, speech, and multimodal modeling. Channel-wise tokenization denotes a family of representation strategies in which the channel axis, rather than only spatial patches or temporal windows, becomes an explicit unit of modeling. Across recent work, this idea appears in several technically distinct forms: as an invertible basis change over feature channels before attention and entropy coding in learned image compression, as direct quantization of latent channels for image and video generation, as per-electrode or per-channel tokenization in EEG, as discretization of mel-filterbank channels in speech, and as channel-group processing in multimodal fusion and compact vision backbones (Fu et al., 27 May 2026, Song et al., 25 May 2026, Qing et al., 2 Jun 2026, Bai et al., 2024). The common premise is that channel structure is not merely an implementation detail of hidden representations; it can itself define the token space on which attention, quantization, prediction, and decoding operate.

1. Definitions and representational forms

In the narrowest sense, channel-wise tokenization replaces or complements spatial tokenization by treating channels, channel groups, or transformed channel coordinates as the primary atomic units. The exact meaning depends on the representation. In learned image compression, the spatial tokenization can remain unchanged while the channel coordinate system is reparameterized through an invertible wavelet transform, so that attention and entropy models operate on channel subbands rather than raw channels (Fu et al., 27 May 2026). In visual quantization, an image can instead be represented as a one-dimensional sequence of latent channels, each channel carrying a full spatial pattern; this is the formulation adopted by Channel-wise Vector Quantization (CVQ) and by ChannelTok (Song et al., 25 May 2026, Paul et al., 3 Jun 2026). In biosignals, channel-wise tokenization can mean that each EEG electrode is treated as a first-class token, or that a single EEG channel is discretized independently into a motif sequence before any cross-channel modeling (Qing et al., 2 Jun 2026, Pradeepkumar et al., 22 Feb 2025). In speech, the same logic appears when mel-filterbank channels are discretized independently into intensity bins at each frame (Bai et al., 2024).

Work	Token unit	Channel-wise mechanism
ChWDTA / ChWP	Transformed channel coordinates	Wavelet or wavelet-packet transform along channels (Fu et al., 27 May 2026)
CVQ	Entire latent channel	Nearest-neighbor quantization of $h \times w \times 1$ channel maps (Song et al., 25 May 2026)
ChannelTok	Entire latent channel	Per-channel quantization plus stochastic tail-dropping (Paul et al., 3 Jun 2026)
EEG-to-music reconstruction	Electrode-specific temporal patches	Each electrode remains a separate token stream before attention (Qing et al., 2 Jun 2026)
dMel	Mel-channel intensities per frame	Scalar discretization of each mel channel into bins (Bai et al., 2024)

This diversity suggests that “channel token” is not a single architectural primitive but a general representational stance. A channel token may be an invertible basis coordinate, a quantized latent slice, a sensor stream, or a grouped feature subspace. What unifies these cases is that the channel axis is made structurally explicit rather than being mixed immediately by dense linear layers, convolutions, or spatially organized tokenizers.

2. Transform-domain channel tokenization in learned compression

A particularly explicit formulation appears in learned image compression, where channel-wise tokenization is implemented as a structured, invertible basis change over channels. In "ChWDTA: Channel-wise Wavelet-Domain Transformer Attention and Entropy Modeling for Learned Image Compression" (Fu et al., 27 May 2026), the codec follows a standard variational LIC framework with analysis transform $g_a$, synthesis transform $g_s$, hyperprior $h_a,h_s$, and channel-wise autoregressive modeling over latent slices. The contribution is to insert channel-wise wavelet transforms into both the transformer blocks and the entropy model.

For a feature tensor $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$, the channel-wise transform $\mathrm{WT}_c$ is a one-dimensional wavelet transform along the channel axis. At each spatial site, a channel vector $f_{hw} \in \mathbb{R}^{C}$ is mapped to

$u_{hw} = W f_{hw},$

where $W \in \mathbb{R}^{C \times C}$ is invertible and implemented via lifting. The transformed channels are arranged as smooth and detail branches, concatenated, passed through standard windowed spatial multi-head self-attention, and then mapped back by $\mathrm{IWT}_c$. Spatial tokenization remains Swin-style windowed attention; only the channel representation of each spatial token changes. In kernel terms, the pre-softmax attention score changes from $g_a$0 to $g_a$1, so the wavelet basis changes the attention kernel rather than acting as a preprocessing heuristic (Fu et al., 27 May 2026).

The stated motivation is covariance sparsification. The paper defines a cross-branch coupling ratio

$g_a$2

for the block decomposition of channel covariance in the transformed basis. A well-chosen wavelet basis can reduce this ratio, and the paper reports that $g_a$3 can drop by approximately $g_a$4, indicating that wavelet-domain channel tokens are less entangled than raw channels. The lifting parameters $g_a$5 are initialized from CDF 9/7 and can be learned, so the basis is structured and invertible rather than unconstrained (Fu et al., 27 May 2026).

The same work extends channel-wise tokenization into entropy coding through a two-level channel-wise wavelet packet decomposition: $g_a$6 producing four equal-sized channel subbands. These subbands are then used to define slice-based autoregressive entropy modeling, either as eight slices by splitting each subband in two or as four slices by coding each subband as a single slice. With ChWDTB and the default 8-slice ChWP configuration, the scheme reports BD-rate reductions of $g_a$7, $g_a$8, and $g_a$9 on Kodak, CLIC Professional Validation, and Tecnick, respectively. Even the lighter 4-slice configuration retains most gains while reducing kMACs/pixel from $g_s$0 to $g_s$1, parameters from $g_s$2M to $g_s$3M, and latency from $g_s$4 ms to $g_s$5 ms (Fu et al., 27 May 2026).

This formulation is distinctive because it does not create new spatial tokens. Instead, it reparameterizes the channel dimension into structured subbands and lets both attention and entropy models operate in that transformed channel-token space. The resulting notion of tokenization is therefore basis-theoretic rather than patch-theoretic.

3. Quantized channel tokens in image and video generation

A second line of work makes channels themselves the quantized discrete tokens. In CVQ, an encoder maps an image to $g_s$6, but instead of quantizing the $g_s$7 spatial positions, it quantizes each channel slice $g_s$8 using a global channel-wise codebook $g_s$9, with

$h_a,h_s$0

This turns an image into a one-dimensional sequence of length $h_a,h_s$1, one token per channel, rather than a two-dimensional grid of patch tokens (Song et al., 25 May 2026). The paper argues empirically that different channels carry different levels of visual information and uses nested channel dropout to impose an order in which early channels must support reconstruction when later channels are zeroed. With $h_a,h_s$2, this yields a coarse-to-fine hierarchy: rFID improves from $h_a,h_s$3 at 32 channels to $h_a,h_s$4 at 256 channels, with corresponding gains in SSIM and PSNR. Under matched 256-token settings on ImageNet-1K, CVQ reports $h_a,h_s$5 codebook utilization at codebook size $h_a,h_s$6, compared with $h_a,h_s$7 for reproduced vanilla VQ, and reaches rFID $h_a,h_s$8 versus $h_a,h_s$9 for that VQ baseline. At $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$0 entries, CVQ still reports $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$1 utilization (Song et al., 25 May 2026).

The same paper couples channel-wise tokenization to Channel-wise Autoregressive modeling: $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$2 framed as “next-channel prediction.” The reported text-to-image results include GenEval overall $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$3 and DPG overall $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$4 for the 8B CAR model, with the claim that the channel sequence is more aligned with coarse-to-fine semantics than raster-ordered patch tokens (Song et al., 25 May 2026).

ChannelTok adopts a closely related but flexible-length formulation. Its encoder produces $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$5, and the $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$6-th latent channel $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$7 is treated as a visual token (Paul et al., 3 Jun 2026). The crucial training device is stochastic tail-dropping: with probability $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$8, a retention ratio $$\mathbf{F} \in \mathbb{R}^{B \times C \times H' \times W'}$$9 is sampled, the first $\mathrm{WT}_c$0 channels are kept active, and later channels are passed forward with stop-gradient. This forces early channels to carry globally important information and later channels to store refinements. At inference, flexible compression is obtained by simply retaining the first $\mathrm{WT}_c$1 channels. On ImageNet, the model reports rFID $\mathrm{WT}_c$2 at 256 tokens and $\mathrm{WT}_c$3 at 512 tokens, while being $\mathrm{WT}_c$4 faster in decoding and $\mathrm{WT}_c$5 smaller than the next-best alternative; the total parameter count is $\mathrm{WT}_c$6M (Paul et al., 3 Jun 2026). For variable-length autoregressive generation, the paper serializes channel codes and applies a position-weighted cross-entropy loss with

$\mathrm{WT}_c$7

so that earlier channels receive larger loss weight. With 32 generated tokens, the reported speedup is $\mathrm{WT}_c$8 relative to full 256-token generation (Paul et al., 3 Jun 2026).

In video tokenization, channel-wise quantization appears in a different form as channel-split quantization. MambaVideo encodes a video to a latent with $\mathrm{WT}_c$9 channels, splits them into $f_{hw} \in \mathbb{R}^{C}$0 groups $f_{hw} \in \mathbb{R}^{C}$1, quantizes each group independently with LFQ or FSQ, and compensates by increasing spatiotemporal compression so that total token count remains unchanged (Argaw et al., 6 Jul 2025). The effective sequence length stays

$f_{hw} \in \mathbb{R}^{C}$2

but each site is represented by an ordered $f_{hw} \in \mathbb{R}^{C}$3-tuple of channel tokens rather than one token. The paper argues that this yields an effective codebook capacity exceeding $f_{hw} \in \mathbb{R}^{C}$4 without increasing token count. Empirically, MambaVideo with CS-FSQ improves reconstruction over its non-channel-split counterpart, for example from $f_{hw} \in \mathbb{R}^{C}$5 dB to $f_{hw} \in \mathbb{R}^{C}$6 dB on Xiph-2K and from $f_{hw} \in \mathbb{R}^{C}$7 dB to $f_{hw} \in \mathbb{R}^{C}$8 dB on DAVIS for the authors’ tokenizer, and improves downstream VideoGPT FVD to $f_{hw} \in \mathbb{R}^{C}$9 on SkyTimelapse and $u_{hw} = W f_{hw},$0 on UCF-101 (Argaw et al., 6 Jul 2025).

Taken together, these works define a coherent quantized view of channel-wise tokenization: channels can be the discrete sequence themselves, can be retained as a prefix for flexible-rate decoding, or can be split into multiple independently quantized channel groups at fixed token budgets.

4. Channel-wise tokenization in EEG and speech

In biosignals, channel-wise tokenization is motivated less by compression than by preservation of weak, distributed, and channel-sensitive structure. In EEG-to-music reconstruction, the central claim is that early channel mixing destroys weak but discriminative EEG signals (Qing et al., 2 Jun 2026). The proposed encoder therefore treats each electrode as a first-class token. For an EEG window $u_{hw} = W f_{hw},$1, each channel $u_{hw} = W f_{hw},$2 is partitioned into one-dimensional temporal patches of size $u_{hw} = W f_{hw},$3, embedded to $u_{hw} = W f_{hw},$4, and concatenated into a token sequence of length $u_{hw} = W f_{hw},$5 including a CLS token. The transformer then attends jointly across channel and time tokens. This per-channel tokenization is coupled to multi-view self-distillation over temporal crops and random channel subsets, plus structured channel dropout at rate approximately $u_{hw} = W f_{hw},$6. The empirical effect is large: replacing channel-wise tokenization with block tokenization over groups of five contiguous electrodes drops 50-way identification from $u_{hw} = W f_{hw},$7 to $u_{hw} = W f_{hw},$8 and 14-way song identification from $u_{hw} = W f_{hw},$9 to $W \in \mathbb{R}^{C \times C}$0. The full channel-oriented model reports CLAP score $W \in \mathbb{R}^{C \times C}$1, 50-way identification $W \in \mathbb{R}^{C \times C}$2, 14-way song identification $W \in \mathbb{R}^{C \times C}$3, and 10-way genre classification $W \in \mathbb{R}^{C \times C}$4 on NMED-T + NMED-H (Qing et al., 2 Jun 2026).

A second EEG line operates at the opposite granularity: single-channel tokenization. TFM-Tokenizer assumes that critical time-frequency features can be effectively captured from a single channel and therefore learns a shared vocabulary that is applied independently to each EEG channel (Pradeepkumar et al., 22 Feb 2025). For a single channel $W \in \mathbb{R}^{C \times C}$5, the method computes an STFT spectrogram using $W \in \mathbb{R}^{C \times C}$6, hop $W \in \mathbb{R}^{C \times C}$7, Hann window, and one-sided magnitude spectrum, alongside aligned raw temporal patches. A frequency path and a temporal path are fused, passed through a temporal transformer, and vector-quantized with codebook size $W \in \mathbb{R}^{C \times C}$8 and embedding dimension $W \in \mathbb{R}^{C \times C}$9. At inference, each channel of a multichannel recording is tokenized independently, yielding per-channel token sequences that are then combined by TFM-Encoder. On TUEV, the full model reports balanced accuracy $\mathrm{IWT}_c$0 and Cohen’s Kappa $\mathrm{IWT}_c$1; on TUAB, balanced accuracy $\mathrm{IWT}_c$2, AUC-PR $\mathrm{IWT}_c$3, and AUROC $\mathrm{IWT}_c$4; on IIIC, Kappa $\mathrm{IWT}_c$5 and F1 $\mathrm{IWT}_c$6 (Pradeepkumar et al., 22 Feb 2025). The paper further reports lower token utilization but higher class-token uniqueness than LaBraM’s neural tokenizer, with TUEV class-token uniqueness increasing from $\mathrm{IWT}_c$7 to $\mathrm{IWT}_c$8.

Speech offers a third instantiation, this time with fixed, training-free channel quantization. dMel discretizes mel-filterbank channels into intensity bins. For a mel representation $\mathrm{IWT}_c$9 with $g_a$00 channels, a global codebook of $g_a$01 linearly spaced bins is defined from dataset-wide minimum and maximum values, and each mel value is discretized independently: $g_a$02 A frame token is thus the vector $g_a$03, with 80 channel-wise 4-bit symbols (Bai et al., 2024). The model embeds each channel value independently, concatenates the embeddings, and linearly projects them to a single transformer embedding per frame; all frequency channels at time frame $g_a$04 are predicted independently and in parallel. Main experiments use 40 Hz frame rate. With this representation, RichTTS reports WER $g_a$05 and CER $g_a$06 on LibriSpeech, while RichASR reports test-clean $g_a$07 and test-other $g_a$08, outperforming corresponding HuBERT-KM- and SpeechTokenizer-based variants within the same architecture (Bai et al., 2024).

Across these biosignal and speech settings, channel-wise tokenization serves three related purposes: it preserves sensor or frequency identity, enables explicit control over missing or noisy channels, and postpones cross-channel fusion to later sequence models rather than entangling channels at the tokenizer stem.

5. Channel-group tokenization as a broader architectural pattern

Not all channel-wise tokenization work creates explicit discrete tokens. Some papers treat groups of channels as token-like subspaces inside larger architectures. CMFusion, for multimodal hate video detection, processes video, audio, and text features and introduces a channel-wise fusion module in which the last feature dimension $g_a$09 is partitioned into $g_a$10 heads; each head operates on its own $g_a$11-dimensional chunk, and the transformed chunks are concatenated and refined by an output linear layer (Zhang et al., 17 May 2025). The paper notes that this can be reinterpreted as a form of channel-wise tokenization: the feature vector is split into channel groups, each group is processed independently, and interaction occurs only after concatenation. In the associated modality-wise fusion, gating is explicitly channel-wise through $g_a$12. On the HateMM dataset, the full model reports accuracy $g_a$13, F1 $g_a$14, precision $g_a$15, and recall $g_a$16, while the ablation without the full CMFusion design performs worse (Zhang et al., 17 May 2025).

An earlier vision example is RecNets, where the channel dimension is split into $g_a$17 disjoint segments and processed recurrently: $g_a$18 Here each segment acts as a channel token, and the recurrence imposes an ordered dependency graph over those tokens (Retsinas et al., 2019). The paper shows the parameter savings explicitly: a standard $g_a$19 convolution from 160 to 640 channels requires $g_a$20 parameters, whereas CRC with $g_a$21 requires $g_a$22. The preferred nonlinearity is separate BN + ReLU per recurrent step, and the full RecNet family demonstrates a strong size–accuracy trade-off on CIFAR-10 and CIFAR-100; for example, RecNet-120-2880 reports $g_a$23 on CIFAR-100 with $g_a$24M parameters (Retsinas et al., 2019).

These architectures broaden the concept of channel-wise tokenization beyond compression and discrete modeling. A “channel token” can also be a contiguous channel group or head-sized subspace on which a shared operator acts recurrently or independently. The general pattern is factorization of the feature dimension into explicit subunits, delayed mixing, and parameter sharing across those subunits.

6. Limitations, evaluation, and extensions

The literature identifies several recurring limitations. First, channel ordering is often not intrinsic and must be induced. CVQ relies on nested channel dropout to force low-index channels to carry global information; without this ordering, autoregressive performance degrades from GenEval $g_a$25 to $g_a$26 and DPG from $g_a$27 to $g_a$28, while reconstruction remains nearly unchanged (Song et al., 25 May 2026). ChannelTok likewise depends on stochastic tail-dropping; the baseline without channel masking does not exhibit the same coarse-to-fine hierarchy (Paul et al., 3 Jun 2026). Second, channel tokens may be globally entangled in space. CVQ notes that each channel-token encodes a global spatial map, which can make fine-grained local variation more dependent on cross-channel interactions (Song et al., 25 May 2026). Third, explicit preservation of channels can be computationally expensive. The EEG-to-music model processes roughly $g_a$29 tokens plus CLS per 8-second window at 125 channels and patch size 50, and the paper notes computational overhead as a limitation (Qing et al., 2 Jun 2026). Fourth, extreme compression can dominate channel-wise capacity gains: MambaVideo reports that as $g_a$30 increases under very high spatiotemporal compression, the benefits of channel-split quantization saturate or disappear (Argaw et al., 6 Jul 2025).

A separate limitation concerns evaluation. "Two Counterexamples to Tokenization and the Noiseless Channel" (Cognetta et al., 2024) shows that Rényi efficiency of the unigram distribution can be increased while downstream performance decreases. RANDOM-DROP BPE and DUPLICATION BPE both provide constructions in which intrinsic tokenization metrics improve but BLEU degrades, demonstrating that unigram-based intrinsic measures can be gamed. Although this work is not about channel-wise tokenization in the architectural sense, it is directly relevant to tokenization research: it shows that any attempt to evaluate tokenizers solely through symbol-distribution balance, without sequence length, structural coherence, or model interaction, is fragile (Cognetta et al., 2024). This suggests that channel-wise tokenization should be evaluated primarily through downstream reconstruction, generation, alignment, and robustness metrics rather than by channel-distribution heuristics alone.

Several papers also point toward broader axis-wise generalizations. TivTok is a temporal, not channel-wise, factorization, but it offers a reusable design principle: role separation can be induced architecturally by giving different token groups different attention scopes, and shared components can be broadcast across frames and chunks (Chen et al., 16 Jun 2026). The paper explicitly frames this as an instance of axis-wise factorization and argues that similar scoped latent groups and broadcasting mechanisms could be transferred to channel-wise designs. Related future directions stated across the corpus include learned orthogonal or DCT-like transforms over channels, adaptive per-image channel budgets, graph priors for variable EEG montages, channel-wise tokenization for video and multimodal generation, and explicit joint temporal $g_a$31 channel tokenization (Fu et al., 27 May 2026, Paul et al., 3 Jun 2026, Qing et al., 2 Jun 2026, Pradeepkumar et al., 22 Feb 2025).

The cumulative picture is that channel-wise tokenization is neither a single method nor a narrow vision-specific trick. It is an axis-selection principle: channels may be reparameterized, discretized, ordered, split, recurrently processed, or preserved as separate sensor streams. Where spatial tokenization exposes locality, channel-wise tokenization exposes basis structure, latent detail hierarchy, or sensor identity. Recent work shows that this can improve rate–distortion performance, codebook utilization, flexible-length generation, multimodal fusion, and biosignal alignment, but only when the channel axis is given an explicit semantics through transforms, masking, ordering, or architectural constraints (Fu et al., 27 May 2026, Song et al., 25 May 2026, Paul et al., 3 Jun 2026, Qing et al., 2 Jun 2026).