Unified Wavelet Token Schema

• Unified Wavelet Token Schema is a continuous token interface that converts audio, images, and video into shared subband coefficients using a one-level Haar transform.
• It employs a shared token-wise encoder-decoder trunk with modality-specific adapters to reconstruct signals with competitive performance compared to separate baselines.
• The scheme supports both dense and sparse tokenization, using energy-based selection for effective rate control and maintaining high reconstruction quality.

The unified wavelet token schema is a modality-spanning representation proposal in which audio, images, and video are expressed through a common continuous wavelet-token interface rather than through separate modality-specific latent grids. In its primary formulation, a one-level separable Haar transform converts each signal into subband coefficients, which are then flattened into a shared token layout indexed by subband and position, processed by a shared token-wise encoder-decoder trunk, and reconstructed through modality-specific inverse wavelet paths. The central claim is deliberately limited: the scheme supports a unified continuous token grammar across natural signal modalities, but it does not establish a universal discrete vocabulary (Ding, 30 May 2026).

1. Conceptual scope and defining claim

The defining claim of the unified wavelet token schema is that audio, images, and video can share one structural token interface after wavelet analysis. In this view, the token is not a patch token, not a learned modality-specific latent cell, and not a discrete codebook entry. It is a coefficient token indexed by wavelet subband and grid position, with a value vector carrying the local channel coefficients. The proposal is therefore about a shared grammar for continuous coefficient-bearing tokens, not about a shared symbolic vocabulary in the sense used by vector-quantized or residual-quantized generative tokenizers (Ding, 30 May 2026).

This distinction is essential because much of the surrounding literature uses the term “tokenizer” for discrete visual or multimodal codebooks. The unified wavelet token schema instead exposes structure that conventional latent grids typically leave implicit in tensor shape and convolutional geometry: modality, rank, scale, subband, and location are treated as token-level semantics rather than as hidden layout conventions. A plausible implication is that the schema is better understood as an interface specification than as a finalized compression or generation standard.

A related abstraction appears in the wavelet tree and wavelet matrix literature, where the same logical wavelet construction stream can admit different physical layouts distinguished only by node ordering and metadata. In that setting, wavelet tree and wavelet matrix are described as two physical layouts of the same logical wavelet construction stream, with the difference lying in the levelwise concatenation order of node blocks rather than in the underlying routing semantics (Dinklage, 2020). This broader usage of “schema” clarifies that the unified wavelet token schema is fundamentally about representation invariants under different implementations.

2. Formal structure of the token grammar

The formal starting point treats each modality as a sampled field

$x:\Omega \rightarrow \mathbb{R}^C,$

where $\Omega$ is time for audio, a 2D spatial grid for images, and spacetime for video. The frontend is a one-level separable Haar DWT/IDWT. For rank $d \in \{1,2,3\}$, the transform produces $2^d$ subbands, so audio has 2 subbands, images 4, and video 8. The resulting coefficient tensor has shape

$$B \times (C\,2^d) \times g_1 \times \cdots \times g_d,$$

with batch size $B$, channel count $C$, and downsampled grid sizes $g_j$ (Ding, 30 May 2026).

Tokenization is defined by flattening subband-position pairs into a dense sequence. Each token stores the channel coefficient vector, and the token tensor is

$$V \in \mathbb{R}^{B \times N \times C}, \qquad N = 2^d \prod_{j=1}^{d} g_j .$$

Because $N$ is always “number of subbands $\Omega$0 number of downsampled positions,” the layout is structurally identical across modalities. This is the core unification mechanism: 1D audio, 2D images, and 3D video differ in rank and subband count, but not in the abstract token schema (Ding, 30 May 2026).

The grammar is described as coefficient values plus structural metadata,

$\Omega$1

and elsewhere as

$\Omega$2

In the reported experiments, only a one-level transform is used, so $\Omega$3. Subband ids run from $\Omega$4 to $\Omega$5, and positions are normalized coordinates in $\Omega$6. The coordinate convention is always $\Omega$7: audio uses $\Omega$8, images use $\Omega$9, and video uses $d \in \{1,2,3\}$0. The subband order is the low/high channel order produced by the separable Haar implementation, and that order is preserved exactly through flattening and unflattening (Ding, 30 May 2026).

This formalization makes the schema explicitly invertible at the representation level. The paper reports no-model round-trip tests showing that tokenization and scattering add no numerical error beyond floating-point precision when coefficients are passed through unchanged. That result is significant because it establishes that the schema itself is not lossy; any loss arises from the learned encoder-decoder, masking, or coefficient selection rather than from the token format.

3. Shared architecture and modality-specific interfaces

The implementation separates a small modality-specific value adaptation stage from a shared token-wise trunk. What is shared is the token layout, the token interface, and the main encoder-decoder trunk. What is not fully shared is the low-level mapping from raw coefficient values into the common token width: lightweight modality-specific linear adapters are retained because mono audio, RGB images, and RGB video have different channel counts and different coefficient statistics even after Haar analysis (Ding, 30 May 2026).

The shared trunk is intentionally minimal. The reported system uses a LayerNorm-MLP encoder from token width to latent dimension and a decoder that maps latent tokens back to token width. There is no attention, no token mixer, no quantizer, and no codebook. This architectural restriction is deliberate: the aim is to isolate whether the shared schema itself is trainable, rather than to test the strongest possible multimodal autoencoder. After decoding, predicted token values are scattered back into the wavelet coefficient tensor and reconstructed with the corresponding Haar IDWT (Ding, 30 May 2026).

The modality-specific instantiations preserve the same abstract pattern. For audio, the input is a mono waveform of length 16,384, normalized by 32768; a one-level 1D Haar DWT yields two subbands on a half-resolution temporal grid, and each token value is scalar before adaptation because $d \in \{1,2,3\}$1. For images, the input is $d \in \{1,2,3\}$2 RGB; a one-level 2D separable Haar DWT yields four subbands on a $d \in \{1,2,3\}$3 grid, and each token stores a 3-channel coefficient vector. For video, the input is 8 RGB frames at $d \in \{1,2,3\}$4; a one-level 3D separable Haar transform yields eight subbands over a downsampled $d \in \{1,2,3\}$5 grid, and each token again stores a 3-channel coefficient vector (Ding, 30 May 2026).

One implementation issue is especially important: audio required explicit value rescaling to avoid severe underperformance in the shared model. The paper introduces $d \in \{1,2,3\}$6 and reports the best dense shared result at $d \in \{1,2,3\}$7. This is not presented as evidence against the shared schema. Rather, it indicates that cross-modal value-range alignment is a practical precondition for shared processing.

Training uses pure reconstruction in signal space,

$d \in \{1,2,3\}$8

PSNR is computed from signal-space MSE with data range 1.0. The model is trained on Speech Commands v0.02, EuroSAT RGB, and DAVIS 2017 using round-robin modality updates. The default shared configuration is token width 32, latent dimension 16, hidden dimension 64, AdamW with learning rate $d \in \{1,2,3\}$9, batch size 2, and 300 training steps, with validation every 50 steps and averaging over two validation batches per modality (Ding, 30 May 2026).

4. Reconstruction evidence and matched-rate comparisons

The main dense reconstruction experiment provides the primary empirical support for the schema. Separate modality baselines achieve 40.79 dB audio, 22.91 dB image, and 20.91 dB video PSNR. A shared schema model with all value scales set to 1 performs very poorly on audio, at 26.30 dB, but improves image to 30.15 dB and video to 22.53 dB. After the audio scaling fix $2^d$0, the shared model reaches 39.92 dB audio, 29.37 dB image, and 23.93 dB video PSNR. Relative to the separate baseline, this places the shared model within 0.87 dB on audio while exceeding the baseline by 6.46 dB on images and 3.03 dB on video (Ding, 30 May 2026).

The paper places unusual emphasis on rate accounting, but only through proxies because the tokens remain continuous. Dense token counts per sample are 16,384 for audio, 4,096 for images, and 32,768 for video. For dense autoencoders, comparison budgets are defined as continuous latent scalar budgets, namely token count times latent dimension. In matched-rate comparisons, the separate baseline’s latent channels are set to $2^d$1 times the shared latent dimension for a modality of rank $2^d$2, so total latent scalar counts are matched (Ding, 30 May 2026).

Under this matched-rate sweep at shared latent dimension 16, the separate matched model scores 43.93 dB audio, 22.47 dB image, and 21.76 dB video. The shared model with full metadata gives 39.92, 29.37, and 23.93 dB, while the shared model with no metadata gives 41.29, 30.41, and 29.42 dB. Two conclusions follow directly from these numbers. First, the visual gains of the shared schema are not explained solely by latent scalar capacity, because image and video remain far better than the matched separate baseline. Second, additive metadata embeddings are not a universal source of improvement: in this dense setting, removing them improves image and video substantially (Ding, 30 May 2026).

The audio scaling ablation reinforces the same point. At 150 steps, unit scale gives 27.56 dB audio; scaling by 2 gives 30.36 dB; scaling by 4 gives 30.81 dB; scaling by 8 falls back to 28.95 dB. Image and video remain roughly stable across these settings. This suggests that the shared schema is structurally viable, but that cross-modal normalization can dominate performance when coefficient statistics differ sharply.

5. Sparse token interface and rate proxies

The schema is also evaluated as a sparse token interface. The nonparametric fixed-rate selector is based on token energy. For token value $2^d$3, token energy is

$2^d$4

Given a keep ratio $2^d$5, a boolean mask $2^d$6 is constructed and dropped tokens are zeroed: $2^d$7 These masked values are scattered back into the wavelet tensor and decoded directly through Haar IDWT (Ding, 30 May 2026).

The paper compares global energy top-$2^d$8, per-subband energy top-$2^d$9, uniform stride, random, and lowpass-first selection. Over keep ratios $$B \times (C\,2^d) \times g_1 \times \cdots \times g_d,$$0, energy_global improves average PSNR over uniform selection by 16.73 dB for audio, 16.90 dB for images, and 15.86 dB for video. It also outperforms random by nearly the same amounts. The paper further notes that per-subband energy selection helps audio but is much weaker for image and video, while lowpass-first nearly matches energy_global for images, is close but weaker for video, and is substantially worse for audio (Ding, 30 May 2026).

These results are important because they show that wavelet coefficient energy functions as a cross-modal importance signal for token retention. In other words, the schema does not merely unify token layout; it also admits a modality-agnostic ranking criterion for sparse selection under compressed keep ratios. That property is especially relevant for any future rate-controlled or entropy-coded extension.

The second sparse setting is masked sparse training. Here the exact same dense token layout is retained, but masked token values are provided as input while the target remains full signal reconstruction. The masking configuration uses energy_global selection, keep ratios $$B \times (C\,2^d) \times g_1 \times \cdots \times g_d,$$1, latent dimensions 8 and 16, and metadata modes none and full. The best sparse shared settings are 32.60 dB for audio with metadata none, latent 8, keep ratio 0.50; 29.98 dB for image with metadata full, latent 8, keep ratio 0.50; and 34.45 dB for video with metadata none, latent 16, keep ratio 0.50 (Ding, 30 May 2026).

The video result is the most striking because 34.45 dB at 50% dense tokens substantially exceeds the dense shared video baseline of 23.93 dB. The paper treats this as evidence that sparsity may regularize the task for the simple token-wise model. A plausible implication is that, for at least some modalities, a sparse coefficient interface is not merely a compression device but also a representational bias that can improve reconstruction.

6. Relation to adjacent wavelet tokenization paradigms

The unified wavelet token schema belongs to a broader family of wavelet-based tokenization ideas, but it occupies a specific position within that family. It is multimodal rather than image-only, continuous rather than discrete, coefficient-centric rather than patch-centric, and interface-oriented rather than codebook-oriented.

“Spectral Image Tokenizer” defines an image-only discrete tokenizer based on a 2D Haar DWT, per-scale patchification, equal token count per scale, separate codebooks per scale, and coarse-to-fine ordering with scale-causal attention. Its autoregressive model consumes discrete code indices ordered first by scale and then by token index, so it is a coarse-to-fine spectral codebook system rather than a shared continuous coefficient interface (Esteves et al., 2024).

“Wavelet-Based Image Tokenizer for Vision Transformers” likewise remains image-specific. It converts RGB to $$B \times (C\,2^d) \times g_1 \times \cdots \times g_d,$$2, downsamples chroma, applies a multilevel 2D DWT per channel, aggregates coefficients associated with each coarse-grid cell into a local multiscale vector $$B \times (C\,2^d) \times g_1 \times \cdots \times g_d,$$3, zeros 80% of coefficients by default in experiments, and then projects the result into standard ViT token space. This produces localized multiscale frequency descriptors on a uniform coarse grid, but not a modality-shared schema across 1D, 2D, and 3D signals (Zhu et al., 2024).

“Wavelets Are All You Need for Autoregressive Image Generation” takes a different route again: its token is an event in an embedded wavelet coding process rather than a coefficient vector. The vocabulary has exactly 7 symbols—Group4x4, Group2x2, NowSignificantNeg, NowSignificantPos, Insignificant, NextAccuracy0, and NextAccuracy1—and the sequence is bitplane-first globally, coarse-to-fine across scales within each bitplane, and stateful because token meaning depends on decoder-side significance and refinement state. The paper also notes that low-resolution scaling coefficients need not be tokenized and may instead be supplied separately, which makes the representation not fully self-contained as a token stream (Mattar et al., 2024).

“WaveMix” is closer to a structured coefficient field than to a sequence tokenizer. It uses a multi-scale 2D DWT for spatial token mixing, keeps representation in 2D tensors indexed by scale, subband, spatial position, and channel, and avoids sequence flattening altogether. Its natural atomic unit is a coefficient indexed by $$B \times (C\,2^d) \times g_1 \times \cdots \times g_d,$$4, or a grouped tuple of subband coefficients, which suggests a hierarchical token field rather than an autoregressive token language (Jeevan et al., 2022).

Taken together, these lines of work indicate that “wavelet tokenization” is not a single design pattern. It may mean a continuous shared coefficient interface, a discrete multiscale codebook, an event-based significance language, a coarse-grid multiscale descriptor for transformer backbones, or a structured feature pyramid for token mixing. The unified wavelet token schema is specifically the first of these.

7. Limitations, misconceptions, and prospective extensions

A common misconception is that the unified wavelet token schema establishes a universal discrete tokenizer across natural signals. The paper explicitly rejects that interpretation. Tokens are continuous throughout; there is no VQ, RVQ, FSQ, entropy model, code utilization analysis, dead-code analysis, or bits-per-sample/pixel estimate. The supported claim is narrower: audio, image, and video can share one wavelet-structured continuous token grammar (Ding, 30 May 2026).

Other limitations are equally clear. The frontend is restricted to one-level Haar only, so scale is a dummy field rather than an active multiscale variable. The latent representation is continuous and scalar-valued. The trunk is a token-wise MLP with no attention and no spatial or temporal mixing. Evaluation is reconstruction-centric and limited to MSE/PSNR on relatively small datasets and low resolutions. Reported rate measures are token counts, latent scalar counts, and keep ratios rather than true entropy-coded bitrates. Results are mostly single-run rather than multi-seed statistics (Ding, 30 May 2026).

The status of metadata is also unresolved. Structural metadata is conceptually central to the schema, but the tested implementation uses simple additive embeddings, and those embeddings do not consistently help. In the matched-rate dense setting, removing them improves image and video substantially. This does not invalidate metadata as a concept, but it does show that the particular embedding mechanism should not be treated as established.

The paper identifies several natural extensions: multi-level wavelet pyramids, where scale would become a genuine token field rather than a placeholder; stronger architectures; learned sparse rate control; FSQ or VQ/RVQ discretization; and extensions of the same field-based view beyond video tensors to continuous 3D representations. These directions indicate that the schema is best interpreted as a starting point for unifying wavelet-structured interfaces across modalities, not as a finished universal tokenizer (Ding, 30 May 2026).