Wavelet-Based Spectral Tokenization

• Wavelet-based spectral tokenization is a method that decomposes multi-dimensional signals using wavelet transforms to create discrete tokens encoding localized frequency, phase, and energy information.
• It leverages a multi-stage pipeline including wavelet decomposition, patchification, and optional DCT enhancements to produce compact, informative representations for machine learning and synthesis.
• The approach exploits structural and algebraic redundancy, such as non-decimated and quaternion wavelet transforms, to improve classification accuracy, generative fidelity, and feature compression.

Wavelet-based spectral tokenization refers to the process of decomposing input data—spanning 1D signals, images, hyperspectral cubes, or fMRI time series—into compact, discrete tokens via multiscale wavelet transforms, optionally augmented by additional spectral decompositions (e.g., DCT) or quantization schemes. These tokens encode multi-resolution frequency, phase, and energy information, providing low-dimensional yet highly informative representations for downstream machine learning, generative modeling, or statistical analysis. This methodology leverages the ability of wavelets to isolate localized transient and structural features while affording both invertibility and stability across a variety of application domains.

1. Mathematical Foundations of Wavelet-Based Tokenization

The foundational mechanism across applications is the discrete wavelet transform (DWT), typically using compactly supported wavelets such as Haar, Daubechies, or even the non-decimated quaternion wavelet transform (NDQWT). Let $x[n]$ denote a real- or vector-valued signal or image, with the DWT yielding a set of approximation and detail coefficients across multiple scales and locations:

$$a_j[k] = \sum_{n} x[n] \varphi_{j,k}[n], \qquad d_j[k] = \sum_{n} x[n] \psi_{j,k}[n]$$

where $\varphi_{j,k}[n]$ are scaled and translated versions of the scaling function, and $\psi_{j,k}[n]$ for the wavelet. In 2D (images), separable filtering yields four downsampled subbands per level: $LL$, $LH$, $HL$, $HH$. In the NDQWT setting, coefficients are quaternion-valued, $$d_{j,k} = \Re(d_{j,k}) + i\,\Im^i(d_{j,k}) + j\,\Im^j(d_{j,k}) + k\,\Im^k(d_{j,k})$$.

Spectral token construction proceeds by forming compact descriptors from these coefficients. For NDQWT, at each level $j$, the modulus, and three phases extracted from Euler-angle decomposition serve as spectral tokens:

$$a_j[k] = \sum_{n} x[n] \varphi_{j,k}[n], \qquad d_j[k] = \sum_{n} x[n] \psi_{j,k}[n]$$0

with $$a_j[k] = \sum_{n} x[n] \varphi_{j,k}[n], \qquad d_j[k] = \sum_{n} x[n] \psi_{j,k}[n]$$1 denoting the empirical log-energy spectrum.

In modern generative pipelines, additional frequency transforms such as blockwise 2D DCT are applied post-wavelet, yielding spectral tokens with enhanced localized energy compaction. This dual-tokenization is fundamental in domains such as fMRI time series synthesis, where physiological plausibility and structured frequency priors are critical (Tew et al., 28 May 2026). Each token is ultimately a structured vector summarizing multi-scale energies and/or phases over a limited spatial, temporal, or spectral support.

2. Tokenization Pipeline Architectures

A standardized tokenization pipeline involves the following stages:

1. Signal decomposition: Apply 1D or 2D DWT, optionally in non-decimated or quaternionic form. For example, the SIT approach arranges the output as $$a_j[k] = \sum_{n} x[n] \varphi_{j,k}[n], \qquad d_j[k] = \sum_{n} x[n] \psi_{j,k}[n]$$2 for $$a_j[k] = \sum_{n} x[n] \varphi_{j,k}[n], \qquad d_j[k] = \sum_{n} x[n] \psi_{j,k}[n]$$3 levels (Esteves et al., 2024).
2. Patchification and splitting: Segment the transformed output into fixed-size spatial, temporal, or frequency-localized patches. In hyperspectral contexts, spatial and spectral tokens are explicitly split before further processing (Ahmad et al., 2024).
3. Spectral enhancement: Optionally apply further linear transforms (e.g., blockwise DCT) to wavelet subbands, as in dual-spectral tokenization for fMRI (Tew et al., 28 May 2026).
4. Embedding and quantization: Each patch is embedded into a vector (via linear projection or neural encoder), optionally quantized to a discrete codebook. The SIT framework maintains a separate codebook per scale.
5. Sequence construction and ordering: Tokens are arranged in a specific order (e.g., coarse-to-fine scale) to facilitate autoregressive modeling or causal prediction.
6. Downstream model ingestion: Tokens serve as inputs to machine learning models (e.g., gradient boosting, SVM, Mamba, transformers), generative decoders, or spectral flow matching modules.

This staged methodology ensures stability, flexibility, and statistical richness of the resulting representation, while maintaining invertibility up to quantization.

3. Key Representational Properties and Redundancy

Two forms of redundancy are central to the effectiveness of wavelet-based spectral tokenization:

• Structural redundancy (non-decimation): In non-decimated (shift-invariant) wavelet transforms, coefficients are not subsampled, providing enhanced shift-invariance, improved slope estimation, and lower-variance phase statistics (Kong et al., 2019).
• Algebraic redundancy (complex/quaternion coefficients): Quaternionic wavelets encode four dimensions per coefficient (modulus and three phases), unlike the real (1D) or complex (modulus+phase) cases. This carries additional directional and texture information not present in the standard DWT (Kong et al., 2019).

In the context of generative or discriminative models, dual-spectral tokens (wavelet+DCT) further decorrelate and compact spectral energy, concentrating relevant physiological or categorical information into fewer, more informative tokens (Tew et al., 28 May 2026).

4. Applications in Machine Learning and Signal Processing

The wavelet-based spectral tokenization paradigm has been deployed in a range of domains:

Classification

• 1D signal and 2D image classification: NDQWT-derived spectral tokens, especially when combining spectrum slopes and quaternion phase statistics, achieve significantly higher accuracy over standard DWT or complex-valued DWT features in audio signal and steel rolling image analysis. For instance, quaternion NDWT slopes and phase averages yielded $$a_j[k] = \sum_{n} x[n] \varphi_{j,k}[n], \qquad d_j[k] = \sum_{n} x[n] \psi_{j,k}[n]$$4 accuracy for 1D sound signals, compared to 38% for standard DWT (Kong et al., 2019).
• Hyperspectral image classification: WaveMamba employs one-level 2D Haar DWT on spatial/spectral tokens derived from HS cubes, achieving performance gains of 2-4% OA over prior CNN, Transformer, and Mamba-based models. Spectral tokenization via DWT enables disentangling of frequency-local texture and context, which is particularly effective in small-sample regimes (Ahmad et al., 2024).

Generative Modeling and Synthesis

• Autoregressive image generation: The Spectral Image Tokenizer (SIT) replaces spatial raster-order tokenization by DWT-based, scale-organized tokens, enabling coarse-to-fine generation, improved conditioning, multiscale reconstruction, and resolution-agnostic upsampling (Esteves et al., 2024).
• fMRI time series synthesis: DSFM converts BOLD signals into DWT and blockwise DCT “dual spectral tokens,” providing a compact, physiologically-informed representation that supports efficient and high-fidelity generation via spectral flow matching (Tew et al., 28 May 2026).

Feature Compression and Transfer

Tokenization in wavelet (and optionally, DCT) domains naturally culls irrelevant high-frequency detail and achieves compressibility due to the rapid decay of energy in higher-frequency components, which is prominent in natural images and physiological signals alike (Esteves et al., 2024, Tew et al., 28 May 2026).

5. Algorithmic and Implementation Details

The following summarizes the algorithmic routines for spectral token extraction and usage:

• NDQWT spectral tokens: Construction involves forming the NDQWT matrix, applying forward transform, extracting modulus and phase features levelwise, and assembling the spectral token vector per sample. These are then used with classical classifiers (e.g., SVM, random forest) (Kong et al., 2019).
• SIT integration: The pipeline consists of DWT decomposition, patchification, transformer-based embedding and quantization, causal sequence ordering, transformer autoregressive decoding, and inverse DWT for reconstruction. Each scale and patch is mapped to a code index and corresponding embedding, with attention masking enforcing causality from low- to high-frequency (Esteves et al., 2024).
• WaveMamba: Following patch extraction, spatial and spectral splitting, DWT enhancement, and concatenation of subbands, tokens are projected and fed into a state-space Mamba sequence model, with final aggregation and classification by a regularized linear head (Ahmad et al., 2024).

Framework	Signal Domain	Frequency Decomposition	Token Construction
NDQWT	1D, 2D	Non-decimated quaternion wavelet	Modulus, 3 phases per level
SIT	Images	Multilevel Haar DWT	Patch + quantized embedding/code
WaveMamba	HSI Cubes	1-level Haar DWT on tokens	Patchwise concatenated subbands
DSFM	fMRI Series	DWT + blockwise 2D DCT	Patchwise frequency vectors

A plausible implication is that coupled redundancy—structural (non-decimation), algebraic (quaternionic), and spectral (wavelet+DCT/quantization)—is the principal driver of invariance, feature richness, and downstream model performance in diverse signal domains.

6. Evaluation and Empirical Impact

Empirical studies consistently demonstrate the efficacy of wavelet-based spectral token representations:

• NDQWT spectral tokens outperformed standard and complex wavelet features across classification tasks, with substantial accuracy increases (e.g., up to 93% in defect detection when combining complex and quaternionic features) (Kong et al., 2019).
• In hyperspectral image classification, WaveMamba with wavelet-enhanced tokens achieves $$a_j[k] = \sum_{n} x[n] \varphi_{j,k}[n], \qquad d_j[k] = \sum_{n} x[n] \psi_{j,k}[n]$$5 higher OA than second-best state-space and transformer architectures on University of Houston and Pavia University datasets (Ahmad et al., 2024).
• SIT outperforms raster-scan tokenizers by automatically compressing high-frequency bands and facilitating efficient autoregressive prediction, multiscale decoding, and flexible upsampling/editing workflows. Key image metrics such as PSNR, LPIPS, FID, and IS consistently improve under SIT (Esteves et al., 2024).
• DSFM’s spectral tokenization captures the multi-scale, non-stationary statistics of physiological fMRI dynamics and yields synthetic data that are statistically indistinguishable from empirical BOLD signals, preserving power-spectral density and domain-specific temporal features (Tew et al., 28 May 2026).

7. Inductive Bias, Interpretability, and Applicability

Wavelet-based spectral tokenization explicitly encodes hierarchical frequency, phase, and local energy information, endowing downstream models with strong priors for multi-scale structure and transient detection. This leads to improved conditioning for autoregressive and flow-based architectures, supports partial reconstruction and upsampling, and provides interpretable axes for human understanding (e.g., scaling indices, phase statistics, or band-specific energy).

The invertibility of the wavelet+DCT tokenization (modulo quantization) ensures lossless or near-lossless recovery of the original signal, valuable for both reconstruction-focused and discriminative contexts. Finally, the approach is generically applicable across one-dimensional, two-dimensional, and spatio-spectral signal domains, limited primarily by available computational resources and the intrinsic multi-scale structure of the source data.