Wavelet as Tokenizer: Preliminary Results on a Shared Wavelet Token Schema for Natural Signals

Abstract: This paper studies whether audio, images, and video can share a common wavelet token schema rather than relying on separate modality-specific latent grids. It introduces a preliminary continuous-token model built around a one-level Haar DWT/IDWT frontend, a shared coefficient-token layout, optional structural metadata, lightweight modality value adapters, and a shared token-wise encoder-decoder trunk. On Speech Commands, EuroSAT RGB, and DAVIS 2017 data, a dense shared model reaches 39.92 dB audio, 29.37 dB image, and 23.93 dB video PSNR. A matched-rate sweep under continuous latent scalar budgets indicates that the visual gains are not explained solely by latent capacity, while also showing that additive metadata embeddings are not a universal source of improvement. Finally, fixed-rate energy selection provides a strong non-parametric baseline: energy_global improves average PSNR over uniform selection by 16.73 dB for audio, 16.90 dB for images, and 15.86 dB for video under compressed keep ratios. Masked sparse training reaches 34.45 dB video PSNR with 50% of dense tokens. The results support a unified wavelet token schema and sparse token interface, while stopping short of establishing a universal discrete vocabulary.

• The paper demonstrates that a unified Haar wavelet token schema enables continuous autoencoding with competitive PSNR improvements across multiple modalities.
• It combines explicit wavelet coefficients with structured metadata to facilitate adaptive sparse token selection using energy-based metrics.
• Experimental results show that modality-specific normalization and selective metadata integration critically impact reconstruction fidelity and token efficiency.

Wavelet as Tokenizer: An Expert Analysis of a Shared Wavelet Token Schema for Natural Signals

Introduction and Motivation

The proliferation of modality-specific tokenizers for generative models—spanning text, audio, images, and video—has led to significant progress in downstream modeling, yet the question remains whether a unified token schema for natural signals is feasible. "Wavelet as Tokenizer: Preliminary Results on a Shared Wavelet Token Schema for Natural Signals" (2606.02631) addresses this by proposing a continuous, structured tokenization scheme based on the Haar Discrete Wavelet Transform (DWT) that is applicable to audio, images, and video. The core innovation is the construction of a token interface incorporating both coefficient values and structured metadata (modality, rank, scale, subband, and spatial/temporal position), processed through a shared encoder-decoder trunk with lightweight modality-specific adapters.

This approach situates itself explicitly between classical transform coding (e.g., JPEG 2000) and recent neural tokenizers (e.g., VQ-VAE, VQGAN, Cosmos), aiming for an architecture- and modality-agnostic token language that could serve as a foundation for multimodal generative models and universal codecs.

Methodology

Shared Wavelet Token Schema

The schema is formulated such that any modality (audio, images, video) is treated as a sampled field $x: \Omega \rightarrow \mathbb{R}^C$, where $\Omega$ spans the relevant domain—time, space, or spacetime. The one-level Haar DWT is applied, yielding $2^d$ subbands for rank-$d$ signals, which are then mapped into a sequential token layout; each token consists of channel values and explicit metadata that encodes the modality and signal-theoretic attributes. This explicit separation enables cross-modal token compatibility and supports adaptive sparse tokenization.

Shared Model Architecture

The continuous autoencoder comprises (1) input value scaling and lightweight adapter layers per modality, (2) learned embeddings of discrete metadata, and (3) a shared MLP trunk (sans attention or token mixing), trained with a mean squared error (MSE) loss in the original signal space. Notably, a fixed audio scaling factor ($s_{\text{audio}}=4$) is used after observing severe degradation in audio reconstruction under naive unit scaling.

Sparse and Adaptive Token Selection

A salient aspect is the evaluation of token importance via coefficient energy ($\ell_2$ norm squared), supporting fixed-rate token pruning for efficient representation. Non-parametric experiments compare energy-based, uniform, random, and lowpass-prioritized selection strategies, directly measuring retained signal fidelity (PSNR) under various token retention ratios.

Masked Sparse Training

The model is further tested via masked autoencoding, where input tokens are randomly or energy-selected and masked, and the network is tasked with full recovery, probing the descriptive completeness and inferential capacity of the shared schema.

Experimental Results

Rate-Distortion Analysis

On Speech Commands, EuroSAT RGB, and DAVIS 2017 datasets, the shared model achieves 39.92 dB (audio), 29.37 dB (image), and 23.93 dB (video) PSNR, closely matching or improving upon separate modality-specific autoencoders when run at comparable continuous token budgets. The benefit is especially marked in images (+6.46 dB PSNR) and video (+3.03 dB) over the separate baselines for the evaluated settings and with proper audio scaling.

The rate-distortion sweep reveals that image and video gains from the shared schema cannot be solely attributed to latent capacity, as controlled experiments match latent scalar budgets across baselines. The impact of explicit metadata is found to be non-uniform; removing additive metadata improves image and video PSNR in many dense settings.

Figure 1: Rate-distortion sweep. The shared schema remains favorable on image and video under matched continuous latent scalar budgets, while audio persists in exhibiting a normalization gap that requires per-modality scaling.

Non-Parametric Sparse Token Selection

Energy-based selection at fixed keep ratios achieves large average PSNR improvements compared to uniform and random selection: +16.73 dB (audio), +16.90 dB (image), and +15.86 dB (video). This demonstrates the cross-modal effectiveness of wavelet coefficient energy as a proxy for token importance.

Figure 2: Non-parametric fixed-rate token selection. Energy-based selection consistently preserves more reconstruction quality than uniform or random fixed-rate token layouts.

Masked Sparse Autoencoding

Sparse masked training with energy-informed selection shows that the shared model can robustly reconstruct signals with as little as 50% token retention, particularly excelling on video. Metadata is beneficial for images in some settings but frequently detrimental for audio and video when keep ratios are low.

Figure 3: Masked sparse shared training. The best video performance appears at 50% keep ratio without additive metadata, highlighting effective sparse inferential capacity.

Analysis and Implications

The empirical findings validate several key claims:

• A unified, wavelet-structured token schema is feasible for 1D audio, 2D image, and 3D video signals under small-scale autoencoding conditions, even when minimizing architecture complexity.
• Energy-based sparse token selection is highly effective and modality-agnostic, providing a simple, interpretable mechanism for cross-modal rate control in the absence of learned allocation.
• Criticality of modality-specific normalization: Audio requires explicit scaling to match the value distribution of visual coefficients, highlighting the importance of cross-modal normalization for decoder convergence and stable shared representations.
• Metadata is not universally beneficial, and may require more sophisticated integration (gating, selective conditioning, etc.) beyond naive additive embeddings.

Practical implications include the potential for developing universal tokenizers or codecs capable of handling diverse natural signals with a single architecture, simplifying system design for multimodal applications. The framework could also support efficient, interpretable sparse tokenization—key for bandwidth- and compute-constrained deployments.

Theoretically, the findings suggest that the signal-processing grammar (scale, location, subband) is a viable intermediate representation for cross-domain tokenization—aligning with classic observations about the structure of natural signals and energy compaction in wavelet bases.

Limitations and Prospective Directions

The study is intentionally prototype-scale: low-resolution data, shallow models, continuous tokens only (no FSQ/VQ/RVQ or entropy modeling), and preliminary statistical breadth (single-run measurements for learned models). The measured PSNRs should not be interpreted as indicative of competitive production performance. True compression efficiency, distributional and perceptual fidelity, and downstream generative modeling capabilities remain untested.

Future work should involve:

• Scaling to production-size datasets, higher resolutions, and comprehensive metric suites (e.g., LPIPS, FID, rFID, SSIM, rFVD).
• Extending the wavelet frontend to multi-level pyramids to expose scale as an active informative token field.
• Integrating advanced token mixing mechanisms (attention, convolutional, SSMs) to capture both local and global contexts.
• Introducing discrete coding (FSQ, RVQ, VQ) and explicit entropy budgets for rigorous rate-distortion benchmarks.
• Expanding to 3D scene representations such as Gaussian Splatting, neural fields, or tri-planar parameterizations.

Conclusion

This work demonstrates that a shared Haar wavelet-based token schema, endowed with explicit scale, subband, and spatial/temporal metadata, can serve as a functional interface for dense and sparse autoencoding across audio, image, and video modalities. Strong cross-modal PSNR improvements are achievable under unified models with appropriately normalized continuous tokens. Energy-based token selection is shown to be a robust, non-parametric baseline for adaptive sparsity.

While a universal discrete vocabulary is not yet established—and improvements from additive metadata are inconsistent—the schema is validated as a sound foundation for ongoing research into unified multimodal tokenization. This opens the door to scalable, universal codecs and generative models capable of handling heterogeneous natural signals via a coherent token interface.