Time-Frequency Tokenization

• Time-frequency tokenization transforms continuous signals into discrete, localized tokens that capture essential spectral and temporal features.
• Modern methods combine mathematically principled dictionaries, adaptive thresholding, and multi-resolution analysis to achieve high-fidelity compression and robust signal reconstruction.
• Hierarchical and vector quantization techniques ensure explicit time-frequency locality and noise robustness across applications like EEG, audio synthesis, and communications.

Time-frequency tokenization is the process of transforming temporal signals into discrete, localized representations in both time and frequency, yielding a compact, structured vocabulary that preserves the essential spectral–temporal content. This approach underpins a wide spectrum of contemporary representation learning systems, signal analysis pipelines, and domain-specific neural architectures, providing a foundation for compression, feature learning, and generative modeling across neuroscience, audio, and communications. Recent advances integrate mathematically principled dictionaries, data-driven codebooks, and hierarchical quantization mechanisms to create tokens that capture complex oscillatory, multiscale, and nonstationary structures present in real-world temporal data.

1. Theoretical Foundations of Time–Frequency Tokenization

Time–frequency tokenization generalizes classical signal expansions by discretizing both temporal and spectral content. At the core are two mathematical traditions:

• Frames and localized atoms: Building on Gabor- and wavelet-inspired frames, arbitrary finite-energy signals $x(t)$ are expanded in a dictionary $\{g_{m,n}(t)\}$, where each "atom" is exponentially localized around specific times and frequencies (Muzhikyan et al., 2011). Given lattice spacings $a = b = \sqrt{\pi}$, the tokens

$c_{m,n} = \langle x, g_{m,n} \rangle$

serve as the time–frequency coefficients, and the original signal is reconstructed (exactly in $L^2$) from these tokens and the atoms.

• Time–frequency filtering and Schmidt modes: In classical and quantum optical communications, sequential applications of spectral and temporal filters lead to integral operators whose Schmidt (singular value) decompositions yield near-orthogonal temporal modes. Each mode coefficient constitutes a token, providing optimal discrimination and compactification of the information content, subject to physical efficiency–discrimination tradeoffs (Raymer et al., 2020).

These theoretical templates guarantee—under appropriate localization and completeness conditions—that time–frequency tokenization is invertible, robust, and sharply resolves local spectral features.

2. Algorithmic Frameworks and Model Architectures

Modern time–frequency tokenizers encompass both handcrafted and learned pipelines. Representative strategies include:

• STFT-based masking and decomposition: Time–frequency mode decomposition (TFMD) leverages the STFT to compute a spectrogram, employs local smoothing and adaptive thresholding to segment high-energy regions, and applies image-processing techniques (connected-component labeling, size filtering) to extract contiguous masks representing signal modes (Zhou et al., 16 Jul 2025). Each mode corresponds to a binary time–frequency tokenization, and reconstructed via ISTFT.
• Wavelet-domain quantization: WaveToken (Masserano et al., 2024) implements a discrete wavelet transform (DWT) to decompose signals into approximation and detail coefficients, applies optional thresholding for sparsity, and then quantizes coefficients into a compact codebook (e.g., 1024 bins via Freedman–Diaconis binning). Token sequences directly encode localized frequency and time scales, supporting autoregressive forecasting.
• Codebook-driven quantization: Transformers for EEG and audio often use vector quantization (VQ) or residual vector quantization (RVQ) to map short (e.g., 1 s) patches or 2D spectrogram patches into discrete tokens (Barmpas et al., 15 Oct 2025, Pradeepkumar et al., 22 Feb 2025, Cheng et al., 15 May 2026). The codebooks are trained (sometimes via exponential moving average updates) to optimize reconstruction loss (MSE, spectral divergence, adversarial objectives).
• Data-driven time–frequency tessellation: Algorithms such as TETRIS partition the time–frequency plane adaptively, guided by estimated instantaneous frequency ridges, and average over phase-shifted realizations to generate refined TFRs and robustly tokenize non-harmonic, multicomponent signals (Laine et al., 3 May 2026).

3. Design Principles: Multi-Scale, Locality, and Quantization

Effective tokenization exploits a set of general inductive biases and architectural choices:

• Multi-scale analysis: Multi-branch or multi-resolution pipelines—e.g., parallel convolution layers with varying kernel sizes (NeuroRVQ (Barmpas et al., 15 Oct 2025)), multi-level wavelet decompositions (WaveToken), or multi-band spectral partitioning (BandTok (Cheng et al., 15 May 2026))—allow detection and compact encoding of both slow-varying trends and fast, localized transients.
• Explicit time–frequency locality: Tokenizers are constructed so that each token captures a specific, interpretable region in the joint time–frequency plane, enabling reconstruction and analysis of localized motifs (Muzhikyan et al., 2011, Pradeepkumar et al., 22 Feb 2025, Cheng et al., 15 May 2026). In learned systems, this property is reinforced by position- and band-aware embeddings (e.g., 2D RoPE (Cheng et al., 15 May 2026)) and transformer attention constrained to local neighborhoods.
• Hierarchical and residual quantization: To minimize information loss and support fine-grained representation, hierarchical RVQ stacks are used, with each stage encoding residual information at progressively finer scales (Barmpas et al., 15 Oct 2025). Single-level vector quantization and shared band codebooks (e.g., BandTok) prioritize modeling independence across time or frequency dimensions.

4. Objective Functions and Reconstruction Losses

Tokenization quality is enforced via carefully crafted loss functions:

• Spectral and phase-aware objectives: For oscillatory signals (notably EEG), losses combine log-amplitude spectrum error, unit-circle (wrapped phase) alignment losses, and optional time-domain MSE between the reconstructed and original waveforms (Barmpas et al., 15 Oct 2025).
• Adversarial and perceptual metrics: Audio-oriented frameworks (BandTok) train the quantizer–decoder stack against PatchGAN discriminators at multiple scales, supplementing standard L1/L2 losses to enhance fidelity of reconstructed high-frequency detail (Cheng et al., 15 May 2026).
• Pretraining via masked modeling: Downstream effectiveness is bolstered by pretraining transformers to predict masked tokens, adapting BERT-style MLM and minimizing cross-entropy on token predictions, with masking schemes tailored to the domain (e.g., across EEG channels, time, or bands) (Pradeepkumar et al., 22 Feb 2025).

5. Empirical Performance and Interpretability

Empirical evaluations across domains demonstrate the utility of time–frequency tokenization:

• Signal reconstruction: RVQ EEG tokenizers achieve 10–100× lower per-band reconstruction MSE vs. standard neural tokenizers, preventing high-frequency dropout and phase artifacts (Barmpas et al., 15 Oct 2025). In music generation, BandTok yields superior Mel- and STFT-distance metrics relative to residual-codebook tokenizers, and supports autoregressive synthesis without error amplification (Cheng et al., 15 May 2026). For time-series forecasting, WaveToken outperforms deep learning and foundation models on “in-domain” and “zero-shot” datasets with a much smaller vocabulary, maintaining faithful timing of trends, spikes, and frequency evolution (Masserano et al., 2024).
• Downstream classification and motif discovery: In EEG, tokens learned via TFM-Tokenizer encode class-distinctive, frequency-specific motifs, enabling interpretable mapping between codes and phenomena such as periodic discharges or event-related potentials (Pradeepkumar et al., 22 Feb 2025). Similar gains are observed in precision metrics for retrieval and uniqueness of class-defining tokens.
• Noise robustness and decomposition fidelity: TFMD demonstrates denoising capability—raising output SNR well above input SNR at low SNR levels—and robust mode recovery across a suite of synthetic and real-world signals (Zhou et al., 16 Jul 2025). TETRIS further sharpens TFRs by averaging over phase-shifted realizations, reducing variance by ~1/(Q+1), and achieves high-fidelity respiratory signal extraction from PPG in both simulation and clinical tests (Laine et al., 3 May 2026).

6. Applications Across Scientific and Engineering Domains

Time–frequency tokenization techniques are foundational in several major domains:

Domain	Representative Method	Key Outcomes
EEG / Brainwave Modeling	NeuroRVQ, TFM-Tokenizer	Nearly lossless compression, interpretable motif tokens, improved downstream accuracy (Barmpas et al., 15 Oct 2025, Pradeepkumar et al., 22 Feb 2025)
Time Series Forecasting	WaveToken	Compact vocabulary, structured prediction, better generalization (Masserano et al., 2024)
Audio / Music Generation	BandTok, STFT+Masking	Autoregressive synthesis, token grid structure, high-fidelity detail (Cheng et al., 15 May 2026)
Biomedical Oscillatory	TETRIS, TFMD	Noise-robust TFR, component ridge mapping, physiological signal extraction (Laine et al., 3 May 2026, Zhou et al., 16 Jul 2025)
Quantum/Optical Comm.	Schmidt-mode filtering	Efficiency–discrimination tradeoff, mode multiplicity (Raymer et al., 2020)

These methods provide pipelines for generative modeling, event detection, data compression, and physiological state estimation, often with explicit guarantees regarding reconstruction or interpretability.

7. Open Problems and Formal Tradeoffs

• Non-commutative geometry and phase-space tilings: Token dictionaries derived from exponentially localized atoms (e.g., (Muzhikyan et al., 2011)) must obey anticommutation relations, leading to multiple interleaved sublattices and a minimal tile area bound by the Heisenberg uncertainty limit.
• Efficiency–discrimination tradeoff: In physical implementations (especially for quantum and optical systems), increasing mode selectivity leads to reduced transmission efficiency, governed quantitatively by the product of temporal and spectral filter bandwidths, $BT$, and the resulting Schmidt eigenvalue distribution (Raymer et al., 2020).
• Modeling long-range dependencies: Flattening schemes in 2D token grids present a tradeoff: while band-first orders break strong URQ dependencies (BandTok), they may also decouple physically linked spectral components, affecting language modeling perplexity and error propagation (Cheng et al., 15 May 2026).
• Generalization to non-harmonic, nonstationary signals: Data-driven tessellation (TETRIS) and TFMD pipelines adaptively segment the time–frequency plane for mode extraction, but further work is needed to universalize such models to signals exhibiting rapid, complex cross-scale interactions (Laine et al., 3 May 2026, Zhou et al., 16 Jul 2025).

A plausible implication is that ongoing research will refine both the mathematical guarantees of invertibility and discrimination, as well as adaptive, interpretable tokenization protocols for increasingly complex temporal signals.