---
title: Joint Time-Frequency Scattering (JTFS)
url: https://www.emergentmind.com/topics/joint-time-frequency-scattering-jtfs
type: topic
---

# Joint Time-Frequency Scattering (JTFS)

Joint Time-Frequency Scattering (JTFS) is a multilayer, convolutional-wavelet transform yielding a structured, locally invariant representation of the spectrotemporal modulations of audio signals. Originally framed as an extension of time scattering, JTFS integrates a constant-Q Morlet filterbank in time with a two-dimensional wavelet transform along both time and log-frequency axes, followed by modulus nonlinearities and local averaging. This architecture produces a feature space with high discriminative power for complex nonstationary phenomena, such as frequency and amplitude modulation, tremolo, vibrato, and transient-rich textures, while providing mathematical guarantees of time-shift invariance and Lipschitz stability to deformations [1512.02125][1807.08869][2602.11896][2204.08269].

## 1. Mathematical Formulation and Cascade Architecture

JTFS operates by cascading wavelet convolution, modulus, and local averaging in multiple layers, extending one-dimensional time scattering to a two-dimensional time–frequency setting:

1. **First-layer: Constant-Q Temporal Wavelets**

   - Input \( x(t) \) (finite energy).
   - Morlet wavelets \( \psi_\lambda(t) = \lambda \exp\bigl(-\frac{\lambda^2 t^2}{2Q^2}\bigr)(e^{2\pi i \lambda t} - \kappa) \), where \( Q \) is quality factor and \( \kappa \) ensures zero mean. Frequencies \( \lambda = 2^{n/Q}, n \in \mathbb Z \) cover the frequency axis at constant-Q resolution.
   - Compute the first-layer modulus "scalogram":
     \[
     U_1 x(t,\lambda) = |x * \psi_\lambda|(t).
     \]
   
2. **Second-layer: 2D Joint Time–Frequency Wavelets**
   - Temporal modulation wavelets \( \psi_\alpha(t) \) (center rate \( \alpha \), Q=1), and spectral modulation wavelets \( \psi_\beta(\log_2 \lambda) \) (center scale \( \beta \), Q=1), both discretized geometrically (\( \alpha, \beta = \pm 2^m \)).
   - The joint filter is the tensor product:
     \[
     \Psi_{\alpha,\beta}(t,u) = \psi_\alpha(t)\psi_\beta(u), \quad u = \log_2 \lambda.
     \]
   - The joint 2D convolution, followed by modulus, isolates spectrotemporal modulations:
     \[
     U_2 x(t, \lambda, \alpha, \beta) = \left|\iint U_1 x(\tau, s)\,\psi_\alpha(t-\tau)\, \psi_\beta(\log_2 \lambda - s)\,d\tau ds\right|.
     \]
     
3. **Local Averaging and Invariance**
   - Temporal and frequency domain low-pass filters (\( \phi_T(t) \), \( \phi_F(\log_2 \lambda) \)), yielding:
     \[
     S_1 x(t, \lambda) = U_1 x * \phi_T * \phi_F,\quad S_2 x(t, \lambda, \alpha, \beta) = U_2 x * \phi_T * \phi_F.
     \]
   - Local shift-invariance up to scales \( T \) (time) and \( F \) (frequency, in octaves).

4. **Scattering Feature Vector**
   - JTFS features are typically the concatenation of \( S_1 \) and \( S_2 \), often after stabilized logarithmic compression:
     \[
     \Phi_{\text{JTFS}}(x) = \log(1 + \text{JTFS}(x)/\epsilon),\ \epsilon \sim 10^{-3}.
     \]
   
The feature dimensionality depends on the configuration of the filterbanks, typically \( 10^2 \)–\( 10^3 \) for audio textures with realistic parameter sweeps [2602.11145][2311.14213][2204.08269][1512.02125].

## 2. Tiling of the Time–Frequency Plane and Parameterization

JTFS achieves a detailed tiling of the time–frequency plane through joint selection of filter center frequencies and bandwidths:
- The first-layer wavelets implement a constant-Q (log-frequency) coverage for high frequencies, with bandwidth limited by \( \lambda/Q \), transitioning to constant bandwidth below an "elbow scale".
- The second layer's 2D wavelets form a separable basis on the scalogram, localizing joint modulations in rectangles of size \( \sim 1/|\alpha| \) (time) and \( \sim 1/|\beta| \) (log-freq), permitting analysis of localized and oriented Gabor-like atoms.
- Hyperparameters include:
  - \( J \): octaves, \( Q^{\text{tm},1} \): temporal quality factor (typ. 8–12).
  - \( Q^{\text{tm},2} \), \( Q^{\text{fr},1} \): second-layer quality factors, often 1.
  - \( T, F \): temporal and frequency averaging windows.
  - Subsampling factors may be introduced for computational tractability, dependent on the redundancy of the wavelet cover [2602.11896][1512.02125][1810.04506].

## 3. Computational Realization and Differentiable Implementations

JTFS can be efficiently implemented using FFT-based convolutions for both 1D and 2D wavelet transforms. The critical operators are differentiable, supporting integration with deep learning frameworks. Notable open-source implementations include Kymatio and its higher-level derivatives:
- GPU kernels apply FFT-based convolution, modulus, and 2D average pooling [2204.08269].
- Pseudocode exposes sequential layering: constant-Q filtering, 1D modulus+pooling, 2D joint convolution+modulus+pooling, and aggregation over scattering paths [2602.11896][2204.08269].
- Advanced stochastic optimization schemes such as SCRAPL further address computational burden by randomly sampling over JTFS paths and employing adaptive stochastic gradient techniques (P-Adam, P-SAGA, θ-importance sampling). This provides ~20× acceleration at modest cost in perceptual accuracy when JTFS is used as a differentiable loss [2602.11145].

| Library         | Backend         | Differentiable | GPU Support | Notes          |
|-----------------|----------------|---------------|-------------|----------------|
| Kymatio         | NumPy/PyTorch/TF| Yes           | Yes         | Widest support |
| scattering.m    | MATLAB/Python   | Partial       | No (MATLAB) | Early reference|
| SCRAPL          | Python          | Yes           | Yes         | JTFS loss focus|

## 4. Theoretical Guarantees and Stability Properties

JTFS inherits and extends Mallat's scattering transform guarantees:
- **Translation Invariance**: Averaging with \( \phi_T, \phi_F \) produces invariance to time shifts up to \( T \) and frequency shifts up to \( F \) [1512.02125][2007.10926].
- **Lipschitz Stability**: The modulus-convolution cascade is provably stable to time-warpings and small deformations in \( (t,\lambda) \), implying robust embeddings under pitch- or rate-modulated transformations [1807.08869][1512.02125][1606.08677].
- **Energy Preservation and Exponential Decay**: The sequence of modulus convolutions with uniform covering frames retains most energy in low-depth, frequency-decreasing paths, and total energy in deeper layers decays exponentially, supporting practical truncation at second order [1606.08677].
- **Approximate Invertibility**: For sufficiently deep and dense JTFS, the mapping is invertible up to global translations, and phase retrieval through gradient-based optimization enables convincing audio texture resynthesis [1906.09334].

## 5. Comparison with Related Spectrotemporal Representations

JTFS generalizes and improves upon prior biologically and physically motivated spectrotemporal feature extractors:
- **Spectrotemporal Receptive Fields (STRF)** and **Gabor Filterbanks (GBFB)**: Employ 2D Gabor-like filtering on spectrograms but use linear, non-cascaded filtering and typically do not employ nonlinearity or multiresolution parameterization.
- **Modulation Power Spectrum (MPS)**: Computes the 2D Fourier transform of the squared-magnitude STFT, capturing global modulation statistics but sacrificing localization and invertibility [2602.11896][1906.09334].
- JTFS is distinguished by: (i) multilayer wavelet cascades with complex modulus, (ii) explicit adaptation of bandwidths across scales (constant-Q/elbow construction), (iii) controlled local invariance through separable averaging, and (iv) Lipschitz continuity for stability [2602.11896][1512.02125].

## 6. Applications in Machine Learning, Audio Analysis, and Synthesis

JTFS is employed in diverse applications including:
- **Texture Synthesis and Musical Metamerism**: JTFS coefficients serve as differentiable statistics for audio resynthesis and metamer generation via inverse optimization; JTFS-based synthesis demonstrates improved preservation of spectrotemporal structure relative to MFCC, pure time-scattering, or MPS, key for “musical metamerism” [2602.11896][1906.09334].
- **Perceptual Sound Matching and Inverse Problems**: Integration with neural synthesis pipelines (e.g., DDSP) enables perceptually meaningful, gradient-friendly loss functions for sound matching tasks; JTFS-based loss functions such as the PNP framework yield state-of-the-art results and 100× acceleration via precomputation and quadratic-model approximations [2311.14213].
- **Classification and Retrieval**: JTFS features assigned to simple linear metric learning suffice to recover human similarity judgments between instrumental techniques, outperforming MFCC and shallow scattering representations on isolated-note clustering and instrument/technique identification [2007.10926]. In TIMIT phone classification and acoustic scene classification, JTFS outperforms both hand-engineered and learned CNN/MLP features, especially in data-scarce conditions [1512.02125][2204.08269].
- **Unsupervised and supervised feature learning**: JTFS has been shown to better linearize independent generative factors in manifold learning tasks compared to MFCC, Scat1D, STRF, and OpenL3 [2204.08269].

## 7. Parameter Regimes, Design Trade-Offs, and Practical Guidelines

The discriminative capacity and invariances of JTFS are tunable via its parameters:
- Typical first-order temporal Q: 8–12 per octave, matching human auditory discrimination; second-order modulations: Q=1, a few filters per octave [2311.14213][1512.02125].
- Averaging window \( T \) is chosen to match the time scale over which local invariance is desired (hundreds of ms for musical notes).
- Spectral averaging \( F \) provides invariance to pitch or timbral transpositions; often 1–2 octaves for note-level analyses [2602.11896].
- Multirate downsampling is applied in each layer proportional to the effective bandwidth, reducing feature dimensionality by up to 30× vs. fully sampled 2D STRF grids [2007.10926].
- Logarithmic or median-based nonlinearity stabilizes dynamic range prior to use in learning or distance metrics [2311.14213][2007.10926].
- Stochastic or importance path sampling (SCRAPL) enables practical use as a loss for deep stochastic gradient descent [2602.11145].

| Parameter         | Typical values           | Effect                                    |
|-------------------|-------------------------|-------------------------------------------|
| Q (temporal)      | 8–12 / octave           | Frequency vs. time resolution             |
| T (avg window)    | 100–500 ms              | Time shift invariance                     |
| F (spectral avg)  | 1–2 octaves             | Frequency invariance                      |
| Q_mod (2D layer)  | 1 / octave              | Joint modulation resolution               |
| Downsampling      | Critical (by α, β)      | Dimensionality reduction                  |

Empirical results support JTFS as the currently most effective handcrafted auditory representation for nonstationary timbre, transient structure, and cross-frequency modulation analysis [2007.10926][2311.14213][1512.02125][2204.08269].

Source: https://www.emergentmind.com/topics/joint-time-frequency-scattering-jtfs