Scattering Transformer Models

• Scattering Transformer is a set of models that embed fixed scattering transforms into transformer architectures to preserve fine-grained, invariant signal details.
• They utilize wavelet, Gabor, and spectral techniques to separate frequency components, enhancing feature extraction and resilience to noise.
• Empirical studies in heart murmur detection, EEG, and image classification show that these models boost performance while reducing reliance on extensive training.

Scattering Transformer denotes a class of transformer architectures that incorporate scattering-based signal representations into tokenization, feature extraction, or contextual aggregation. In the narrow sense, the name refers to a training-free architecture for heart murmur detection that combines a Wavelet Scattering Network (WSN), a parameter-free transformer-like contextualization module, and an SVM classifier (Zewail, 22 Sep 2025). In broader usage, closely related models such as “ScatterFormer” (Zheng et al., 2023) and “Scattering Vision Transformer” (Patro et al., 2023) integrate invariant scattering transforms or spectral scattering networks into hierarchical transformer pipelines to preserve fine-grained, high-frequency, or locally invariant structure while retaining long-range dependency modeling.

1. Scattering-theoretic foundations

The conceptual substrate of scattering-transformer architectures is the scattering transform, a wavelet-based model of Convolutional Neural Networks originally introduced by S. Mallat (Perlmutter et al., 2021). In this line of work, scattering layers are built from fixed filter banks, pointwise modulus nonlinearities, and low-pass averaging. The resulting representations are designed rather than learned, and their appeal is tied to explicit invariance and stability guarantees.

A representative formulation on compact manifolds defines spectral integral operators

$$K_\eta(x, y) = \sum_{k} \eta(\lambda_k)\, \varphi_k(x)\, \overline{\varphi_k(y)}, \qquad T_\eta f(x) = \int_M K_\eta(x, y)f(y)\,dV(y),$$

with the propagator and scattering coefficients given by

$$U_\gamma f(x) = |T_{h_\gamma}f(x)|, \qquad U_{\vec{\gamma}} f = U_{\gamma_m}\cdots U_{\gamma_1}f,$$

$$S_{\vec{\gamma}} f(x) = T_g\,U_{\vec{\gamma}} f(x), \qquad S f = \{ S_{\vec{\gamma}} f : \vec{\gamma} \in \Gamma_\infty \}.$$

In this setting, geometric scattering provides localized isometry invariant descriptions of manifold signals and is stable to families of diffeomorphisms formulated in intrinsic manifold terms; under a Littlewood–Paley condition with $B=1$, the transform is non-expansive (Perlmutter et al., 2018).

A second foundational strand is the hybrid scattering transform for signals with isolated singularities. There, a first wavelet layer promotes sparsity and a second Gabor layer leverages the induced sparse structure. For piecewise polynomials with isolated singularities, the first layer can reduce the signal to a sparse train of Dirac deltas near the singularity locations, after which Gabor measurements provide translation-invariant and distinguishing features and can also be used to synthesize sparse signals up to natural ambiguities such as translation, reflection, and sign (Perlmutter et al., 2021). This hybrid construction is directly relevant to later transformer hybrids because it establishes that different layers may legitimately use different filter families when the intermediate representation changes character.

2. Terminology and representative uses

The label “Scattering Transformer” is not attached to a single canonical architecture. It has been used both for a specific biomedical audio model and, more broadly, for transformer systems that embed scattering transforms or spectral scattering modules into their front ends or intermediate stages. The term therefore functions more as a family resemblance than as a uniquely standardized architecture.

Model	Domain	Scattering component
Scattering Transformer (Zewail, 22 Sep 2025)	Heart murmur detection	WSN front-end with parameter-free transformer-like contextualization
ScatterFormer (Zheng et al., 2023)	Multispectral EEG epileptiform detection	Invariant scattering token embedding with disentangled frequency-aware attention
Scattering Vision Transformer (Patro et al., 2023)	Image classification and instance segmentation	DTCWT-based spectrally scattering network with spectral gating

Within this family, the common architectural move is to use scattering as an information-preserving or invariance-inducing stage before, or alongside, transformer-style contextual modeling. The precise role varies: in some cases scattering replaces early learned attention, in others it acts as a token embedding mechanism, and in the training-free murmur model it provides the entire front-end representation.

3. Training-free Scattering Transformer for heart murmur detection

The 2025 model titled “Scattering Transformer” is a lightweight, training-free neural model for heart murmur detection (Zewail, 22 Sep 2025). Its architecture has three components: a WSN front-end for handcrafted feature extraction, a parameter-free transformer-like contextualization module, and a supervised SVM classifier with quadratic kernel.

The WSN computes scattering coefficients of the form

$$S_m f = |\ldots|f * \psi_{\lambda_1}| * \ldots * \psi_{\lambda_m}| * \phi_J|,$$

with aggregate representation

$S f = \{S_m f\}_{0 \le m \le M}.$

This front end produces a deep, low-variance feature representation robust to deformation and noise. Contextualization is then added without learnable projections. After positional encoding, attention is computed directly from the scattering features:

$$\text{Attention}(X_{pos}) = \text{softmax}\left( \frac{X_{pos} X_{pos}^T}{\sqrt{d_k}} \right) X_{pos}.$$

Two operational modes are described: “Paths-as-Sequence Mode,” in which scattering paths from a single segment form the sequence, and “Multi-Segment Mode,” in which multiple temporal segments yield a sequence of scattering embeddings (Zewail, 22 Sep 2025).

A defining property of this architecture is that the core feature extractor and contextualizer contain no learnable parameters. The only trained component is the lightweight SVM classifier. The paper states that no gradient descent or backpropagation is used at any stage except for the supervised SVM head, and attributes the training-free character to the fixed mathematical construction of the WSN and the algebraic attention mechanism (Zewail, 22 Sep 2025).

On the public CirCor DigiScope dataset, the model achieves a Weighted Accuracy of 0.786 and an Unweighted Average Recall of 0.697, and reports Recall for the “Unknown” class of 0.692 (Zewail, 22 Sep 2025). The same source describes this performance as highly competitive with contemporary state-of-the-art methods and notes that the parameter-free attention contextualization step boosts performance by 63% over plain WSN. The design is explicitly positioned as a lightweight alternative to general-purpose audio foundation models in resource-constrained setups.

4. Learned scattering-transformer architectures in EEG and vision

In EEG analysis, “ScatterFormer” is an invariant scattering transform-based hierarchical Transformer for patient-independent multispectral detection of epileptiform discharges (Zheng et al., 2023). The model starts from a saliency-aware multispectral representation obtained through continuous wavelet transform, then introduces invariant scattering token embedding at the patch-embedding stage. For an input $f(x)$, the scattering propagator is

$$U[\lambda]f(x) := \left| (\psi_\lambda * f)(x) \right|,$$

with pathwise propagation

$$U[p]f(x) = U[\lambda_m] \cdots U[\lambda_2] U[\lambda_1] f(x),$$

and scattering coefficients

$$U_\gamma f(x) = |T_{h_\gamma}f(x)|, \qquad U_{\vec{\gamma}} f = U_{\gamma_m}\cdots U_{\gamma_1}f,$$0

ScatterFormer uses second-order scattering in the initial embedding and first-order scattering in later stages. Its disentangled frequency-aware attention (FAA) splits processing into high-frequency and low-frequency branches so that clinically informative high-frequency components can be emphasized (Zheng et al., 2023). The paper reports median AUCROC and accuracy of 98.14% and 96.39% in patients with Rolandic epilepsy, and states that on a neonatal seizure detection benchmark it outperforms the state of the art by 9% in terms of average AUCROC.

In computer vision, “Scattering Vision Transformer” (SVT) replaces early self-attention with a spectrally scattering network based on the Dual-Tree Complex Wavelet Transform (DTCWT) (Patro et al., 2023). The DTCWT separates low-frequency and high-frequency components,

$$U_\gamma f(x) = |T_{h_\gamma}f(x)|, \qquad U_{\vec{\gamma}} f = U_{\gamma_m}\cdots U_{\gamma_1}f,$$1

thereby preserving global structure and directional detail. SVT couples this decomposition to a spectral gating network that uses Einstein multiplication for token and channel mixing. The low-frequency branch uses tensor blending, while the high-frequency branch uses Einstein blending to reduce parameter count and FLOPs relative to naïve full mixing (Patro et al., 2023).

SVT is explicitly motivated by two issues in vision transformers: attention complexity and the information loss introduced by non-invertible down-sampling. The paper reports that SVT-H-S reaches 84.2% top-1 accuracy, SVT-H-B reaches 85.2%, and SVT-H-L reaches 85.7% on ImageNet, with SVT showing a 2% improvement over LiTv2 and iFormer; it also reports comparable results in instance segmentation and strong transfer performance on CIFAR10, CIFAR100, Oxford Flower, and Stanford Car (Patro et al., 2023).

5. Recurring design principles

Across these architectures, scattering is consistently used to expose structure before full transformer contextualization. In the murmur model, the WSN supplies deformation-robust coefficients and the transformer-like block adds contextual dependencies without backpropagation (Zewail, 22 Sep 2025). In ScatterFormer, scattering token embedding and FAA are used to prevent oversmoothing and preserve subtle high-frequency EEG patterns (Zheng et al., 2023). In SVT, early attention is replaced by DTCWT-based spectral scattering so that low-frequency and high-frequency components can be mixed separately and more efficiently (Patro et al., 2023).

A second recurring principle is explicit frequency separation. The hybrid scattering transform for isolated singularities already argued that wavelets are appropriate in the first layer because they localize and sparsify singularities, whereas Gabor filters are more effective for the sparse outputs of that first layer (Perlmutter et al., 2021). ScatterFormer carries this logic into a high-/low-frequency dual-branch transformer, and SVT encodes it in low-/high-frequency DTCWT channels. This suggests that scattering-transformer designs are often motivated by a layerwise change in signal regime: smooth or low-frequency structure is handled differently from sparse, local, or high-frequency detail.

A third principle is the retention of formal invariance or stability properties. Geometric scattering on manifolds establishes isometry invariance up to scale, Lipschitz stability to diffeomorphisms, and non-expansiveness under appropriate conditions (Perlmutter et al., 2018). Although the transformer hybrids are task-specific and often empirical in emphasis, they inherit the broader rationale that designed scattering operators can regularize the representation prior to contextual modeling.

6. Distinctions from other uses of “scattering” in transformer research

The phrase should not be conflated with every transformer paper that uses the word “scattering.” In scientific machine learning, a hybrid FNO-transformer architecture has been proposed for high-contrast Helmholtz problems by decomposing the operator into smooth background propagation and a high-contrast scattering correction, with the FNO learning the smooth component and a vision transformer modeling the residual scattering interactions (Balaji et al., 30 Jan 2026). This is a transformer for wave scattering maps, not a scattering-transform front end in the Mallat sense.

In SAR interpretation, SAR-GTR integrates Attributed Scattering Center parameters into a graph transformer, distinguishing discrete and continuous electromagnetic scattering parameters and adding edge information enhancement, global node encoding, and edge position encoding (Xiong et al., 13 May 2025). Here “scattering” refers to electromagnetic inverse scattering information encoded in graph nodes and edges rather than to wavelet scattering networks.

In zero-shot video object segmentation, the “Semantic Gathering-Scattering Transformer” in Isomer uses semantic query gathering, key/value soft merging, and token scattering back to original positions for foreground and background branches; the paper reports a 13-times speed increase over a baseline using vanilla Transformers for multi-stage fusion (Yuan et al., 2023). The term “scattering” in that context describes a token redistribution mechanism rather than a scattering transform.

A further distinct usage appears in theoretical high-energy physics, where an encoder-decoder transformer has been trained to simplify scattering amplitudes expressed in spinor-helicity variables and can recover the Parke–Taylor formula for five-point gluon scattering (Cheung et al., 2024). The shared vocabulary is terminological rather than architectural.

A common misconception is therefore that “Scattering Transformer” names a single standardized model class. The literature instead shows several non-equivalent constructions linked by one of two ideas: either the direct use of scattering transforms as invariant or multiscale front ends, or the use of “scattering” in a domain-specific physical or algorithmic sense. Within the wavelet-scattering lineage, the most specific use of the name remains the training-free murmur detector of 2025 (Zewail, 22 Sep 2025), while the broader research theme encompasses trained EEG and vision transformers that use scattering-based spectral representations to preserve informative structure before global contextual mixing.