Sound Matching Models Overview

Updated 6 July 2026

Sound matching models are systems optimized to align synthesized or processed audio with target specifications using criteria such as field fidelity or perceptual similarity.
They employ methodologies like weighted pressure and mode matching, neural inverse synthesis with perceptual loss, and iterative optimizer frameworks for robust performance.
These models underpin diverse applications including room acoustics reproduction, sound source extraction, and anomaly detection, with task-dependent metrics ensuring accurate alignment.

Searching arXiv for recent and foundational papers on “sound matching” to ground the article. Sound matching model denotes a family of formulations in which a system is optimized or trained to make a synthesized, retrieved, localized, transferred, or scored acoustic object match a target specification under an explicit criterion. In the literature, that target specification may be a desired sound field over a spatial region, a target waveform to be approximated by parametric synthesis, a target acoustic environment inferred from an image, a text description of a source to be extracted from a mixture, or a normal reference memory against which a test sound is compared (Koyama et al., 2022, Han et al., 2023, Chen et al., 2022, Yuan et al., 2024, Saengthong et al., 14 Mar 2026). The common structure is an input–output mapping constrained by a task-specific notion of correspondence: field fidelity, perceptual similarity, semantic alignment, room-acoustic consistency, or anomaly discriminability.

1. Regional acoustic field matching

In spatial acoustics, a sound matching model is often a sound field reproduction model. The objective is to choose loudspeaker driving signals so that a synthesized field reproduces a desired field inside a target region $\Omega$ , not merely at isolated control points. A representative formulation writes the synthesized field as

$u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$

with secondary-source driving signals $d_l$ and transfer functions $g_l$ , and defines the continuous objective

$J=\int_{\Omega}\left|\sum_{l=1}^{L} d_l g_l(\mathbf r)-u_{\mathrm{des}(\mathbf r)\right|^2\, d\mathbf r.$

This is the formulation adopted in "Weighted Pressure Matching Based on Kernel Interpolation For Sound Field Reproduction" (Koyama et al., 2022).

Conventional pressure matching (PM) replaces the regional objective by a regularized least-squares fit at discrete control points: $\min_{\mathbf d\in\mathbb C^L}\ \|\mathbf G \mathbf d-\mathbf u^{\mathrm{des}}\|^2 + \eta \|\mathbf d\|^2,$ with closed-form solution

$\hat{\mathbf d}= \left(\mathbf G^{\mathsf H}\mathbf G+\eta \mathbf I\right)^{-1} \mathbf G^{\mathsf H}\mathbf u^{\mathrm{des}}.$

Its appeal is simplicity and flexibility with respect to array geometry, but its weakness is equally explicit: the space between control points is not represented in the objective (Koyama et al., 2022).

The main generalization is weighted pressure matching (WPM). WPM begins from kernel interpolation of Helmholtz-consistent sound fields in an RKHS and converts the regional error into a weighted discrete least-squares criterion: $\min_{\mathbf d\in\mathbb C^L} \ (\mathbf G\mathbf d-\mathbf u^{\mathrm{des}})^{\mathsf H} \mathbf W (\mathbf G\mathbf d-\mathbf u^{\mathrm{des}}) +\eta \|\mathbf d\|^2,$ with solution

$\hat{\mathbf d} = \left( \mathbf G^{\mathsf H}\mathbf W\mathbf G+\eta \mathbf I \right)^{-1} \mathbf G^{\mathsf H}\mathbf W\mathbf u^{\mathrm{des}}.$

The weighting matrix $\mathbf W$ is induced by regional integration of interpolation functions, so the residual vector is no longer penalized uniformly. This makes WPM a discrete surrogate of a continuous regional matching problem rather than a pure point-matching problem (Koyama et al., 2022).

A closely related formulation is weighted mode matching (WMM), which represents source-free fields by spherical wavefunctions and matches modal coefficients with a regional weighting matrix. "Sound Field Reproduction With Weighted Mode Matching and Infinite-Dimensional Harmonic Analysis: An Experimental Evaluation" reports that WMM, combined with infinite-dimensional harmonic analysis (IDHA) for coefficient estimation from arbitrarily placed microphones, outperforms conventional PM, especially when the number of microphones is small (Koyama et al., 2021). "Weighted Pressure and Mode Matching for Sound Field Reproduction: Theoretical and Experimental Comparisons" further shows that WPM is a special case of WMM when modal coefficients are estimated from pressure observations by infinite-dimensional harmonic analysis (Koyama et al., 2023).

A psychoacoustically guided variant changes the matched quantity itself as a function of frequency. "Perceptual Quality Enhancement of Sound Field Synthesis Based on Combination of Pressure and Amplitude Matching" uses full complex pressure at low frequencies and amplitude-only matching at high frequencies: $u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$0 The method is solved with ADMM rather than in closed form and is motivated by the claim that, above $u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$1 Hz, matching amplitude distribution is sufficient for horizontal localization, whereas low-frequency phase remains important (Kimura et al., 2023).

2. Inverse synthesis and parameter-space sound matching

A second major use of the term refers to inverse control of a parametric synthesizer. Here the task is: given a target waveform $u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$2 and a known synthesizer $u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$3, estimate parameters $u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$4 such that $u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$5 perceptually resembles $u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$6. "Perceptual-Neural-Physical Sound Matching" formalizes this as supervised encoder learning, $u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$7, and argues that nonstationary, inharmonic sounds such as percussion expose a mismatch between parameter error and perceptual error (Han et al., 2023).

That paper introduces Perceptual–Neural–Physical (PNP) loss, obtained by linearizing the perceptual map $u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$8 around the ground-truth parameters: $u_{\mathrm{syn}(\mathbf r,\omega)=\sum_{l=1}^{L} d_l(\omega)\, g_l(\mathbf r,\omega),$9

$d_l$ 0

The resulting metric tensor acts as a local perceptual sensitivity matrix over parameter space. The paper uses a physical modeling decoder based on the functional transformation method for a rectangular drum PDE and a joint time–frequency scattering transform (JTFS) perceptual representation. In the harder unknown-pitch setting, PNP slightly outperforms parameter loss and clearly outperforms multi-scale spectral loss, while training at essentially the same speed as parameter loss and far faster than a full differentiable JTFS loss (Han et al., 2023).

"Learning to Solve Inverse Problems for Perceptual Sound Matching" generalizes the same idea under the name perceptual sound matching (PSM) and emphasizes the inverse-problem perspective: $d_l$ 1 It reports that PNP offers a 100-fold speedup during gradient descent compared to DDSP and that PNP-accelerated JTFS has greater influence on performance than parameter rescaling, pretraining, auditory representation alternatives, or gradient clipping (Han et al., 2023).

A distinct engineering path avoids differentiable synthesis entirely and treats sound matching as direct regression from audio to synthesizer controls. "Synthesizer Sound Matching Using Audio Spectrogram Transformers" formulates the task as

$d_l$ 2

with an Audio Spectrogram Transformer predicting a vector of 16 continuous synthesizer parameters from a 64-bin Mel spectrogram. The model outperforms MLP and CNN baselines on both parameter MSE and rendered-audio spectral convergence, and the paper positions it as a general-purpose inverse controller for a fixed commercial synthesizer rather than a differentiable-synth method (Bruford et al., 2024).

DiffMoog extends this line in the opposite direction by making the synthesizer itself differentiable and modular. "DiffMoog: a Differentiable Modular Synthesizer for Sound Matching" defines

$d_l$ 3

with a user-defined architecture $d_l$ 4, and combines parameter supervision with a signal-chain loss that compares intermediate outputs across synthesizer stages. The paper is explicit that optimization remains fragile, especially for FM and frequency estimation, but the framework unifies modular routing, differentiable DSP, and architecture-conditioned parameter prediction (Uzrad et al., 2024).

A third formulation dispenses with neural inverse mapping altogether. "Evaluating Sound Similarity Metrics for Differentiable, Iterative Sound-Matching" treats sound matching as iterative optimization of a differentiable synthesizer under a chosen similarity metric,

$d_l$ 5

and concludes that loss-function performance is highly dependent on the synthesizer: STFT-based losses for subtractive filtered noise, JTFS for additive synthesis, and soft-DTW on envelopes for AM-like synthesis (Salimi et al., 27 Jun 2025). This suggests that “sound matching model” can denote not only an encoder–decoder but also a differentiable synthesis–loss–optimizer loop.

A third family uses external modalities—video, images, or language—to specify the target sound. In this setting, sound matching is a conditional retrieval, selection, or reconstruction problem rather than a pure inverse-synthesis problem.

"Soundify: Matching Sound Effects to Video" is explicit that it is not a single end-to-end learned sound matching model but an interactive retrieval-and-remixing pipeline. It uses CLIP image embeddings for scene representation, CLIP text embeddings for sound labels, cosine similarity for sound-category retrieval, thresholded framewise similarity for interval detection, and Grad-CAM localization for panning and gain control. The matching engine is therefore zero-shot visual-to-text retrieval over sound labels rather than learned audiovisual embedding alignment (Lin et al., 2021).

A more direct conditional extraction model appears in "FlowSep: Language-Queried Sound Separation with Rectified Flow Matching." There the input is a mixture waveform $d_l$ 6 and a natural-language query $d_l$ 7, and the model generates the target source in VAE latent space via rectified flow matching: $d_l$ 8

$d_l$ 9

$g_l$ 0

This is not a retrieval score model; it is a text-conditioned generative extraction model that reconstructs the matched source through a VAE decoder and BigVGAN (Yuan et al., 2024).

Visual prompting can also define the target sound indirectly. "VP-SelDoA: Visual-prompted Selective DoA Estimation of Target Sound via Semantic-Spatial Matching" introduces Cross-Instance Audio-Visual Localization (CI-AVL), where the prompt is an image from a different instance of the same sound event category. The model first builds a semantic prompt representation $g_l$ 1, aligns it to spatial acoustic features through cross-attention and self-attention, generates a target-selective mask, and then predicts a DoA posterior: $g_l$ 2 This is a sound matching model in the sense of category-conditioned source selection and localization inside a mixture (Chen et al., 10 Jul 2025).

A much older but conceptually related sequence-matching formulation appears in "Audio to score matching by combining phonetic and duration information." There the query is a pre-segmented singing phrase, the candidates are score phrases, and the model builds one lyric-path HMM or HSMM per candidate. Matching is achieved by ranking posterior probabilities of the decoded most likely state sequences. The paper argues that melody alone is ambiguous in jingju and that phonetic information plus explicit duration modeling is more discriminative; HSMM outperforms a post-processor duration model because duration affects decoding itself rather than only rescoring (Gong et al., 2017).

4. Acoustic environment and room-style matching

Another well-developed sense of sound matching concerns acoustic environment transfer rather than source identity. Here the target is a room, hall, or depicted environment, and the model seeks to preserve content while matching environmental acoustics.

"Visual Acoustic Matching" defines the task as learning

$g_l$ 3

where $g_l$ 4 is source audio and $g_l$ 5 is an image of the target environment. AViTAR uses a ResNet-18 image encoder, a 1D convolutional audio encoder, and a cross-modal conformer block with attention

$g_l$ 6

followed by a waveform generator trained with adversarial, feature-matching, and mel losses. The paper treats the task as acoustic style transfer: preserve semantic content while injecting reverberation and room effects implied by visible geometry and materials (Chen et al., 2022).

"Self-Supervised Visual Acoustic Matching" extends this by removing the requirement for acoustically mismatched source–target pairs. Its model, LeMARA, composes a de-biaser $g_l$ 7 with a visually guided reverberator $g_l$ 8,

$g_l$ 9

and introduces an acoustic residue metric

$J=\int_{\Omega}\left|\sum_{l=1}^{L} d_l g_l(\mathbf r)-u_{\mathrm{des}(\mathbf r)\right|^2\, d\mathbf r.$0

which measures how much room information remains in the de-biased audio. The model is self-supervised because it relies only on naturally matched $J=\int_{\Omega}\left|\sum_{l=1}^{L} d_l g_l(\mathbf r)-u_{\mathrm{des}(\mathbf r)\right|^2\, d\mathbf r.$1 pairs from videos and a learned de-biasing mechanism to create usable training structure (Somayazulu et al., 2023).

"One-Shot Acoustic Matching Of Audio Signals -- Learning to Hear Music In Any Room/ Concert Hall" addresses the same broad goal without vision. It uses a CNN encoder to infer a latent acoustic signature from a single conditioning clip recorded in the target room and a Transformer to predict a residual log-spectrogram over the input: $J=\int_{\Omega}\left|\sum_{l=1}^{L} d_l g_l(\mathbf r)-u_{\mathrm{des}(\mathbf r)\right|^2\, d\mathbf r.$2 The model matches environmental acoustics and reverberation characteristics, not event semantics, and treats arbitrary target-room audio as a surrogate observation of the room transfer function (Verma et al., 2022).

These formulations suggest a useful conceptual distinction. In inverse synthesis, the matched object is usually a source timbre or synthesis control state. In acoustic transfer, the matched object is the room response, reverberation profile, or environmental transfer characteristic. This suggests that “sound matching model” is polysemous across at least source-centered and environment-centered tasks.

5. Matching metrics, embeddings, and evaluation criteria

Across these formulations, the matched quantity is not fixed. In acoustic field control it may be regional mean-square pressure error, modal mismatch, amplitude distribution, or ILD-related structure (Koyama et al., 2022, Kimura et al., 2023). In inverse synthesis it may be parameter error, multi-scale spectral error, or JTFS distance (Han et al., 2023, Han et al., 2023). In audiovisual or language-conditioned settings it may be category retrieval accuracy, source separation quality, or semantic query relevance (Lin et al., 2021, Yuan et al., 2024).

A recent line of work asks whether sound matching models learn representations aligned with human perceptual similarity. "Assessing the Alignment of Audio Representations with Timbre Similarity Ratings" evaluates style embeddings and task embeddings from a sound matching model trained to invert the Vital synthesizer. The study finds that the ordinary task embedding is not the best representation for timbre similarity, whereas style embeddings derived from intermediate activations—both Gram-matrix style and mean–standard-deviation style—show much better alignment with human timbre judgments (Tian et al., 10 Jul 2025). This supports a broader interpretation: sound matching models can serve not only as generators or regressors but also as feature extractors for perceptual metrics.

The paper’s conclusion that style embeddings from the sound matching model outperform its task bottleneck, while CLAP-Huang style remains the strongest overall representation, suggests that the utility of a sound matching model depends strongly on which layer and which statistic are treated as the representation (Tian et al., 10 Jul 2025). A plausible implication is that internal statistics learned for control inversion may capture timbral organization better than the final control-prediction bottleneck.

A related idea appears in anomaly detection. "Sub-Band Spectral Matching with Localized Score Aggregation for Robust Anomalous Sound Detection" formulates anomaly detection as training-free memory-based sound matching. The baseline uses one global nearest neighbor

$J=\int_{\Omega}\left|\sum_{l=1}^{L} d_l g_l(\mathbf r)-u_{\mathrm{des}(\mathbf r)\right|^2\, d\mathbf r.$3

whereas BEAM matches each spectral sub-band independently and averages local distances: $J=\int_{\Omega}\left|\sum_{l=1}^{L} d_l g_l(\mathbf r)-u_{\mathrm{des}(\mathbf r)\right|^2\, d\mathbf r.$4 The paper argues that global matching inflates normal-score variance because it forces all bands to share one tied reference and couples scoring to band energy, whereas localized sub-band matching reduces variance and improves discriminability under noise and domain shift (Saengthong et al., 14 Mar 2026). This extends the notion of sound matching from synthesis and retrieval to one-class acoustic decision systems.

6. Recurrent assumptions, limitations, and research directions

Despite their diversity, sound matching models share several recurrent assumptions. Many inverse-synthesis models assume a known synthesizer $J=\int_{\Omega}\left|\sum_{l=1}^{L} d_l g_l(\mathbf r)-u_{\mathrm{des}(\mathbf r)\right|^2\, d\mathbf r.$5, synthetic training pairs, and a parameterization in which perceptual geometry is locally tractable (Han et al., 2023, Han et al., 2023). Sound field models assume source-free acoustic structure inside the target region and premeasured or known transfer functions (Koyama et al., 2022, Koyama et al., 2021). Room-acoustic matching models assume that environmental acoustics are inferable from images or proxy recordings and that content can be disentangled from room style (Chen et al., 2022, Somayazulu et al., 2023, Verma et al., 2022). Query-conditioned extraction models assume that the target source is semantically specified well enough by text or visual prompting to guide separation or localization (Yuan et al., 2024, Chen et al., 10 Jul 2025).

A persistent limitation is that the “right” matching metric is task-dependent. PNP is motivated precisely by the inadequacy of plain parameter MSE and the computational burden of full perceptual feature losses (Han et al., 2023). Iterative differentiable sound matching finds that no single loss dominates across subtractive, additive, and AM synthesizers (Salimi et al., 27 Jun 2025). BEAM shows that even nearest-neighbor anomaly scoring depends strongly on whether matching is global or localized by band (Saengthong et al., 14 Mar 2026). This suggests that sound matching is not a single modeling problem with a universal objective, but a family of inverse, retrieval, transfer, and control problems whose metrics must be aligned with the task’s physical or perceptual structure.

Another recurrent issue is ambiguity. Different synthesizer parameters may yield similar sounds; different score phrases may share similar melodic contours; different sound sources may overlap in mixtures; and different rooms may be only partially observable in an image (Gong et al., 2017, Bruford et al., 2024, Chen et al., 2022). Several methods resolve ambiguity by adding structure rather than merely enlarging the model: explicit duration models in HSMMs, regional weighting in WPM/WMM, style statistics rather than task bottlenecks, semantic-spatial bridging in VP-SelDoA, or de-biasing stages in self-supervised room matching (Koyama et al., 2023, Tian et al., 10 Jul 2025, Chen et al., 10 Jul 2025, Somayazulu et al., 2023).

Recent work also points toward richer generative acoustic representations. "SF-Flow: Sound field magnitude estimation via flow matching guided by sparse measurements" formulates 3D ATF magnitude reconstruction as conditional flow matching from sparse microphone observations and a permutation-invariant set encoder, suggesting that canonical dense acoustic fields can be generated from sparse evidence (Erdem et al., 11 May 2026). Although that paper does not define a matching system directly, a plausible implication is that such latent or reconstructed field representations could become the basis of future sound-field similarity and retrieval models.

Taken together, the literature indicates that a sound matching model is best understood not as a single architecture class but as a principled correspondence mechanism between a target acoustic specification and a predicted acoustic output. The specification may be spatial, perceptual, symbolic, visual, linguistic, or statistical; the output may be a field, a parameter vector, a reverberated waveform, a separated source, a localized direction, or an anomaly score. What unifies these systems is the attempt to make “match” itself operational through an explicit objective aligned with the physics, perception, or semantics of the task (Koyama et al., 2022, Han et al., 2023, Yuan et al., 2024).