PeakNetFP: Sparse Neural Audio Fingerprinting

• The paper introduces PeakNetFP, a neural audio fingerprinting framework that leverages sparse spectral peaks to achieve robust retrieval even under extreme time stretching.
• The system employs a PointNet++-style encoder to hierarchically extract features from 3D spectral peak coordinates, significantly reducing input size and memory requirements.
• Contrastive training with NT-Xent loss enables the generation of compact 128-dimensional fingerprints, maintaining over 90% Top-1 hit rate across a broad tempo range.

PeakNetFP is a neural audio fingerprinting (AFP) system designed around sparse spectral peaks rather than dense spectrogram patches. Introduced in "PeakNetFP: Peak-based Neural Audio Fingerprinting Robust to Extreme Time Stretching" (Cortès-Sebastià et al., 26 Jun 2025), it combines peak-based AFP, PointNet++-style hierarchical point-feature extraction, and NeuralFP-style contrastive learning to produce compact fingerprint embeddings that remain robust under severe tempo modification. The central claim of the method is that it preserves the lightweight, sparse, and privacy-friendlier character of traditional peak-based pipelines while approaching the retrieval robustness of dense neural AFP, particularly for time stretching from $0.5\times$ to $2\times$ (Cortès-Sebastià et al., 26 Jun 2025).

1. Problem domain and design objective

Audio fingerprinting identifies a query excerpt by matching it against a reference database. PeakNetFP targets a failure mode that is especially difficult for conventional peak-based AFP: time stretching, defined as changing tempo without changing pitch. The paper focuses on stretch factors from $0.5\times$ to $2.0\times$, corresponding to half speed through double speed, and frames this regime as relevant to DJs, remixes, mashups, licensing circumvention, and broadcast audio (Cortès-Sebastià et al., 26 Jun 2025).

Classical peak-based systems such as Shazam-style methods, Panako, and QuadFP are described as efficient and scalable, but under strong stretching their geometric peak relationships drift, fewer valid peak constellations survive, and matching degrades sharply. PeakNetFP addresses this by avoiding brittle hand-crafted hashing rules alone. Instead, it keeps the sparse peak representation, learns a neural embedding over peaks, and uses contrastive learning so that stretched and unstretched versions of the same audio are mapped nearby in embedding space (Cortès-Sebastià et al., 26 Jun 2025).

The paper characterizes this as a hybridization of three lines of work: traditional peak-based AFP, neural AFP, and point-cloud learning. This suggests that PeakNetFP is best understood not as a replacement for peak extraction, but as a learned representation layer built on top of a peak pipeline that would already be familiar in industrial AFP settings.

2. Sparse spectral representation

PeakNetFP operates on sparse spectral coordinates extracted from a mel-spectrogram. For each 1-second segment, the system extracts local maxima using a $3 \times 3$ kernel with stride 1 and retains the 256 highest-amplitude peaks per second. These peaks are described in the paper as 3-dimensional coordinates, used directly as input in place of dense spectrogram patches (Cortès-Sebastià et al., 26 Jun 2025).

This input choice is the defining representational difference between PeakNetFP and dense neural AFP. QuadFP also uses peaks directly, but does so in a rule-based hashing scheme; PeakNetFP uses the same sparse peaks as the substrate for a learned encoder. The paper emphasizes several consequences of this representation: less data to transmit from client to server, reduced memory and inference cost, more privacy-preserving behavior than dense spectrograms, and compatibility with systems that already precompute peaks (Cortès-Sebastià et al., 26 Jun 2025).

A key reported quantitative comparison is input size. Dense spectrograms are described as roughly $11\times$ larger than the sparse 256-peak input used by PeakNetFP. This is not merely a storage observation; it also conditions the architecture. Because the input is sparse, irregularly distributed, and unordered, the encoder cannot assume the regular grid structure exploited by CNN-based spectrogram models.

3. PointNet++-style architecture

PeakNetFP’s encoder is explicitly inspired by PointNet++, treating spectral peaks as a point cloud in time-frequency space. The paper states that the network uses hierarchical point feature extraction with two Set Abstraction (SA) layers employing Multi-Scale Grouping (MSG), followed by a final global abstraction that yields a 128-dimensional fingerprint vector (Cortès-Sebastià et al., 26 Jun 2025).

At each SA layer, the model performs sampling, grouping plus feature extraction, and multi-scale grouping. Sampling selects the top-$N$ peaks by amplitude as anchor points, with $N^{(1)} = 200$ in the first SA layer and $N^{(2)} = 100$ in the second. Grouping is implemented through a query ball that collects nearby peaks within a radius, and each local neighborhood is processed by a 3-layer MLP and summarized by max pooling. Each SA layer contains three parallel grouping branches with different radii and group sizes, so the network can learn local structure at multiple scales simultaneously (Cortès-Sebastià et al., 26 Jun 2025).

Layer	Reported parameters	Output role
SA + MSG 1	$N=200$; $2\times$0; $2\times$1; MLPs $2\times$2, $2\times$3, $2\times$4	First-stage local and multi-scale feature extraction
SA + MSG 2	$2\times$5; $2\times$6; $2\times$7; MLPs $2\times$8, $2\times$9, $0.5\times$0	Deeper regional aggregation
Final SA	MLP $0.5\times$1	Global 128-dimensional fingerprint vector

The rationale given in the paper is that peaks are sparse, irregularly distributed, and unordered, so a point-set encoder is a more natural fit than a grid-based CNN. The hierarchical, multi-scale construction is presented as important for time-stretch robustness because exact local spacing may vary under stretching, while broader structural regularities can remain informative (Cortès-Sebastià et al., 26 Jun 2025).

4. Contrastive training and retrieval pipeline

PeakNetFP uses the same contrastive learning framework as NeuralFP, but replaces dense spectrogram input and a CNN encoder with sparse peak coordinates and a PointNet++-style point-set encoder. Training operates on 1-second windows with 50% overlap. Positive pairs are constructed by applying time stretching to audio snippets, and each mini-batch contains $0.5\times$2 samples together with their $0.5\times$3 augmented replicas, for a total size of $0.5\times$4 (Cortès-Sebastià et al., 26 Jun 2025).

The training objective is the normalized temperature-scaled cross-entropy loss (NT-Xent), with pairwise similarity defined by inner product:

$0.5\times$5

where

$0.5\times$6

and the full loss averages both directions across positive pairs:

$0.5\times$7

(Cortès-Sebastià et al., 26 Jun 2025)

At inference time, embeddings are indexed with Faiss IVFPQ, the top 20 candidates are retrieved, and sequence matching ranks query-candidate pairs by inner product. The paper therefore places PeakNetFP within the same retrieval paradigm as NeuralFP while changing the representational substrate and encoder family. A plausible implication is that the method is intended to preserve operational compatibility with neural AFP retrieval workflows even as it reduces input and model size.

5. Empirical performance under extreme stretching

The evaluation compares PeakNetFP with QuadFP, identified as the strongest conventional peak-based baseline for time stretching, and with NeuralFP, identified as the spectrogram-based neural state of the art in the paper’s framing. The reported setup uses Test-Query/DB tracks, a dummy reference database of 100,000 tracks, query lengths of 2, 3, 5, 6, and 10 seconds, and stretch factors including speed-up values $0.5\times$8 and slow-down values $0.5\times$9 (Cortès-Sebastià et al., 26 Jun 2025).

The key metric is Top-1 hit rate, denoted $2.0\times$0. Because the neural systems always return a match, the paper notes that in this setup $2.0\times$1 is effectively both precision and recall. PeakNetFP’s headline result is that it maintains over 90% Top-1 hit rate across the full stretch range $2.0\times$2 to $2.0\times$3. For 10-second queries in the commonly studied range $2.0\times$4 to $2.0\times$5, it achieves over 98% $2.0\times$6. Under more extreme stretching, performance decreases somewhat but remains above 90% $2.0\times$7 (Cortès-Sebastià et al., 26 Jun 2025).

Against QuadFP, the improvement is large at the extremes. The paper reports that QuadFP collapses badly under severe stretch and reaches only 3.6% $2.0\times$8 at factor $2.0\times$9. Against NeuralFP, PeakNetFP is described as very close: for stretch factors $3 \times 3$0 to $3 \times 3$1, the difference is at most $3 \times 3$2 $3 \times 3$3, and at the most extreme factor $3 \times 3$4, the maximum gap is 1.85% $3 \times 3$5 in favor of NeuralFP (Cortès-Sebastià et al., 26 Jun 2025).

The paper also identifies an important limitation. PeakNetFP requires at least 5-second queries to keep performance consistently above 0.9 under extreme stretch factors. This constrains its operating regime: it is strong for short-to-medium query-by-example, but the reported evidence does not support the same robustness for very short queries under the harshest tempo deformations.

6. Efficiency profile, claimed contributions, and disambiguation

One of PeakNetFP’s strongest reported advantages is efficiency. The model has 169k trainable parameters, compared with 16.9M for NeuralFP, corresponding to $3 \times 3$6 fewer parameters. Its sparse input is described as $3 \times 3$7 smaller than NeuralFP’s $3 \times 3$8 spectrogram input. The paper also reports lower inference memory usage, measured with batch size 125 on a single RTX 3090: 800 MiB for PeakNetFP and 2338 MiB for NeuralFP (Cortès-Sebastià et al., 26 Jun 2025).

These measurements support the paper’s broader positioning of PeakNetFP as easier to deploy, more scalable for catalog embedding generation, more suitable for client/server setups with limited bandwidth, and attractive for AFP pipelines where sparse peaks are already available. The paper correspondingly presents several firsts or contributions: the first neural AFP built specifically around spectral peaks, the first use of a point-cloud network for AFP, and a hybrid system that merges classical sparse peak input, PointNet++ hierarchical point-feature learning, and NeuralFP-style contrastive training (Cortès-Sebastià et al., 26 Jun 2025).

PeakNetFP should be distinguished from the unrelated XFEL serial crystallography system PeakNet. "PeakNet: An Autonomous Bragg Peak Finder with Deep Neural Networks" (Wang et al., 2023) concerns autonomous Bragg peak finding in diffraction images, not audio fingerprinting. In that crystallography paper, the term “PeakNetFP” does not appear explicitly; the safest inference there is that it would refer to false-positive reduction behavior or to an informal variant label rather than to a separately defined architecture (Wang et al., 2023). In current usage, PeakNetFP denotes the audio fingerprinting model introduced in 2025 (Cortès-Sebastià et al., 26 Jun 2025).

Overall, PeakNetFP occupies a specific position in AFP research: it is a sparse-input neural embedding system optimized for robustness to extreme time stretching. Its reported significance lies in showing that a peak-based representation, when paired with point-set learning and contrastive training, can closely approach dense neural AFP on a demanding deformation class while materially reducing parameter count, input size, and memory footprint (Cortès-Sebastià et al., 26 Jun 2025).