AudioProtoPNet: Prototype-Based Audio Models

• AudioProtoPNet are prototype-based audio models that use learned representations to mediate classification through explicit, audible comparisons.
• They operate in both spectrogram and embedding domains, applying reconstruction error and cosine similarity to facilitate interpretable, case-based reasoning.
• The models integrate adversarial training and robust optimization to maintain reliability and clear evidence paths under distribution shifts.

AudioProtoPNet denotes a line of prototype-based audio models in which prediction is mediated by explicit comparisons to learned prototypes rather than by an opaque terminal classifier. In the original formulation, AudioProtoPNet is an audio identification model based on learnable spectral prototypes and prototype-specific transformation networks that reconstruct a log-Mel spectrogram and classify by minimum reconstruction error (Loiseau et al., 2022). Later work reused the name for an adaptation of ProtoPNet to multi-label bird sound classification, combining a ConvNeXt backbone with a prototype learning classifier trained on spectrogram embeddings and yielding prototype-based explanations at inference time (Heinrich et al., 2024). A subsequent study examined the same bird-sound formulation under adversarial training and out-of-distribution evaluation (Heinrich et al., 18 Jul 2025). Related work on controllable music recommendation employs an audio prototypical network over audio embeddings, but this system is presented as APRON rather than AudioProtoPNet (Öncel et al., 31 Jul 2025).

1. Conceptual scope and defining characteristics

The unifying idea across AudioProtoPNet variants is that a model decision should be expressible as a relation between an input audio representation and a finite set of learned prototypes. This makes the model intrinsically interpretable in the sense that the prototypes are part of the forward computation rather than an auxiliary explanation layer. In the 2022 formulation, the prototypes are themselves spectrogram templates and can be inverted to audio and “played” (Loiseau et al., 2022). In the bird-sound formulations, the prototypes live in embedding space and correspond to representative spectro-temporal patches from the training set, supporting “this looks like that” explanations (Heinrich et al., 2024).

A common misconception is that AudioProtoPNet refers to a single fixed architecture. The literature instead uses the name for at least two distinct instantiations. One operates directly in the spectrogram domain with prototype-specific transformation networks and a reconstruction-based distance (Loiseau et al., 2022). Another adapts ProtoPNet to bird sound classification with a ConvNeXt backbone and prototype matching in latent space (Heinrich et al., 2024, Heinrich et al., 18 Jul 2025). The shared commitment is not a particular metric or backbone, but prototype-mediated inference and case-based reasoning.

Another common misconception is that interpretability is purely post-hoc. In all of these formulations, the prototypes directly determine the output: either the input is assigned to the prototype with minimum reconstruction error, or class scores are aggregated from prototype activations. The explanation is therefore structurally coupled to the prediction rather than retrofitted afterward.

2. Original spectrogram-domain formulation

In "A Model You Can Hear: Audio Identification with Playable Prototypes" (Loiseau et al., 2022), the input is a log-Mel spectrogram

$x \in \mathbb{R}^{F \times T},$

and the model learns $K$ spectral prototypes

$p_k \in \mathbb{R}^{F \times T}.$

In practice each $p_k$ is implemented as a learned parameter of dimension $F \times 1$, broadcast to all $T$ frames. For each prototype $k$, a small U-Net style network $T_k(\cdot;\theta_k)$ predicts four per-time-step transformation parameter sets: gain $G_k(x)\in\mathbb{R}^T$, pitch-shift $S_k(x)\in\mathbb{R}^T$, low-frequency filter $K$0, and high-frequency filter $K$1 (Loiseau et al., 2022).

These transformations are applied in sequence to reconstruct the input from the prototype. The full reconstruction for prototype $K$2 is

$K$3

with the elementary spectral transforms defined as gain, pitch-shift, low-frequency filter, and high-frequency filter operations in the spectrogram domain. The reconstruction error is the average frame-wise $K$4 distance: $K$5

Classification can then proceed either by minimum reconstruction error,

$K$6

or by a softmax over negative reconstruction errors,

$K$7

with inverse-temperature $K$8. The model supports both unsupervised clustering and supervised classification. The unsupervised objective is

$K$9

and the supervised objective combines correct-prototype reconstruction loss with a cross-entropy term on the softmax of negative reconstruction errors: $p_k \in \mathbb{R}^{F \times T}.$0 with $p_k \in \mathbb{R}^{F \times T}.$1, learned $p_k \in \mathbb{R}^{F \times T}.$2, and no additional triplet or contrastive terms.

The decisive architectural feature is that each prototype is itself a log-Mel spectrogram. Because it can be inverted to audio, the model’s internal reference points are not merely vectors in latent space but audible exemplars. This grounds the term “playable prototypes” in a literal sense.

3. ProtoPNet-style bird sound classification variants

The 2024 bird-sound adaptation, "AudioProtoPNet: An interpretable deep learning model for bird sound classification" (Heinrich et al., 2024), places the prototype mechanism on top of a ConvNeXt embedding network. The model is structured as

$p_k \in \mathbb{R}^{F \times T}.$3

where $p_k \in \mathbb{R}^{F \times T}.$4 maps an input spectrogram $p_k \in \mathbb{R}^{F \times T}.$5 to an embedding tensor $p_k \in \mathbb{R}^{F \times T}.$6, $p_k \in \mathbb{R}^{F \times T}.$7 is a prototype-matching layer with $p_k \in \mathbb{R}^{F \times T}.$8 learned prototypes, and $p_k \in \mathbb{R}^{F \times T}.$9 is a final fully-connected classification layer. Each prototype $p_k$0 is associated with class $p_k$1, and in practice $p_k$2, so each prototype is a $p_k$3-dimensional vector.

Similarity is computed by cosine matching between $p_k$4-normalized embedding patches and $p_k$5-normalized prototypes: $p_k$6 The final layer uses a fixed weight matrix so that each prototype votes only for its own class: $p_k$7 Training uses a weighted sum of asymmetric classification loss, prototype clustering loss, prototype separation loss, and an orthogonality loss: $p_k$8 The training schedule is two-phase: first the ConvNeXt backbone is frozen and only the prototypes are trained; then the backbone and prototypes are jointly fine-tuned.

The 2025 robustness study adopts a related AudioProtoPNet for bird-sound recognition with a ConvNeXt-B backbone pre-trained on ImageNet and fine-tuned on spectrogram inputs (Heinrich et al., 18 Jul 2025). Here the embedding extractor outputs

$p_k$9

with $F \times 1$0, $F \times 1$1, and $F \times 1$2. The prototype layer contains $F \times 1$3 learnable prototype vectors $F \times 1$4, with $F \times 1$5 prototypes per class. Distances are computed patchwise,

$F \times 1$6

and converted into similarity activations by negative patch-max-pool,

$F \times 1$7

A fully-connected matrix $F \times 1$8 then maps prototype activations to logits,

$F \times 1$9

The coexistence of cosine-similarity and squared-distance instantiations should not be read as a contradiction. It indicates that the AudioProtoPNet label spans multiple prototype-based realizations of inherently interpretable audio classification, unified by prototype matching, class-linked activations, and inspectable evidence paths.

4. Training objectives and optimization regimes

The optimization strategy depends on the variant. In the original spectrogram-domain model, learning is reconstruction-centric. Unsupervised learning minimizes the best achievable reconstruction across prototypes, which is formally analogous to a k-means style objective in the spectrogram domain. Supervised learning augments that reconstruction term with cross-entropy on the softmax of negative reconstruction errors, while still keeping the reconstruction pathway central to classification (Loiseau et al., 2022).

In the 2024 bird-sound model, the objective is explicitly multi-label and prototype-structuring. The asymmetric loss handles label imbalance, the clustering loss encourages at least one active prototype for a positive class, the separation loss pushes embeddings away from prototypes of negative classes, and the orthogonality loss decorrelates prototypes within each class (Heinrich et al., 2024). This means that prototype learning is not merely a representational convenience; it is directly regularized to produce distinct, class-informative evidence.

In the 2025 robustness study, ordinary training minimizes a composite loss comprising asymmetric multi-label classification loss $T$0, prototype projection losses $T$1, and weight decay regularization: $T$2 with $T$3 and $T$4 (Heinrich et al., 18 Jul 2025). Training alternates between a prototype projection step, which re-assigns each prototype to its closest latent patch of its class in the current mini-batch, and a joint fine-tuning step that backpropagates the full loss with respect to $T$5, $T$6, $T$7, and $T$8.

That same study further extends TRADES–AWP to AudioProtoPNet through two adversarial training procedures. Output-space adversarial training (AT-O) uses FGSM on the classification loss to generate $T$9, whereas embedding-space adversarial training (AT-E) uses FGSM on average spatial cosine distance in the embedding extractor to generate $k$0. The reported hyperparameters are $k$1, $k$2, AWP budget $k$3, and warm-up of 8 epochs (Heinrich et al., 18 Jul 2025). This is important because it shows that prototype-based interpretability and adversarial training are not treated as mutually exclusive design goals.

5. Interpretability and explanation semantics

Interpretability in AudioProtoPNet is implemented through prototypes as explicit evidence carriers. In the original model, each $k$4 is a log-Mel spectrogram that can be inverted to audio, and the explanation for a decision reduces to which prototype best reconstructs the input and which learned gain, pitch, and spectral-envelope transformations were required (Loiseau et al., 2022). The model therefore exposes both a reference exemplar and a deformation pathway.

In the bird-sound formulations, prototypes correspond to representative spectro-temporal patches from the training data rather than directly playable spectrogram templates. For the 2024 model, one can identify the prototypes with highest activation for a predicted class, visualize the training spectrogram patch onto which each prototype was projected, and overlay an up-sampled activation map on the test spectrogram to show which time-frequency region triggered the match (Heinrich et al., 2024). This yields a case-based rationale of the form “this time-frequency patch looks like that prototype.”

The 2025 study makes the explanation structure explicit by distinguishing local and global explanations. Local explanations are of the form “This 5s spectrogram patch activates prototype $k$5,” whereas global explanations are given by the linear weights $k$6, which indicate how much each prototype votes for class $k$7 (Heinrich et al., 18 Jul 2025). Because each prototype is anchored to an actual training spectrogram patch, the explanation remains human-readable at the level of training exemplars.

A further misconception is that prototype models are automatically stable under perturbation. The robustness study specifically evaluates prototype stability under targeted embedding-space attacks and finds that stability depends strongly on training regime (Heinrich et al., 18 Jul 2025). This indicates that interpretability in the prototype sense does not, by itself, guarantee robustness; it must be supported by the optimization procedure.

6. Empirical results, robustness evidence, and related extensions

The original AudioProtoPNet was evaluated on SOL, with 33 instruments and 24 450 single-note spectrograms, and on LibriSpeech, with 128 speakers (Loiseau et al., 2022). In unsupervised clustering, AudioProtoPNet achieved OA $k$8 and AA $k$9 on SOL, and OA $T_k(\cdot;\theta_k)$0 and AA $T_k(\cdot;\theta_k)$1 on LibriSpeech. In supervised classification, the reported results were OA $T_k(\cdot;\theta_k)$2 and AA $T_k(\cdot;\theta_k)$3 on SOL, and OA $T_k(\cdot;\theta_k)$4 and AA $T_k(\cdot;\theta_k)$5 on LibriSpeech, exceeding both direct CNN classification and APNet in those experiments.

In bird sound classification, the 2024 AudioProtoPNet was trained on the BirdSet training dataset, described as consisting of 9,734 bird species and over 6,800 hours of recordings, and evaluated on the seven test datasets of BirdSet, covering different geographical regions (Heinrich et al., 2024). It outperformed Perch, achieving an average AUROC of 0.90 and a cmAP of 0.42, with relative improvements of 7.1% and 16.7%, respectively. These figures situate prototype-based interpretability within a high-capacity, large-scale multi-label setting rather than a narrowly constrained benchmark.

The 2025 robustness study reports clean-data cmAP on one validation plus seven OOD soundscape datasets (Heinrich et al., 18 Jul 2025). For AudioProtoPNet, ordinary training (OT) yielded 0.49 on POW and 0.38 mean over 7 test sets; AT-E yielded 0.57 $T_k(\cdot;\theta_k)$6 and 0.41 $T_k(\cdot;\theta_k)$7; and AT-O yielded 0.59 $T_k(\cdot;\theta_k)$8 and 0.42 $T_k(\cdot;\theta_k)$9. Against embedding-space PGD at $G_k(x)\in\mathbb{R}^T$0, the reported PRS values were approximately 0.00 for OT, 0.01 for AT-E, and 0.35 for AT-O. Against output-space PGD at $G_k(x)\in\mathbb{R}^T$1, the corresponding PRS values were approximately 0.03, 0.09, and 0.15. Under targeted attacks that align $G_k(x)\in\mathbb{R}^T$2 with a random target prototype, TARS at $G_k(x)\in\mathbb{R}^T$3 was 0.27 for OT, 0.37 for AT-E, and 0.65 for AT-O. This indicates that adversarial training, particularly output-space adversarial training, can simultaneously improve clean-data generalization under distribution shift and stabilize prototype-based explanations.

A neighboring but distinct development is APRON, the "Audio Prototypical Network For Controllable Music Recommendation" (Öncel et al., 31 Jul 2025). APRON relies on MERT-v1-330M as a fixed front-end, uses 1024-dimensional audio embeddings, selects $G_k(x)\in\mathbb{R}^T$4 common song-level Last.fm tags as prototypes, and constructs a scrutable user profile

$G_k(x)\in\mathbb{R}^T$5

It is trained with recommendation, controllability, and prototype-separability losses, and supports direct editing of prototype weights to steer recommendations. On the Million Song Dataset, APRON with $G_k(x)\in\mathbb{R}^T$6 heads achieved Recall@20 $G_k(x)\in\mathbb{R}^T$7, Recall@50 $G_k(x)\in\mathbb{R}^T$8, NDCG@100 $G_k(x)\in\mathbb{R}^T$9, and average $S_k(x)\in\mathbb{R}^T$0 for controllability. Its stated limitations are that it requires access to all audio, relies on the quality and coverage of the chosen tags, has no user study yet, and uses a fixed set of 80 prototypes. This suggests that prototype-based audio modeling has broadened from interpretable classification toward controllable user-facing systems, while retaining the central idea that semantically inspectable prototypes should mediate prediction.