Deep Positive-Negative Prototype (DPNP)

• DPNP is a discriminative prototype-based learning approach that models class prototypes as both latent anchors and classifier weights for enhanced interpretability.
• It employs a composite loss combining cross-entropy, prototype alignment, and negative repulsion to achieve tight intra-class clustering and clear inter-class separation.
• The framework extends to adversarial robustness and multi-label settings, demonstrating empirical gains in accuracy and geometric regularity across standard datasets.

The Deep Positive-Negative Prototype (DPNP) framework is a class of discriminative prototype-based learning models that unify the interpretability and geometric structure of prototype-based learning with the robust decision boundaries of discriminative classifiers. DPNP achieves this by jointly modeling class prototypes as both the latent space anchors and the classifier weights, while introducing a system of positive and negative prototypes to control intra-class compactness and inter-class separation in the feature space. The model leverages a composite objective function incorporating cross-entropy, prototype alignment, and explicit repulsion between class prototypes. DPNP has been adopted across standard and adversarially robust classification settings, has empirical superiority over previous prototype-based and center-loss methods, and underlies advances in interpretable and robust deep learning (Zarei-Sabzevar et al., 5 Jan 2025, Sabzevar et al., 3 Apr 2025, Yang et al., 2019).

1. Model Architecture and Core Definitions

DPNP models a $D$-class classification problem by defining a nonlinear feature extractor $f(x; \theta) \in \mathbb{R}^d$, where $\theta$ are trainable network parameters, and a collection of $M$ class prototypes $\{c_j\}_{j=1}^M$ with $c_j \in \mathbb{R}^d$. These prototypes satisfy a norm constraint $\|c_j\|_2 = \alpha$ for fixed $\alpha$, so all prototypes lie on a hypersphere. The core architectural choices are:

• Prototype–Weight Unification: Each $c_j$ plays a dual role as the $j$-th classifier weight vector in the final linear layer and as the latent-space center (positive prototype) for class $f(x; \theta) \in \mathbb{R}^d$0.
• Negative Prototypes: For each class $f(x; \theta) \in \mathbb{R}^d$1, the negative prototype $f(x; \theta) \in \mathbb{R}^d$2 is defined as its nearest rival in prototype space:

$f(x; \theta) \in \mathbb{R}^d$3

• Softmax Class Score: The class $f(x; \theta) \in \mathbb{R}^d$4 probability for input $f(x; \theta) \in \mathbb{R}^d$5 is computed as

$f(x; \theta) \in \mathbb{R}^d$6

forming the basis for standard cross-entropy classification.

This unified structure facilitates geometric interpretability: prototypes occupy nearly regular simplex positions on the hypersphere, tightly clustered per class, and maximally separated (Zarei-Sabzevar et al., 5 Jan 2025, Sabzevar et al., 3 Apr 2025).

2. Loss Functions and Latent Space Geometry

The DPNP objective integrates several losses to optimize latent space geometry:

• Positive Prototype Alignment Loss (Pulling):

$f(x; \theta) \in \mathbb{R}^d$7

pulling each sample towards its class prototype.

• Cross-Entropy Loss (Classification):

$f(x; \theta) \in \mathbb{R}^d$8

• Negative Prototype Repulsion Losses:
  • Class-Level Repulsion (inter-prototype push):

  $f(x; \theta) \in \mathbb{R}^d$9 - Sample-Level Repulsion (feature to nearest negative prototype):

  $\theta$0

The $\theta$1 pseudo-norm strongly penalizes small inter-prototype distances, maximizing angular margins.

• Total Loss:

$\theta$2

where the $\theta$3’s balance the respective terms (Zarei-Sabzevar et al., 5 Jan 2025).

This loss yields a nearly regular spatial configuration among class centers and tight intra-class clustering, leading to strong inter-class separation and compactness metrics.

3. Training Protocols and Implementation

DPNP training combines prototype-centered and feature-based updates within a standard deep learning pipeline:

• Prototype Renormalization: At each epoch, all $\theta$4 are projected back to the hypersphere $\theta$5.

• Minibatch Training: For each minibatch, latent representations $\theta$6 are computed. For every feature, the nearest negative prototype is identified by exhaustive search over the remaining class centers.

• Joint Gradient Updates:

  • The feature extractor parameters $\theta$7 are updated via SGD on $\theta$8.
  • The prototypes $\theta$9 are updated via gradient steps restricted to loss components not involving adversarial examples, if used.
• Hyperparameters: Key parameters include prototype radius $M$0 (e.g., $M$1), loss weights (e.g., $M$2 in standard ResNet-18), learning rates ($M$3 for $M$4, $M$5 for $M$6), and batch size.

Architectural adaptations include both high-dimensional (e.g., $M$7) and very low-dimensional (e.g., $M$8) embeddings, with prototype geometry remaining stable even in reduced spaces. The computational cost of nearest-neighbor search among prototypes grows with class count $M$9 (Zarei-Sabzevar et al., 5 Jan 2025).

4. Adversarial Robustness: Adv-DPNP Extension

Adv-DPNP is an adversarially robust extension of DPNP, incorporating a dual-branch training regime (Sabzevar et al., 3 Apr 2025):

• Clean and Adversarial Branches: Clean examples update both $\{c_j\}_{j=1}^M$0 and prototypes $\{c_j\}_{j=1}^M$1, whereas adversarial examples ($\{c_j\}_{j=1}^M$2; e.g., PGD attack) update $\{c_j\}_{j=1}^M$3 only. This prevents drift of class anchors under attack.
• Consistency Regularization: A KL-divergence penalty

$\{c_j\}_{j=1}^M$4

enforces invariance of model predictions to adversarial perturbations.

• Composite Loss: The overall loss for a batch becomes

$\{c_j\}_{j=1}^M$5

with $\{c_j\}_{j=1}^M$6 mixing cross-entropy and prototype alignment.

This approach maintains prototype stability, preserves intra-class compactness, and maximizes inter-class margins even under attack, resulting in improved clean and robust accuracy relative to current adversarial-training baselines (Sabzevar et al., 3 Apr 2025).

5. Empirical Performance and Latent Geometry

DPNP and Adv-DPNP have demonstrated empirically superior performance across standard datasets:

Dataset	Model	Accuracy	Inter-Class Margin (MinSep)	SCR
CIFAR-10-512	DPNP	95.40%	91.8°	2.03
CIFAR-100	DPNP	79.01%	–	–
Flowers102	DPNP	95.18%	–	–
CIFAR-10-3	DPNP	94.18%	–	–

Compared to cross-entropy, center loss, and other prototype-based baselines, DPNP achieves:

• Higher test accuracy (by 0.4–0.6%)
• Larger inter-class angular margins (by 8–10 degrees)
• Greater separation-to-compactness ratios (SCR), indicating Fisher-like discrimination
• Robustness to adversarial and common corruptions when using Adv-DPNP

Reduced-dimensional models (e.g., embeddings in $\{c_j\}_{j=1}^M$7) maintain regular prototype arrangements and competitive accuracy, confirming the geometric efficacy of the approach (Zarei-Sabzevar et al., 5 Jan 2025, Sabzevar et al., 3 Apr 2025).

6. Extensions: Multi-Label and Metric Learning Variants

DPNP generalizes to multi-label settings via the construction of multiple positive and negative prototypes for each label, as in the Prototypical Networks for Multi-Label Learning (PNML) approach (Yang et al., 2019):

• Embeddings: Learned via a single fully-connected layer with LeakyReLU.
• Clusters/prototypes: For each label, means of positive and negative clusters are constructed either in a single-prototype or Dirichlet Process (DP) multi-prototype regime.
• Membership Probability: Two-way softmax over distances to positive and negative prototypes with a learned Mahalanobis metric.
• Losses: Include cross-entropy, metric norm regularization, and a correlation term encouraging similar labels to have close prototypes.

PNML achieves state-of-the-art multi-label performance (best average rank on 5 of 5 metrics in 73% of evaluated cases), with notable gains on rare label detection (macro-F$\{c_j\}_{j=1}^M$8) due to improved geometric clustering (Yang et al., 2019).

7. Interpretability, Limitations, and Future Directions

Interpretability is a core advantage: each $\{c_j\}_{j=1}^M$9 can be directly visualized as the archetype of its class, and nearest negative prototypes reveal class-confusability structure. The regular geometric arrangement fosters systematic analysis of learned representations (Zarei-Sabzevar et al., 5 Jan 2025).

Limitations and open research questions include:

• Sensitivity to hyperparameter tuning ($c_j \in \mathbb{R}^d$0, loss weights)
• Nearest-neighbor-based negative prototype selection; extensions could incorporate multiple negatives or global repulsion
• Computational cost for large $c_j \in \mathbb{R}^d$1
• Focus so far on image-classification; generalization to imbalanced, sequence, or multi-modal data is not addressed

Future research directions involve adaptive loss weighting, richer metrics (e.g., angular, Mahalanobis), extension to semi-supervised or low-shot learning, open-set recognition, and scaling to very large class counts (Zarei-Sabzevar et al., 5 Jan 2025).

A plausible implication is that DPNP-style architectures could unify prototype-based interpretability with state-of-the-art discriminative and robust learning more broadly, provided architectural and computational constraints are managed.