--- title: Prototype Normalization (ProtoNorm) url: https://www.emergentmind.com/topics/prototype-normalization-protonorm type: topic --- # Prototype Normalization (ProtoNorm) Searching arXiv for papers on "Prototype Normalization" / "ProtoNorm" and closely related usages. Prototype Normalization (ProtoNorm) is not a single universally standardized method but a family of prototype-conditioned normalization strategies whose meaning depends on research context. In the literature provided here, the term ranges from a prototype-based standardization layer placed before \(L_2\) normalization in person search, to a prototype-guided replacement for LayerNorm in time-series Transformer pretraining, to a prototype-geometry optimization procedure in heterogeneous federated learning. A related few-shot learning line does not use the name “ProtoNorm,” but analyzes why normalizing feature embeddings before constructing class prototypes—especially by \(L_2\) normalization—can make prototype classifiers competitive with both Prototypical Networks and retrained linear classifiers [2110.05076]. Conversely, “Proxy Normalization” is a distinct batch-independent normalization method and is not Prototype Normalization in the prototype or centroid sense [2106.03743]. ## 1. Terminological scope and conceptual commonalities “ProtoNorm” appears explicitly in at least three different technical senses in the material considered here. In person search, it is a normalization layer that computes statistics from within-batch identity prototypes rather than directly from raw instance features, and then applies \(L_2\) normalization [2207.10320]. In time-series foundation model pretraining, it is a prototype-guided dynamic normalization mechanism that replaces LayerNorm with a bank of LayerNorm modules selected by nearest learned distribution prototypes [2504.10900]. In heterogeneous federated learning, it is a prototype-based framework that improves class-prototype geometry by Prototype Alignment and Prototype Upscaling [2507.04310]. Despite these differences, the shared motif is the use of prototypes as normalization anchors rather than relying only on raw instance statistics or a single static normalization parameterization. This suggests that ProtoNorm is better understood as a design pattern than as a single operator: prototypes summarize structure—identity structure, latent distribution structure, or class geometry—and the normalization mechanism is conditioned on that summary. A second commonality is that ProtoNorm is typically introduced in response to a failure mode of conventional normalization. In few-shot prototype classification, raw pre-trained features plus a prototype classifier perform substantially worse than ProtoNet and than pre-trained features plus a newly trained linear classifier, and the paper identifies variation in feature norms as a key unmodeled source of error [2110.05076]. In person search, the authors argue that direct \(L_2\) normalization is unreliable when proposal features are not standardized and that BatchNorm is biased by long-tail identity frequency [2207.10320]. In time-series pretraining, the criticism is that standard LayerNorm has one shared learned affine parameterization for all samples despite substantial distribution mismatch across datasets [2504.10900]. In prototype-based federated learning, the weakness is insufficient separation of aggregated global prototypes under data and model heterogeneity [2507.04310]. ## 2. Few-shot prototype classification as a precursor A direct antecedent to ProtoNorm-like reasoning appears in “A Closer Look at Prototype Classifier for Few-shot Image Classification” [2110.05076]. The paper studies few-shot image classification in the standard \(N\)-way \(K\)-shot setting with feature extractor \(\phi:\mathcal{X}\to\mathbb{R}^D\), prototype \[ \overline{\phi(S_c)}=\frac{1}{K}\sum_{x\in S_c}\phi(x), \] and nearest-prototype classification in squared Euclidean distance: \[ \mathcal{M}(\phi, x, S)_c = \frac{\exp\left(-\|\phi(x)-\overline{\phi(S_c)}\|^2\right)} {\sum_{l=1}^N\exp\left(-\|\phi(x)-\overline{\phi(S_l)}\|^2\right)}. \] The core empirical observation is that directly using feature vectors extracted by standard pre-trained models to construct a prototype classifier in meta-testing does not perform as well as Prototypical Networks or training a new linear classifier on the same features. The paper attributes this to a mismatch in training objectives: ProtoNet encourages features to lie close to class means, whereas standard cross-entropy training only requires linear separability after projection by class weights \(W_c\), with \[ p(y=c|x,W,\phi)=\frac{\exp(\phi(x)^\top W_c)} {\sum_{j=1}^N\exp(\phi(x)^\top W_j)}. \] The paper’s theoretical contribution is a generalization bound for a prototypical classifier whose error depends not only on a term related to the ratio of within-class variance to between-class variance, but also on the variance of feature norms: \[ E\mathrm{V}[h_{L2}(\phi(x))] = \frac{4}{K}E_{c\sim\tau}\left[ Var_{x\sim D_c}\left[\|\phi(x)\|^{2}\right] \right]. \] This directly motivates \(L_2\) normalization, \[ \psi_{L2}(\phi(x)) = \frac{\phi(x)}{\|\phi(x)\|}, \] because it makes all feature norms equal to \(1\), driving the variance of norms to zero. The paper evaluates six transformations before constructing prototypes: \(L_2\) normalization, centering+\(L_2\), LDA, EST, EST+\(L_2\), and LDA+\(L_2\). Its strongest general conclusion is that the same performance can be obtained by using the \(L_2\) normalization and minimizing the ratio of the within-class variance to the between-class variance without training a new classifier or meta-learning [2110.05076]. It also states explicitly that there is no named method called “Prototype Normalization” or “ProtoNorm” in that work. The closest interpretation is simply normalizing feature embeddings before constructing class prototypes and classifying by nearest prototype. ## 3. ProtoNorm in person search The explicit term “ProtoNorm” is introduced in “OIMNet++: Prototypical Normalization and Localization-aware Learning for Person Search” [2207.10320]. There, ProtoNorm is a normalization layer inserted before the final \(L_2\) normalization in the projection module. Its purpose is to calibrate features from pedestrian proposals while considering a long-tail distribution of person IDs, thereby enabling \(L_2\)-normalized person representations to be discriminative. The motivation is specific to person search. Proposal features are extracted from detector outputs rather than clean, tightly cropped re-identification images, so they may be misaligned, include background clutter, contain overlapping people, or only partially cover the target person. The paper states that existing person search methods implicitly assume that features obtained from pedestrian proposals have zero-mean and unit variance and then simply apply \(L_2\) normalization; when that assumption fails, \(L_2\) normalization can degrade discriminative power. A straightforward BatchNorm fix is also argued to be inadequate because mean and variance estimated from the current batch are dominated by majority identities in a long-tail distribution, biasing feature calibration toward frequent IDs and weakening discrimination for minority IDs [2207.10320]. ProtoNorm therefore computes its mean from within-batch identity prototypes. For a mini-batch \[ \bm{X}=\{\mathbf{x}^{1},\cdots,\mathbf{x}^{B}\},\qquad Y=\{y^{1},\cdots,y^{B}\}, \] the prototype for identity \(t\) is \[ \mathbf{p}_{t}(d) = \frac{\sum_{b=1}^{B}\bm{X}^{b}(d)\mathds{1}[y^{b}=t]}{\sum_{b=1}^{B}\mathds{1}[y^{b}=t]}. \] If \(K\) is the number of unique IDs in the batch, ProtoNorm computes \[ \bm{\mu}(d)=\frac{1}{K}\sum_{k=1}^{K}\mathbf{p}_{k}(d), \qquad \bm{\sigma}(d)=\sqrt{\frac{1}{B}\sum_{b=1}^{B}(\bm{X}^{b}(d)-\bm{\mu}(d))^{2}}. \] Each feature is then standardized channel-wise and only afterwards projected onto the unit hypersphere by \(L_2\) normalization. Two details distinguish this method from BatchNorm. First, the mean is computed from prototypes rather than raw batch instances, so each identity contributes more equally regardless of occurrence count. Second, the main ProtoNorm used before \(L_2\) normalization omits a learnable affine transform because the authors found that scaling and offset converge to constant and zero values, respectively, and inferred that the effect is canceled out by \(L_2\) normalization [2207.10320]. During inference, labels are unavailable, so ProtoNorm uses running estimates of mean and variance accumulated during training, analogous to BatchNorm. The reported ablation on PRW isolates ProtoNorm’s effect. With OIM only, Search mAP and rank-1 are \(42.0\) and \(80.5\), and ReID mAP and rank-1 are \(44.3\) and \(82.6\). With BatchNorm + OIM, these become \(44.3\), \(81.5\), \(46.6\), and \(83.2\). With ProtoNorm + OIM, they become \(46.3\), \(82.7\), \(48.4\), and \(84.6\) [2207.10320]. The paper also reports that when combined with the localization-aware OIM update, the full OIMNet++ reaches Search mAP \(46.8\), rank-1 \(83.9\), ReID mAP \(49.0\), and rank-1 \(86.2\). ## 4. Prototype-guided normalization in time-series models In time-series research, ProtoNorm denotes a different mechanism. “Bridging Distribution Gaps in Time Series Foundation Model Pretraining with Prototype-Guided Normalization” proposes a domain-aware adaptive normalization strategy within Transformer architectures by replacing LayerNorm with a prototype-guided dynamic normalization mechanism [2504.10900]. The paper’s argument is that multi-dataset time-series pretraining is harmed by substantial distribution mismatch across datasets—differences in value ranges, morphology, temporal dependencies, noise level, sampling rate, channel count, and sequence length—and that standard LayerNorm is too rigid because it has a single shared learned affine parameterization for all samples. Standard LayerNorm is written as \[ LN(x_i;\gamma,\beta)=\gamma\cdot \hat{x}_i+\beta,\qquad \hat{x}_i=\frac{x_i-\mu}{\sqrt{\sigma^2+\epsilon}}, \] with one shared pair \((\gamma,\beta)\) per layer. ProtoNorm instead contains \(n\) LayerNorm modules \(\{LN_1,\dots,LN_n\}\), a prototype-guided gating network \(\mathcal{G}\), and prototypes \(\{p_1,\dots,p_n\}\). For a sample feature representation \(x\), the routing rule is \[ i^*=\arg\min_{i\in\{1,2,\dots,n\}} d(x,p_i), \] and the normalization is best summarized as \[ \mathrm{ProtoNorm}(x)=LN_{i^*}(x). \] The dynamic part is therefore branch selection: the mean and variance remain LN-style per-sample statistics, while the affine parameters are selected by nearest-prototype hard routing. The prototypes are updated by exponential moving average, \[ p_i^{(t+1)}=(1-\alpha)\cdot p_i^{(t)}+\alpha\cdot x, \] and are regularized by an orthogonality loss \[ \mathcal{L}_{\text{orth}}=\|PP^T-I\|_F^2, \] where \(P\in\mathbb{R}^{n\times d}\) is the prototype matrix. In the reported pretraining framework, the total loss is \[ \mathcal{L}=\mathcal{L}_{\text{NT-Xent}}+\lambda\cdot\mathcal{L}_{\text{orth}}, \] with \(\lambda=0.001\) in the main experiments [2504.10900]. The empirical findings are broad. In classification averaged over groups, ProtoN-FM reaches \(63.05\) accuracy and \(53.18\) Macro-F1, compared with \(60.60\) and \(50.53\) for vanilla multi-dataset pretraining. In forecasting on Monash, integrating ProtoNorm into Moirai improves normalized MAE from \(1.00\) to \(0.8893\), reported as an \(11.07\%\) improvement. Under a controlled distribution-shift analysis on IMS-N3, ProtoNorm yields accuracy \(70.54\) versus \(67.05\) for vanilla and Macro-F1 \(66.28\) versus \(63.84\) [2504.10900]. The paper further reports that prototype-guided gating matters: on MFD average, removing ProtoGate reduces performance from \(70.33/67.13\) to \(66.65/62.48\) in accuracy/Macro-F1. A related but differently framed time-series method is APT: Affine Prototype Timestamp [2511.12945]. That paper does not use the label “ProtoNorm,” but structurally it is described as a prototype-conditioned affine normalization/de-normalization layer: timestamp embeddings are matched to a learned prototype library, the top-\(k\) sparse assignment produces a prototype aggregate, and two MLPs generate \(\gamma_t,\beta_t\) used in a forward affine transform before the forecasting backbone and an inverse affine transform after it. This suggests that the time-series literature contains both explicit ProtoNorm methods [2504.10900] and ProtoNorm-like prototype-conditioned normalization modules that are named differently [2511.12945]. ## 5. ProtoNorm in heterogeneous federated learning In “Heterogeneous Federated Learning with Prototype Alignment and Upscaling,” ProtoNorm is a prototype-based heterogeneous federated learning framework designed to improve the discriminative quality of shared class prototypes [2507.04310]. The setting is a central server with \(M\) clients, each with its own private distribution \(P_i\), possibly different model architecture, and a shared classification task over \(K\) classes. The client model has a feature extractor \(f_i\) and classifier \(g_i\), with penultimate-layer feature \(f_i(\boldsymbol{\theta}_i;x)\in\mathbb{R}^d\). Prototype-based federated learning communicates only class prototypes, namely class-wise mean feature vectors from the penultimate layer. The local prototype for class \(j\) on client \(i\) is \[ \bar{\boldsymbol{c}_{i,j}^{L}}= \frac{1}{n_{i,j}} \sum_{(x,y)\in\mathcal{D}_{i,j}} f_i(\boldsymbol{\theta}_i;x). \] ProtoNorm’s critique of prior PBFL methods is that global prototypes obtained by aggregation alone are insufficiently separated under non-IID data and model heterogeneity. Its remedy has two sequential components. Prototype Alignment (PA) first normalizes aggregated global prototypes onto the unit sphere, \[ \hat{\boldsymbol{c}_j}=\frac{\bar{\boldsymbol{c}_j}}{\|\bar{\boldsymbol{c}_j}\|}, \] and then optimizes them using a Thomson-problem-inspired repulsive energy. The surrogate objective used for \(s=1\) is \[ \boldsymbol{E}=\sum_{j