Tripartite Weight-Space Ensemble (Tri-WE)

• The paper introduces Tri-WE, which combines base, previous, and current classifier states to mitigate catastrophic forgetting and overfitting in FSCIL.
• The method leverages amplified data knowledge distillation with MixUp and CutMix to refine feature representations from limited few-shot samples.
• Empirical results on miniImageNet, CIFAR100, and CUB200 demonstrate that Tri-WE enables full-model updates while outperforming traditional fixed-feature strategies.

Searching arXiv for the named papers and closely related FSCIL context. Tripartite Weight-Space Ensemble, usually abbreviated Tri-WE, is a weight-space interpolation method for few-shot class-incremental learning (FSCIL) in which the deployed classifier is constructed from three temporally distinct classifier states: the base model, the immediately previous session model, and the current model being learned (Lee et al., 4 Jun 2025). The method was introduced to address the coupled FSCIL problems of catastrophic forgetting and overfitting under severe data scarcity while permitting updates to the entire model, rather than freezing the feature extractor after base training. In this formulation, Tri-WE is paired with amplified data knowledge distillation (ADKD), a regularizer that distills both features and logits from the previous model using mixed few-shot inputs (Lee et al., 4 Jun 2025). The phrase “tripartite weight-space” also appears in coding theory, where it denotes a three-stratum nonzero weight distribution in certain $p$-ary linear codes derived from inhomogeneous quadratic forms (Hu et al., 2023). In current machine-learning usage, however, “Tripartite Weight-Space Ensemble” refers specifically to the FSCIL method of classifier interpolation (Lee et al., 4 Jun 2025).

1. Problem setting and motivation

Tri-WE is formulated in the FSCIL setting. A model is first trained in a base session on many base classes with ample labeled data, and then proceeds through a sequence of incremental sessions, each introducing a small number of new classes with only a few labeled examples per class (Lee et al., 4 Jun 2025). The model is written as

$f_{\theta,\phi}=h_\phi \circ g_\theta,$

where $g_\theta$ is the feature extractor and $h_\phi$ is the classification head, specifically a cosine classifier (Lee et al., 4 Jun 2025).

At session $t$, the learner has access only to the current few-shot data $D^{(t)}=\{(x,y)\}$ and to a prototype buffer $\mathcal{M}$ containing one prototype per previously seen class, where a prototype is the average feature of the $K$ examples of a class (Lee et al., 4 Jun 2025). Evaluation after each session is performed on all classes seen so far, not only on the newly introduced classes (Lee et al., 4 Jun 2025). The method is therefore designed for a setting in which old training images are unavailable and memory is restricted to one prototype per old class.

The paper identifies two coupled difficulties. Catastrophic forgetting arises because updating the model on new classes can shift decision boundaries and feature representations away from old classes when old images are not available. Overfitting arises because each new class has only a few examples, so direct parameter updates can fit noise or idiosyncrasies of those few samples (Lee et al., 4 Jun 2025). A common modern FSCIL strategy is to train a strong feature extractor on base classes, freeze it after base training, and update only the classifier for new classes. The critique advanced by Tri-WE is that a fixed feature extractor restricts adaptability to new classes (Lee et al., 4 Jun 2025).

The central thesis is therefore that full-model updating should remain available during FSCIL, but only if forgetting and overfitting are controlled sufficiently strongly. Tri-WE provides the principal stabilization mechanism on the classifier side, while ADKD regularizes the representation update (Lee et al., 4 Jun 2025).

2. Formal structure of Tri-WE

Tri-WE performs weight-space interpolation over classifier parameters rather than prediction-space ensembling at inference time (Lee et al., 4 Jun 2025). The final deployed model remains a single classifier head, not a committee of multiple heads evaluated in parallel. At incremental session $t$, the three relevant classifier states are:

• $\phi_0$: the classification head from the base session model;
• $f_{\theta,\phi}=h_\phi \circ g_\theta,$0: the old-class-specialized head initialized from the previous deployed classifier and adapted to old classes $f_{\theta,\phi}=h_\phi \circ g_\theta,$1;
• $f_{\theta,\phi}=h_\phi \circ g_\theta,$2: the current all-class head covering classes $f_{\theta,\phi}=h_\phi \circ g_\theta,$3 (Lee et al., 4 Jun 2025).

The piecewise definition of the final classifier weight for class $f_{\theta,\phi}=h_\phi \circ g_\theta,$4 is

$f_{\theta,\phi}=h_\phi \circ g_\theta,$5

where $f_{\theta,\phi}=h_\phi \circ g_\theta,$6, $f_{\theta,\phi}=h_\phi \circ g_\theta,$7, and $f_{\theta,\phi}=h_\phi \circ g_\theta,$8 is the classifier dimensionality (Lee et al., 4 Jun 2025).

The normalized interpolation coefficients are computed from two learnable scalars $f_{\theta,\phi}=h_\phi \circ g_\theta,$9:

$g_\theta$0

$g_\theta$1

When $g_\theta$2, $g_\theta$3, and the method sets $g_\theta$4 (Lee et al., 4 Jun 2025).

This class-conditional construction yields a differentiated temporal prior. For base classes, the classifier is anchored jointly to the base, previous, and current heads. For previously added incremental classes, the base head is omitted and interpolation occurs between previous and current heads. For current new classes, only the current head is used (Lee et al., 4 Jun 2025). This design reflects the distinct informational roles of the three sources: the base model preserves the original many-shot geometry, the previous model preserves all classes seen through session $g_\theta$5, and the current model supplies plasticity for the new session (Lee et al., 4 Jun 2025).

A common misconception is to treat Tri-WE as an ordinary test-time ensemble. That is not the formulation given in the paper. The combination occurs directly in weight space, and deployment uses only one backbone and one interpolated head (Lee et al., 4 Jun 2025).

3. Incremental-session pipeline and classifier treatment

The base session $g_\theta$6 is trained conventionally on many-shot base data. The method uses ALICE-style base training and additionally employs an auxiliary geometric classification head to improve generalization, though that auxiliary head is used only in the base session (Lee et al., 4 Jun 2025). After base training, the system retains the base feature extractor $g_\theta$7 and base classifier weights $g_\theta$8 (Lee et al., 4 Jun 2025).

For each incremental session $g_\theta$9, $h_\phi$0, the procedure is as follows. The backbone is initialized from the previous deployed model. The method maintains the stored base head $h_\phi$1, initializes $h_\phi$2 from the previous deployed classifier, initializes the old-class portion of $h_\phi$3 from the previous classifier, and adds weights for the current new classes to $h_\phi$4 (Lee et al., 4 Jun 2025). The deployed classifier $h_\phi$5 is then assembled from Eq. (1), and training proceeds with classification and distillation losses. During training, the feature extractor $h_\phi$6, $h_\phi$7, $h_\phi$8, and the interpolation scalars $h_\phi$9 are updated. At deployment, only $t$0 and the ensembled classifier $t$1 are retained (Lee et al., 4 Jun 2025).

The paper makes a deliberately asymmetric design choice: the feature extractor is updated only mildly, whereas the classification head carries the main anti-forgetting machinery (Lee et al., 4 Jun 2025). This is justified by the claim that the feature extractor has many layers and is vulnerable to overfitting or collapse with few examples, whereas classifier weights encode class boundaries most directly (Lee et al., 4 Jun 2025). The paper also reports an ablation in which Tri-WE is applied to convolutional blocks of ResNet18 and finds that this degrades performance; the best variant is classifier-head-only Tri-WE (Lee et al., 4 Jun 2025).

Two classifier states are trainable during incremental learning. The old-class head $t$2 is updated using old prototypes and the old-class loss, while the all-class head $t$3 is updated by the main classification objective. The fixed base head $t$4 is not updated (Lee et al., 4 Jun 2025). This separation is intended to let the old-class representation track changes in the feature extractor without allowing the current few-shot session to dominate all class parameters indiscriminately.

4. Amplified data knowledge distillation and objective function

The second main component is amplified data knowledge distillation (ADKD). The motivation is that standard knowledge distillation is unreliable in FSCIL when the only available real images are the few current training samples (Lee et al., 4 Jun 2025). The method therefore expands the current few-shot set $t$5 into an amplified set $t$6 by randomly mixing sample pairs using simple augmentations such as MixUp and CutMix (Lee et al., 4 Jun 2025). The paper especially favors intermixing methods over single-image augmentation.

ADKD is defined as

$t$7

The feature-level term is

$t$8

This constrains the current backbone to remain compatible with the previous session’s representation geometry on amplified data (Lee et al., 4 Jun 2025).

The excerpt states that the logit-level term is a KL or cross-entropy style distillation from the previous model’s output distribution on old classes to the current model’s output distribution on those same old classes (Lee et al., 4 Jun 2025). The paper therefore uses both feature-level and logit-level distillation; an ablation indicates that feature-level KD is slightly more important, but the combination of both is best (Lee et al., 4 Jun 2025).

The main classification loss is

$t$9

and the old-class classifier loss is

$D^{(t)}=\{(x,y)\}$0

The total loss is

$D^{(t)}=\{(x,y)\}$1

The implementation details later refer to the balancing coefficients as $D^{(t)}=\{(x,y)\}$2 and set them to $D^{(t)}=\{(x,y)\}$3 and $D^{(t)}=\{(x,y)\}$4, respectively, so the paper contains a notation inconsistency between $D^{(t)}=\{(x,y)\}$5 and $D^{(t)}=\{(x,y)\}$6 while preserving the same functional role (Lee et al., 4 Jun 2025).

The implementation uses ResNet18 and SGD. Base-session learning rates are $D^{(t)}=\{(x,y)\}$7 for miniImageNet, $D^{(t)}=\{(x,y)\}$8 for CUB200, and $D^{(t)}=\{(x,y)\}$9 for CIFAR100, with decay by $\mathcal{M}$0 at epochs 60 and 70 (Lee et al., 4 Jun 2025). In incremental sessions, the learning rate for $\mathcal{M}$1 is $\mathcal{M}$2, whereas the learning rate for the rest, including the feature extractor, is $\mathcal{M}$3 (Lee et al., 4 Jun 2025). The amplified data size is increased from $\mathcal{M}$4 examples to $\mathcal{M}$5, and $\mathcal{M}$6 are initialized to $\mathcal{M}$7 (Lee et al., 4 Jun 2025).

5. Experimental protocol and empirical behavior

Tri-WE is evaluated on miniImageNet, CIFAR100, and CUB200, following the standard FSCIL splits from TOPIC (Lee et al., 4 Jun 2025). For miniImageNet and CIFAR100, the protocol uses 60 base classes and 40 remaining classes split into 8 incremental sessions, each 5-way 5-shot. For CUB200, the protocol uses 100 base classes and 100 remaining classes split into 10 sessions, each 10-way 5-shot (Lee et al., 4 Jun 2025). The evaluation metric is Top-1 accuracy on all classes seen so far after each session, together with the average accuracy across sessions (Lee et al., 4 Jun 2025).

On miniImageNet, the reported session accuracies are 84.13, 81.41, 76.65, 73.59, 70.10, 65.13, 63.42, 61.02, and 60.13, with average 70.62 (Lee et al., 4 Jun 2025). The paper compares this with several prior methods and states that the result exceeds the strongest listed baseline average by at least 1.82% on miniImageNet (Lee et al., 4 Jun 2025). It also states that the method consistently surpasses recent SOTA methods on CUB200 and shows similar superiority trends on CIFAR100 (Lee et al., 4 Jun 2025).

A central empirical pattern is that the method remains strong through later sessions, suggesting a more favorable compromise between old-class retention and new-class adaptation than either naive full-model fine-tuning or fixed-feature methods (Lee et al., 4 Jun 2025). The paper also reports a CUB200 analysis showing better harmonic mean of base and novel class accuracies (Lee et al., 4 Jun 2025). This suggests that the interpolation mechanism and distillation regularizer jointly stabilize the old/new tradeoff rather than optimizing only one side of it.

The miniImageNet ablation results are especially informative:

Variant	Average	Last
Naive (no WE, no ADKD)	51.55	16.73
No WE (ADKD only)	67.92	56.01
Dual-WE old + all	69.31	58.93
Dual-WE base + all	68.54	57.31
Tri-WE base + old + all	70.62	60.13

These results show that naive full-model updating collapses badly, ADKD alone rescues much of that degradation, weight-space interpolation yields further improvement, and the three-head version performs best (Lee et al., 4 Jun 2025). The paper explicitly notes that the immediately previous model is more important than the base model among the dual variants, likely because it contains knowledge of incremental classes learned so far (Lee et al., 4 Jun 2025).

Additional ablations reinforce the method’s design decisions. Updating the feature extractor is beneficial, but the gain is larger when $\mathcal{M}$8 is included; freezing $\mathcal{M}$9 yields around 69.4 average regardless of $K$0, whereas updating $K$1 with the old-head loss yields 70.62 (Lee et al., 4 Jun 2025). Applying weight-space ensembling to feature extractor blocks reduces performance relative to head-only interpolation, and for ADKD the intermixing augmentations MixUp and especially CutMix outperform single-image augmentations such as CutOut and RandAug (Lee et al., 4 Jun 2025).

6. Interpretation, scope, and related uses of the term

Conceptually, Tri-WE is closest to model averaging, model soup, or Wise-FT, but it is temporally structured and class-conditional rather than an average over arbitrary checkpoints (Lee et al., 4 Jun 2025). It also resembles EMA in its stabilizing role, but it is not a moving average over optimization time. Unlike a standard prediction ensemble, it does not require multiple forward passes at test time. Unlike rehearsal-heavy continual learning, it stores only one prototype per old class, not old images (Lee et al., 4 Jun 2025).

The paper states or implies several limitations. The method depends on a good base model, assumes access to a prototype buffer, updates the feature extractor only lightly, and appears effective mainly for the classifier rather than deeper layers (Lee et al., 4 Jun 2025). It also notes that some design choices, such as new-class weight initialization, are not emphasized in the main text (Lee et al., 4 Jun 2025). A plausible implication is that the success of weight interpolation depends on session-to-session parameter compatibility; the paper frames this as a condition under which weight-space interpolation should help rather than hurt (Lee et al., 4 Jun 2025).

The term itself warrants disambiguation. In (Hu et al., 2023), a coding-theoretic paper on three-weight $K$2-ary linear codes from inhomogeneous quadratic forms, the phrase tripartite weight-space structure refers to a deterministic partition of nonzero codewords into exactly three Hamming-weight strata. In that setting, the “tripartite” property is controlled by the rank and sign of a quadratic form, and the work concerns weight distributions, generalized Hamming weights, and secret-sharing relevance (Hu et al., 2023). That usage is mathematically unrelated to the FSCIL method of (Lee et al., 4 Jun 2025), even though both involve a three-part structure in a space indexed by weights. The shared terminology therefore does not indicate shared mechanism.

Within machine learning, “Tripartite Weight-Space Ensemble” denotes a specific answer to a specific FSCIL question: how to permit whole-model adaptation under few-shot incremental updates without allowing the current session to dominate the classifier. The method’s answer is to constrain classifier evolution by interpolating base, previous, and current heads, while distilling from the previous model on amplified mixed data (Lee et al., 4 Jun 2025). This places Tri-WE at the intersection of weight-space model combination, prototype-based incremental learning, and knowledge distillation under extreme data scarcity.