Continual Backpropagation Prompt Network (CBPNet)

• CBPNet is a prompt-based continual learning framework designed for class-incremental image classification using a frozen pretrained ViT with DualPrompt and an Efficient CBP Block.
• It employs a selective reinitialization strategy in the Efficient CBP Block to restore plasticity by tracking and refreshing underutilized hidden units.
• Empirical results on Split CIFAR-100 and Split ImageNet-R show that CBPNet outperforms DualPrompt with state-of-the-art accuracy while maintaining an extremely low trainable parameter footprint.

Searching arXiv for the cited CBPNet paper and closely related prompt-based continual learning papers to ground the article. Continual Backpropagation Prompt Network (CBPNet) is a prompt-based continual learning framework for class-incremental image classification on edge devices. It is designed for the setting in which a pretrained Vision Transformer receives a sequence of tasks and must incorporate new classes without retraining from scratch, without storing all prior data, and without fully fine-tuning the backbone. CBPNet combines a frozen pretrained ViT-B/16 with a DualPrompt-style prompting scheme and an additional Efficient CBP Block that selectively reinitializes underutilized hidden units to mitigate plasticity loss, defined in the paper as the progressive deterioration of the model’s ability to learn new tasks as sequential training proceeds (Shao et al., 19 Sep 2025).

1. Conceptual basis and problem formulation

CBPNet is situated in class-incremental continual learning (CL), where a model encounters a sequence of tasks and must maintain performance on previously seen classes while learning new ones. The paper emphasizes that this regime is particularly relevant for edge devices such as robots, autonomous systems, and embedded platforms, where memory, compute, and storage are constrained and where rehearsal buffers may be undesirable for privacy or deployment reasons (Shao et al., 19 Sep 2025).

A central distinction in the paper is between catastrophic forgetting and plasticity loss. Catastrophic forgetting refers to degradation on earlier tasks after learning later ones. Plasticity loss instead refers to a reduced ability to absorb new knowledge in the first place. CBPNet is motivated by the claim that frozen-backbone prompt tuning, while effective at suppressing forgetting, can induce a “critical secondary problem”: the limited trainable prompt subsystem gradually loses learning vitality. The authors attribute this to underutilized parameters that cease to receive useful adaptation pressure during continual optimization (Shao et al., 19 Sep 2025).

This framing places CBPNet within a specific subfamily of rehearsal-free CL methods: it preserves old knowledge primarily by leaving the large pretrained backbone frozen, but it attempts to restore future-task adaptability by rejuvenating a small trainable module rather than by relaxing backbone freezing. This suggests a design in which stability is inherited from parameter isolation, while plasticity is recovered through controlled internal turnover in the added module.

2. Architecture and prompt-conditioning pipeline

CBPNet retains the DualPrompt mechanism as its prompting substrate and inserts the Efficient CBP Block after the prompted transformer and before the classifier. The overall pipeline is:

1. an input image is passed through a frozen pretrained ViT;
2. G-Prompts are inserted in shallow transformer layers;
3. E-Prompts are selected and inserted in deeper layers;
4. the ViT outputs a pooled feature;
5. the pooled feature is passed through the Efficient CBP Block;
6. a classification head produces the prediction (Shao et al., 19 Sep 2025).

The backbone is ViT-B/16, pretrained on a large-scale dataset such as ImageNet-21k. In the experimental setup, the prompt configuration is fixed as follows: G-Prompt length = 5, applied to transformer layers 0 and 1, and E-Prompt length = 5, applied to transformer layers 2 through 4. Prompt insertion uses Prefix-Tuning, so prompt vectors are prepended to the key and value streams in attention rather than inserted as ordinary tokens (Shao et al., 19 Sep 2025).

The prompt-conditioned attention is written as

$$f_{\text{Pre-T}}(p,h) = \text{MSA}(h_Q, [p_K; h_K], [p_V; h_V]).$$

Here the query is derived from the current hidden state, while the keys and values are augmented with prompt-derived key/value tensors. The DualPrompt training objective inherited by CBPNet is

$$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$

where $g$ denotes G-Prompt parameters, $e_t$ the task-specific E-Prompt, $k_t$ the E-Prompt key, and $\phi$ the classification head (Shao et al., 19 Sep 2025).

The Efficient CBP Block is placed after the Global Pool layer and before the final classification head. Its internal structure is described as:

1. Input Layer
2. GELU activation
3. one CBP Linear Layer
4. Output Layer

The paper also characterizes it as a lightweight MLP with a bottleneck architecture. Its post-hoc placement is deliberate: dynamic plasticity restoration is localized to a small trainable block, while reinitialization is never applied inside the frozen pretrained transformer. This decouples the continual-backpropagation mechanism from the pretrained backbone and preserves the safety of the frozen PTM.

3. Continual backpropagation mechanism inside the CBP block

In CBPNet, “continual backpropagation” does not refer to a modified gradient-propagation rule throughout the whole network. Rather, it denotes a mechanism in the Efficient CBP Block that continuously monitors hidden-unit utility and selectively reinitializes mature low-utility units during sequential training (Shao et al., 19 Sep 2025).

The key statistic is a contribution utility for hidden unit $i$ in layer $l$:

$$u_{i}^{(l)} = \eta \times u_{i}^{(l)} + (1 - \eta) \times |\mathbf{h}_{l,i,t}| \times \left| \sum_{k=1}^{n_{l+1}} \mathbf{w}_{l,i,k,t} \right|.$$

In this expression, $\mathbf{h}_{l,i,t}$ is the output activation of the unit at time $$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$0, $$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$1 is the weight from that unit to unit $$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$2 in the next layer, and $$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$3 is the number of units in the next layer. The exponential moving average, controlled by $$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$4, stabilizes the estimate over time. Low values indicate units that are both weakly active and weakly influential downstream.

Reinitialization is governed by two controls. First, each unit has an age $$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$5, and only units older than the maturity threshold $$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$6 are eligible. In the experiments,

$$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$7

Second, mature low-utility units are refreshed at replacement rate

$$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$8

The reinitialization rule is procedural rather than symbolic: the paper specifies that CBPNet resamples the unit’s input weights from the initial distribution and zeros its output weights. Zeroing outgoing weights is intended to minimize functional disruption to the current network while making the unit available for future learning.

Operationally, after each training batch, the prompt parameters, prompt keys, classifier, and CBP block are updated by standard backpropagation; then utility statistics and ages are updated; then a small subset of mature low-utility units may be reinitialized (Shao et al., 19 Sep 2025). This makes the rejuvenation process sparse, ongoing, and localized to the trainable bottleneck MLP rather than to the frozen ViT.

4. Optimization regime, edge-oriented design, and parameter footprint

CBPNet is explicitly parameter-efficient. The paper reports a ViT-B/16 backbone of approximately 86 million parameters, while the total trainable parameters in CBPNet are approximately 150,000, which is less than 0.2\% of the backbone size (Shao et al., 19 Sep 2025).

Several implementation choices are directed toward edge deployment:

• the ViT-B/16 backbone remains frozen;
• prompt tuning is used instead of backbone fine-tuning;
• the CBP mechanism is realized as a small bottleneck MLP;
• the CBP block is inserted after the backbone, not within transformer blocks;
• the method uses no rehearsal buffer;
• reinitialization is disabled during inference (Shao et al., 19 Sep 2025).

The optimizer is Adam. The paper states that the learning rate and batch size are inherited from DualPrompt and adjusted based on Jetson Orin Nano memory constraints, but it does not provide the exact numerical learning rate, exact batch size, or number of epochs. It also reports experiments under edge constraints on NVIDIA Jetson Orin Nano and elsewhere mentions Jetson Orin (4GB), without resolving the discrepancy (Shao et al., 19 Sep 2025).

A plausible implication is that CBPNet is intended as a deployment-oriented CL method rather than a purely benchmark-driven architecture. The emphasis on a frozen PTM, a tiny trainable head-side module, and inference-time deactivation of rejuvenation logic all support that reading.

5. Empirical performance and ablation evidence

The main reported results are on Split CIFAR-100 and Split ImageNet-R, both in 10-task class-incremental learning protocols. The central comparison is against DualPrompt, with additional baselines spanning rehearsal-based, regularization-based, and prompt-based methods (Shao et al., 19 Sep 2025).

Benchmark	DualPrompt	CBPNet
Split CIFAR-100 Avg. Acc.	$$\min_{g,e_t,k_t,\phi} \mathcal{L}\big(f_{\phi}(f_{g,e_t}(x)), y\big) + \lambda\,\mathcal{L}_{match}(x,k_t),$$9	$g$0
Split CIFAR-100 Forgetting	$g$1	$g$2
Split ImageNet-R Avg. Acc.	$g$3	$g$4
Split ImageNet-R Forgetting	$g$5	$g$6

On Split CIFAR-100, CBPNet improves average accuracy over DualPrompt by +1.01 percentage points. On Split ImageNet-R, it improves by +1.67 percentage points, and the paper identifies 69.41\% as its reported state-of-the-art result (Shao et al., 19 Sep 2025).

The paper also compares CBPNet to rehearsal-free and rehearsal-based baselines. On Split CIFAR-100, the reported averages include L2P: $g$7 and DualPrompt: $g$8, with CBPNet at $g$9. On Split ImageNet-R, L2P: $e_t$0, DualPrompt: $e_t$1, and CBPNet: $e_t$2 are reported (Shao et al., 19 Sep 2025).

Ablation results are used as indirect evidence for the plasticity-loss hypothesis. On Split CIFAR-100, the ablation table gives:

• FT-seq: 33.41
• FT-seq + CBP: 41.29
• DualPrompt: 85.30
• CBPNet: 86.31

On Split ImageNet-R, the same ablation reports:

• FT-seq: 28.64
• FT-seq + CBP: 32.71
• DualPrompt: 68.13 in the ablation table
• CBPNet: 69.41

The paper also notes a discrepancy for DualPrompt on ImageNet-R, where Table 1 lists $e_t$3 but the ablation table lists $e_t$4, without explanation (Shao et al., 19 Sep 2025).

For later-task behavior, the paper states that on Split ImageNet-R, from task 6 to task 10, DualPrompt accuracy drops by about 3.5\%, whereas CBPNet drops by less than 2\%. This is presented as evidence that CBPNet mitigates plasticity loss over longer sequences.

6. Position in the broader literature

CBPNet sits at the intersection of prompt-based continual learning and the older idea of Continual Backprop. The 2021 Continual Backprop paper argued that standard backpropagation loses plasticity over long non-stationary training and proposed persistent randomness via low-utility feature replacement (Dohare et al., 2021). CBPNet imports a closely related intuition—revitalizing stagnant capacity through selective reinitialization—but applies it in a different architectural regime: a frozen pretrained ViT with DualPrompt prompting and a small post-backbone MLP rather than a general hidden-feature turnover mechanism across the full network.

Relative to other prompt-based CL methods, CBPNet does not alter prompt semantics or routing. POP separates task-specific prompts from a continually updated global prompt on a frozen transformer (Hu et al., 2023). OS-Prompt removes the extra query ViT in prompt-pool methods by extracting prompt queries from intermediate backbone states and adds Query-Pool Regularization to recover accuracy (Kim et al., 2024). DPFormer decomposes prompt memory into class and task prototypes inside a transformer with BCE, KD, and auxiliary losses (Huang et al., 9 Jun 2025). SMoPE treats prompt slots as sparse MoE experts and activates only a top-$e_t$5 subset per input to reduce interference without task-wise prompt growth (Le et al., 29 Sep 2025). By contrast, CBPNet keeps the DualPrompt prompt formulation essentially intact and places its novelty in a modular plasticity-restoration block.

The term “continual backpropagation” can therefore be misleading if read too literally. In CBPNet it does not denote a new gradient-propagation algorithm through the backbone. It denotes utility-tracked selective reinitialization inside the Efficient CBP Block. This differs both from the original Continual Backprop algorithm (Dohare et al., 2021) and from prompt-synthesis approaches such as HPrompt-CPT, which generate input-conditioned prompts via a hypernetwork for anytime fine-tuning in continual pre-training (Jiang et al., 2023).

7. Limitations and open directions

The paper does not provide an explicit limitations section, but several constraints are evident from what it reports and omits (Shao et al., 19 Sep 2025).

First, evaluation is restricted to class-incremental image classification on Split CIFAR-100 and Split ImageNet-R. No experiments are reported for language, multimodal CL, detection, segmentation, or fully online non-task-bounded streams. Second, CBPNet depends on a pretrained ViT-B/16 and the DualPrompt framework; the method is therefore best understood as a modular augmentation of prompt-based ViT CL rather than a backbone-agnostic CL principle. Third, hyperparameter sensitivity is underexplored: the paper reports only

$e_t$6

and does not provide sweeps over maturity threshold, replacement rate, prompt size, bottleneck width, or CBP placement. Fourth, runtime, FLOPs, and energy are not quantified, despite the edge-device emphasis. Fifth, the evidence for the underutilized-parameter hypothesis is largely performance-based; the paper explicitly does not include histograms of utility scores, activation sparsity plots, update-norm analyses, or unit-survival statistics.

The paper suggests future work in at least two directions: applying the modular CBP mechanism to other domains, including natural language processing, and combining it with memory-based strategies to balance stability and plasticity more effectively (Shao et al., 19 Sep 2025). This suggests that CBPNet, as presently defined, should be viewed less as a complete theory of plasticity-preserving prompt learning than as a compact architectural intervention demonstrating that prompt-based continual learners may benefit from explicit capacity rejuvenation in the trainable subsystem, even when catastrophic forgetting is already partly controlled by a frozen pretrained backbone.