---
title: 'Black-CL: Black-Box Continual Learning'
url: https://www.emergentmind.com/topics/black-cl
type: topic
---

# Black-CL: Black-Box Continual Learning

Black-CL is a black-box continual learning benchmark for vision-language models (VLMs) introduced to reflect the operational constraints of cloud-hosted multimodal systems, where the learner cannot access or modify backbone weights, must operate under constrained computation, and must perform task-agnostic inference over a growing global label space [2606.22999]. In the same work, the benchmark is paired with BETA, a continual-learning baseline that restricts adaptation to external textual prototypes and combines Semantic Projection Accumulation (SPA), Latent Distribution Replay (LDR), and Test-Time Prototype Adaptation (TTPA) [2606.22999]. The term is therefore principally associated with black-box continual learning in contemporary VLM research, although the same letter string appears in unrelated literatures with different meanings.

## 1. Benchmark definition and motivating constraints

Black-CL is defined by three explicit constraints. First, it enforces **weight and architecture inaccessibility**: the learner has no access to backbone parameters, internal structure, intermediate activations, or the backward graph, and cannot insert adapters or internal modules. Second, it imposes **constrained computation**, requiring a small trainable footprint and lightweight optimization. Third, it requires **task-agnostic inference**, meaning prediction must be made over the global label space of all seen tasks without a task oracle [2606.22999].

These restrictions are meant to distinguish Black-CL from traditional white-box continual learning. In white-box settings, CL methods commonly depend on backpropagation through the backbone, parameter expansion, prompt tuning inside the model, or direct structural intervention. Black-CL disallows all of these. The learner may query only output embeddings or logits, and gradients may be computed only with respect to external parameters. No gradient is allowed to flow through the frozen image encoder $E_I$ or frozen text encoder $E_T$ [2606.22999].

The benchmark is therefore not a simple robustness variant of continual learning. It is a deployment-oriented formulation in which adaptation must happen entirely outside the backbone. This suggests a shift in emphasis from internal representation updating to external calibration, prototype learning, and replay in queried latent spaces.

## 2. Formal setup and evaluation protocol

The Black-CL backbone interface is denoted by $\Phi$ and consists of a frozen image encoder $E_I$ and a frozen text encoder $E_T$. For an image $x$ and a class prompt $t$, the queried normalized embeddings are
$$
z = E_I(x), \qquad u = E_T(t).
$$
When fitting latent replay, the raw image feature prior to normalization is also queried:
$$
r = E_I^{\mathrm{raw}}(x), \qquad z = \frac{r}{\|r\|_2}.
$$
Only an external parameter set $\theta$ is trainable [2606.22999].

The continual stream is
$$
\mathcal{S} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}, \qquad \mathcal{D}_t=\mathcal{D}_t^{train}\cup\mathcal{D}_t^{test},
$$
with disjoint task label spaces,
$$
\mathcal{Y}_t \cap \mathcal{Y}_{t'}=\emptyset \quad \text{for } t\neq t',
$$
and global label set
$$
\mathcal{Y} = \bigcup_{t=1}^T \mathcal{Y}_t.
$$
After learning task $i$, inference uses a hybrid classifier $g_i(c)$ that combines learned embeddings for seen classes and frozen zero-shot embeddings for unseen classes. Prediction is made by cosine similarity:
$$
\hat{y} = \operatorname*{argmax}_{c \in \mathcal{Y}} \cos\!\big(E_I(x), g_i(c)\big).
$$
This hybrid classifier is central to task-agnostic inference because it avoids requiring a task oracle even before the entire stream has been observed [2606.22999].

Evaluation is organized through the matrix entry $R_{i,j}$, the accuracy on test task $j$ after training up to task $i$. Black-CL reports three aggregate metrics. **Transfer Accuracy** is the average of $R_{i,j}$ with $j>i$ across stages and then averaged over the stream. **Average Accuracy** is the mean of all entries $R_{i,j}$ over all training stages and test tasks. **Last Accuracy** is the average of the final-row accuracies $R_{T,j}$ over all test tasks, measuring performance after the full stream [2606.22999].

## 3. BETA: textual prototype optimization under black-box constraints

BETA, short for **Black-box Embedding Tuning and Adaptation**, is the baseline proposed for Black-CL. Its organizing idea is that continual learning under strict black-box constraints can be handled by optimizing only textual prototypes. For task $i$ with label space $\mathcal{Y}_i$, BETA defines learnable textual prototypes
$$
\mathbf{P}_i = \{p_{i,k}\}_{k=1}^{|\mathcal{Y}_i|},
$$
initialized from the frozen text encoder as
$$
p_{i,k}^{(0)} = u_k = E_T(t_k).
$$
During training, only $\mathbf{P}_i$ receives gradients; image and text features are detached and normalized, and cosine similarity defines the logits [2606.22999].

SPA, the **Semantic Projection Accumulation** component, incrementally builds a textual prototype library. For each task, it optimizes $\mathbf{P}_i$ by minimizing cross-entropy on current-task samples, with no gradients flowing through $E_I$ or $E_T$. After optimization, the prototypes are stored and concatenated into a growing classifier together with frozen zero-shot embeddings for unseen tasks [2606.22999].

LDR, or **Latent Distribution Replay**, models per-class latent statistics using spherical Gaussian mixture models on raw CLIP image features. For class $c$, the replay model is
$$
\Theta_c = \big\{\pi_{c,k}, \mu_{c,k}, \sigma^2_{c,k}\big\}_{k=1}^K, \qquad \text{covariance } \sigma^2_{c,k} I,
$$
with $K=4$ components by default. Pseudo raw features $\tilde r$ are sampled from this mixture and normalized to $\tilde z=\tilde r/\|\tilde r\|_2$. Replay is integrated through a unified-label-space cross-entropy objective over current embeddings $Z_i$ and replayed pseudo-embeddings $\tilde Z_{1:i-1}$:
$$
\mathcal{L}_{\text{total}}
=
\mathcal{L}_{\text{CE}}\!\big(Z_i \cup \tilde{Z}_{1:i-1}, \mathbf{P}_{1:i}\big),
$$
where replay samples are weighted by a coefficient with default value $64$ [2606.22999]. The paper characterizes this mechanism as a “semantic firewall” that prevents new-task updates from eroding global discriminability.

TTPA, the **Test-Time Prototype Adaptation** module, updates prototypes online at inference under constrained latency. For batch $\mathcal{B}_b=\{x_j\}_{j=1}^{N}$, normalized visual features are $\mathbf{f}_j=E_I(x_j)$. Reliable anchors are selected by confidence filtering with threshold $\delta=0.95$:
$$
\mathcal{X}_{a}^{(b)}
=
\left\{
x_j \in \mathcal{B}_b
\;\middle|\;
\max \operatorname{softmax}\!\big(\mathbf{f}_j \mathbf{P}^{(b-1)\top}\big)
\geq \delta
\right\}.
$$
At most one prototype update is taken per batch. With learning rate $\eta=5\times 10^{-4}$ and label smoothing $\epsilon=0.1$,
$$
\mathbf{P}^{(b)}
=
\begin{cases}
\mathbf{P}^{(b-1)} - \eta\, \nabla_{\mathbf{P}} \mathcal{L}_{a}^{(b)}\big|_{\mathbf{P}=\mathbf{P}^{(b-1)}}, & \text{if } |\mathcal{X}_{a}^{(b)}|>0,\\
\mathbf{P}^{(b-1)}, & \text{if } |\mathcal{X}_{a}^{(b)}|=0,
\end{cases}
$$
where
$$
\mathcal{L}_{a}^{(b)}
=
\sum_{x_j\in\mathcal{X}_{a}^{(b)}}
\mathcal{L}_{\text{CE}}\!\Big(
\operatorname{softmax}\big(\mathbf{f}_j\mathbf{P}^{\top}\big),
\tilde{\mathbf{y}}_j
\Big).
$$
The adapted state is retained across consecutive batches within a complete evaluation pass and then reset to $\mathbf{P}^{(0)}$ at the end of the pass [2606.22999].

## 4. Datasets, protocols, and empirical performance

The primary Black-CL stream instantiates $\mathcal{S}$ with ten datasets treated as cross-domain tasks with substantial distribution shifts: Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, and UCF101. Label spaces are disjoint, and inference is task-agnostic over the global label set. MNIST is used only for comparison against white-box CL methods and is excluded from the primary benchmark because of limited relevance to complex VLM deployments [2606.22999].

The evaluated frozen backbones include CLIP ViT-B/16-224, CLIP ViT-L/14-224, SigLIP B/16-224, SigLIP So/14-224, SigLIP 2 B/16-224, and SigLIP 2 So/14-224. Cosine similarity between normalized image and text embeddings provides the logits, and class prompts follow dataset-specific templates following CoOp [2606.22999].

On the full-shot primary stream, the zero-shot baseline obtains **Average Accuracy 61.0%**. Under the same protocol, BETA reports **Transfer Accuracy 59.8%**, **Average Accuracy 72.5%**, and **Last Accuracy 85.4%**. Against black-box tuners, the reported Last and Average Accuracy values are: BETA (0.05 M) 85.4 and 72.5; CBBT/Tip-Adapter (2.2 M) 65.3 and 66.3; LFA (0.3 M) 64.3 and 65.0; BlackVIP (0.01 M) 62.3 and 63.1; LCS (0.002 M) 61.2 and 61.9; and Primal-RAIL (18.0 M) 82.8 and 68.6 [2606.22999]. Within this comparison, BETA is reported to outperform existing black-box tuners on Last and Average Accuracy while keeping Transfer Accuracy near the zero-shot baseline.

Against white-box continual-learning baselines on the stream that includes MNIST, BETA reports **Transfer Accuracy 62.9**, **Average Accuracy 74.2**, and **Last Accuracy 86.0**. The paper notes that this Transfer Accuracy is higher than all listed white-box baselines, that Average Accuracy is comparable to MoE-Adapters 67.2 and LADA 75.2, and that Last Accuracy is close to LADA 86.9. The corresponding parameter reductions are reported as **180× vs LADA (9.0 M)**, **3000× vs ZSCL (149.6 M)**, and **360× vs Primal-RAIL (18.0 M)** [2606.22999].

Backbone generalization is likewise emphasized. Reported full-shot Last Accuracy values are 85.4% for CLIP ViT-B/16-224, 88.6% for CLIP ViT-L/14-224, 88.0% for SigLIP B/16-224, 90.9% for SigLIP So/14-224, 89.2% for SigLIP 2 B/16-224, and 92.1% for SigLIP 2 So/14-224. Gains are especially large on EuroSAT, including +50.7 with CLIP-B/16 and +53.1 with SigLIP 2 So/14 [2606.22999].

## 5. Efficiency, ablations, and limitations

BETA’s trainable state is restricted to textual prototypes. For CLIP ViT-B/16 with embedding dimension $D=512$ and representative average class count $n_{\text{avg}}\approx 100$ per task, the active trainable count per stage is
$$
N_{\text{BETA}} = n_{\text{avg}} \times D = 100 \times 512 = 51{,}200 \approx 0.05\,\text{M}.
$$
This is the basis for the reported 180–3000$\times$ parameter reduction relative to competitive methods such as LADA 9.0 M, ZSCL 149.6 M, and Primal-RAIL 18.0 M [2606.22999].

Ablation studies attribute most of BETA’s gains to the interaction of SPA and LDR, with TTPA providing a smaller but consistent improvement. Under 16-shot training, SPA only gives Avg 68.0 and Last 77.1; SPA+LDR gives Avg 70.1 and Last 81.4; SPA+LDR+TTPA gives Avg 70.8 and Last 81.5. Under full-shot training, SPA only gives Avg 68.2 and Last 78.4; SPA+LDR gives Avg 71.7 and Last 85.3; SPA+LDR+TTPA gives Avg 72.5 and Last 85.4 [2606.22999].

Sensitivity analyses for TTPA show that the default settings are $\eta=0.0005$ and $\delta=0.95$. For $\eta \in \{0.0001, 0.0005, 0.001, 0.005\}$, the reported Avg values are 70.7, 70.8, 69.6, and 62.5, with Last values 81.4, 81.5, 81.6, and 81.7. For $\delta \in \{0.8, 0.85, 0.9, 0.95\}$, the Avg values are 68.3, 68.7, 70.2, and 70.8, with Last values 81.4, 81.4, 81.5, and 81.5. Continuous prototype retention across the full evaluation pass gives the best overall performance, namely Transfer 59.8, Avg 72.5, and Last 85.4; batch-wise reset underperforms [2606.22999].

The paper also analyzes LDR’s latent modeling assumptions. It reports that spherical GMM improves held-out log-likelihood over a single spherical Gaussian in **85.7% of classes**, with **average gain 11.75** and **median 7.46**, and that this structured replay outperforms noise-matched Gaussian replay. Under full-shot CLIP-B/16, spherical GMM yields Transfer/Average/Last of **59.7/72.5/85.4**, compared with **58.7/72.5/85.3** for diagonal GMM [2606.22999]. The intended conclusion is that mixture structure, rather than noise magnitude alone, drives LDR’s benefit.

The storage and runtime profile is correspondingly small. Trainable parameters remain **0.05 M per stage (non-cumulative)**, and storing prototypes and LDR summaries across about 1000 classes with $D=512$ requires roughly **5 MB mixed-precision**. LDR storage scales as $K(D+2)$ scalars per class; for $D=1024$, 1000 classes, and $K=4$, the paper gives
$$
1000 \times 4 \times (1024 + 2) \times 4 \text{ bytes} \approx 16.4\,\text{MB}.
$$
Fitting spherical GMMs with $K=4$ takes **<5 s on CPU for \sim100 classes**, and TTPA adds only **34.7% overhead** relative to zero-shot inference, compared with about **6.05×** for TENT and about **29.36×** for CoTTA [2606.22999].

The stated limitations are equally explicit. Spherical GMMs are acknowledged to be compact approximations rather than exact models of CLIP features. In 16-shot settings, $K=4$ can overfit in density terms, though replay performance remains robust. TTPA depends on confident anchors; if a batch has no anchors, no adaptation step is taken. These design choices prioritize stability and low latency under strict black-box constraints [2606.22999].

## 6. Reproducibility, implementation details, and terminological ambiguity

The primary implementation uses CLIP ViT-B/16-224, AdamW for SPA with weight decay $5\times 10^{-4}$, batch size $64$, learning rate $5\times 10^{-3}$, and **200 epochs by default**. LDR uses spherical GMM with **$K=4$ components per class** and replay coefficient **64**. TTPA uses confidence threshold $\delta=0.95$, learning rate $\eta=5\times 10^{-4}$, label smoothing $\epsilon=0.1$, and SGD with momentum **0.9**, taking at most one gradient step per batch if anchors exist. Standard CLIP preprocessing is used, namely RandomResizedCrop and RandomHorizontalFlip for training, and Resize+CenterCrop for evaluation [2606.22999]. The authors state: “The code will be made available soon.”

The string “Black-CL,” however, is not unique across arXiv-adjacent literatures. In person re-identification, it has been used to denote the **black-clothes** regime studied by “Black Re-ID,” where clothing cues collapse and head–shoulder features are emphasized through the Head-Shoulder Adaptive Attention framework [2008.08528]. In extragalactic astronomy, “CL” often denotes **coronal lines** or **changing-look**, producing unrelated usages such as infrared coronal-line diagnostics of accreting black holes in low-metallicity dwarfs and intermediate-mass black-hole searches with JWST [2104.05689; 1805.09351], as well as changing-look AGN and changing-look blazar studies [2308.16521; 2511.07532; 2110.13013; 2501.00094]. A further multimodal recommendation paper discusses “does it come in black?” style retrieval with CLIP-like models, but explicitly notes that “Black-CL” is not the term used in that work [2204.02473].

For current VLM continual-learning research, the precise referent is therefore the benchmark “Black-Box Continual Learning for Vision-Language Models,” in which Black-CL names a constrained continual-learning protocol and BETA serves as its prototype-only baseline [2606.22999].

Source: https://www.emergentmind.com/topics/black-cl