Black-CL: Black-Box Continual Learning

• Black-CL is a black-box continual learning benchmark that enforces constraints like inaccessible weights, restricted computation, and task-agnostic inference in vision-language models.
• It introduces BETA, a baseline that optimizes external textual prototypes using Semantic Projection Accumulation, Latent Distribution Replay, and Test-Time Prototype Adaptation.
• The framework demonstrates competitive accuracy and significant parameter reductions across various datasets, illustrating practical deployment in cloud-hosted multimodal systems.

Black-CL is a black-box continual learning benchmark for vision-LLMs (VLMs) introduced to reflect the operational constraints of cloud-hosted multimodal systems, where the learner cannot access or modify backbone weights, must operate under constrained computation, and must perform task-agnostic inference over a growing global label space (Li et al., 22 Jun 2026). In the same work, the benchmark is paired with BETA, a continual-learning baseline that restricts adaptation to external textual prototypes and combines Semantic Projection Accumulation (SPA), Latent Distribution Replay (LDR), and Test-Time Prototype Adaptation (TTPA) (Li et al., 22 Jun 2026). The term is therefore principally associated with black-box continual learning in contemporary VLM research, although the same letter string appears in unrelated literatures with different meanings.

1. Benchmark definition and motivating constraints

Black-CL is defined by three explicit constraints. First, it enforces weight and architecture inaccessibility: the learner has no access to backbone parameters, internal structure, intermediate activations, or the backward graph, and cannot insert adapters or internal modules. Second, it imposes constrained computation, requiring a small trainable footprint and lightweight optimization. Third, it requires task-agnostic inference, meaning prediction must be made over the global label space of all seen tasks without a task oracle (Li et al., 22 Jun 2026).

These restrictions are meant to distinguish Black-CL from traditional white-box continual learning. In white-box settings, CL methods commonly depend on backpropagation through the backbone, parameter expansion, prompt tuning inside the model, or direct structural intervention. Black-CL disallows all of these. The learner may query only output embeddings or logits, and gradients may be computed only with respect to external parameters. No gradient is allowed to flow through the frozen image encoder $E_I$ or frozen text encoder $E_T$ (Li et al., 22 Jun 2026).

The benchmark is therefore not a simple robustness variant of continual learning. It is a deployment-oriented formulation in which adaptation must happen entirely outside the backbone. This suggests a shift in emphasis from internal representation updating to external calibration, prototype learning, and replay in queried latent spaces.

2. Formal setup and evaluation protocol

The Black-CL backbone interface is denoted by $\Phi$ and consists of a frozen image encoder $E_I$ and a frozen text encoder $E_T$. For an image $x$ and a class prompt $t$, the queried normalized embeddings are

$z = E_I(x), \qquad u = E_T(t).$

When fitting latent replay, the raw image feature prior to normalization is also queried:

$$r = E_I^{\mathrm{raw}}(x), \qquad z = \frac{r}{\|r\|_2}.$$

Only an external parameter set $\theta$ is trainable (Li et al., 22 Jun 2026).

The continual stream is

$E_T$0

with disjoint task label spaces,

$E_T$1

and global label set

$E_T$2

After learning task $E_T$3, inference uses a hybrid classifier $E_T$4 that combines learned embeddings for seen classes and frozen zero-shot embeddings for unseen classes. Prediction is made by cosine similarity:

$E_T$5

This hybrid classifier is central to task-agnostic inference because it avoids requiring a task oracle even before the entire stream has been observed (Li et al., 22 Jun 2026).

Evaluation is organized through the matrix entry $E_T$6, the accuracy on test task $E_T$7 after training up to task $E_T$8. Black-CL reports three aggregate metrics. Transfer Accuracy is the average of $E_T$9 with $\Phi$0 across stages and then averaged over the stream. Average Accuracy is the mean of all entries $\Phi$1 over all training stages and test tasks. Last Accuracy is the average of the final-row accuracies $\Phi$2 over all test tasks, measuring performance after the full stream (Li et al., 22 Jun 2026).

3. BETA: textual prototype optimization under black-box constraints

BETA, short for Black-box Embedding Tuning and Adaptation, is the baseline proposed for Black-CL. Its organizing idea is that continual learning under strict black-box constraints can be handled by optimizing only textual prototypes. For task $\Phi$3 with label space $\Phi$4, BETA defines learnable textual prototypes

$\Phi$5

initialized from the frozen text encoder as

$\Phi$6

During training, only $\Phi$7 receives gradients; image and text features are detached and normalized, and cosine similarity defines the logits (Li et al., 22 Jun 2026).

SPA, the Semantic Projection Accumulation component, incrementally builds a textual prototype library. For each task, it optimizes $\Phi$8 by minimizing cross-entropy on current-task samples, with no gradients flowing through $\Phi$9 or $E_I$0. After optimization, the prototypes are stored and concatenated into a growing classifier together with frozen zero-shot embeddings for unseen tasks (Li et al., 22 Jun 2026).

LDR, or Latent Distribution Replay, models per-class latent statistics using spherical Gaussian mixture models on raw CLIP image features. For class $E_I$1, the replay model is

$E_I$2

with $E_I$3 components by default. Pseudo raw features $E_I$4 are sampled from this mixture and normalized to $E_I$5. Replay is integrated through a unified-label-space cross-entropy objective over current embeddings $E_I$6 and replayed pseudo-embeddings $E_I$7:

$E_I$8

where replay samples are weighted by a coefficient with default value $E_I$9 (Li et al., 22 Jun 2026). The paper characterizes this mechanism as a “semantic firewall” that prevents new-task updates from eroding global discriminability.

TTPA, the Test-Time Prototype Adaptation module, updates prototypes online at inference under constrained latency. For batch $E_T$0, normalized visual features are $E_T$1. Reliable anchors are selected by confidence filtering with threshold $E_T$2:

$E_T$3

At most one prototype update is taken per batch. With learning rate $E_T$4 and label smoothing $E_T$5,

$E_T$6

where

$E_T$7

The adapted state is retained across consecutive batches within a complete evaluation pass and then reset to $E_T$8 at the end of the pass (Li et al., 22 Jun 2026).

4. Datasets, protocols, and empirical performance

The primary Black-CL stream instantiates $E_T$9 with ten datasets treated as cross-domain tasks with substantial distribution shifts: Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, and UCF101. Label spaces are disjoint, and inference is task-agnostic over the global label set. MNIST is used only for comparison against white-box CL methods and is excluded from the primary benchmark because of limited relevance to complex VLM deployments (Li et al., 22 Jun 2026).

The evaluated frozen backbones include CLIP ViT-B/16-224, CLIP ViT-L/14-224, SigLIP B/16-224, SigLIP So/14-224, SigLIP 2 B/16-224, and SigLIP 2 So/14-224. Cosine similarity between normalized image and text embeddings provides the logits, and class prompts follow dataset-specific templates following CoOp (Li et al., 22 Jun 2026).

On the full-shot primary stream, the zero-shot baseline obtains Average Accuracy 61.0%. Under the same protocol, BETA reports Transfer Accuracy 59.8%, Average Accuracy 72.5%, and Last Accuracy 85.4%. Against black-box tuners, the reported Last and Average Accuracy values are: BETA (0.05 M) 85.4 and 72.5; CBBT/Tip-Adapter (2.2 M) 65.3 and 66.3; LFA (0.3 M) 64.3 and 65.0; BlackVIP (0.01 M) 62.3 and 63.1; LCS (0.002 M) 61.2 and 61.9; and Primal-RAIL (18.0 M) 82.8 and 68.6 (Li et al., 22 Jun 2026). Within this comparison, BETA is reported to outperform existing black-box tuners on Last and Average Accuracy while keeping Transfer Accuracy near the zero-shot baseline.

Against white-box continual-learning baselines on the stream that includes MNIST, BETA reports Transfer Accuracy 62.9, Average Accuracy 74.2, and Last Accuracy 86.0. The paper notes that this Transfer Accuracy is higher than all listed white-box baselines, that Average Accuracy is comparable to MoE-Adapters 67.2 and LADA 75.2, and that Last Accuracy is close to LADA 86.9. The corresponding parameter reductions are reported as 180× vs LADA (9.0 M), 3000× vs ZSCL (149.6 M), and 360× vs Primal-RAIL (18.0 M) (Li et al., 22 Jun 2026).

Backbone generalization is likewise emphasized. Reported full-shot Last Accuracy values are 85.4% for CLIP ViT-B/16-224, 88.6% for CLIP ViT-L/14-224, 88.0% for SigLIP B/16-224, 90.9% for SigLIP So/14-224, 89.2% for SigLIP 2 B/16-224, and 92.1% for SigLIP 2 So/14-224. Gains are especially large on EuroSAT, including +50.7 with CLIP-B/16 and +53.1 with SigLIP 2 So/14 (Li et al., 22 Jun 2026).

5. Efficiency, ablations, and limitations

BETA’s trainable state is restricted to textual prototypes. For CLIP ViT-B/16 with embedding dimension $x$0 and representative average class count $x$1 per task, the active trainable count per stage is

$x$2

This is the basis for the reported 180–3000$x$3 parameter reduction relative to competitive methods such as LADA 9.0 M, ZSCL 149.6 M, and Primal-RAIL 18.0 M (Li et al., 22 Jun 2026).

Ablation studies attribute most of BETA’s gains to the interaction of SPA and LDR, with TTPA providing a smaller but consistent improvement. Under 16-shot training, SPA only gives Avg 68.0 and Last 77.1; SPA+LDR gives Avg 70.1 and Last 81.4; SPA+LDR+TTPA gives Avg 70.8 and Last 81.5. Under full-shot training, SPA only gives Avg 68.2 and Last 78.4; SPA+LDR gives Avg 71.7 and Last 85.3; SPA+LDR+TTPA gives Avg 72.5 and Last 85.4 (Li et al., 22 Jun 2026).

Sensitivity analyses for TTPA show that the default settings are $x$4 and $x$5. For $x$6, the reported Avg values are 70.7, 70.8, 69.6, and 62.5, with Last values 81.4, 81.5, 81.6, and 81.7. For $x$7, the Avg values are 68.3, 68.7, 70.2, and 70.8, with Last values 81.4, 81.4, 81.5, and 81.5. Continuous prototype retention across the full evaluation pass gives the best overall performance, namely Transfer 59.8, Avg 72.5, and Last 85.4; batch-wise reset underperforms (Li et al., 22 Jun 2026).

The paper also analyzes LDR’s latent modeling assumptions. It reports that spherical GMM improves held-out log-likelihood over a single spherical Gaussian in 85.7% of classes, with average gain 11.75 and median 7.46, and that this structured replay outperforms noise-matched Gaussian replay. Under full-shot CLIP-B/16, spherical GMM yields Transfer/Average/Last of 59.7/72.5/85.4, compared with 58.7/72.5/85.3 for diagonal GMM (Li et al., 22 Jun 2026). The intended conclusion is that mixture structure, rather than noise magnitude alone, drives LDR’s benefit.

The storage and runtime profile is correspondingly small. Trainable parameters remain 0.05 M per stage (non-cumulative), and storing prototypes and LDR summaries across about 1000 classes with $x$8 requires roughly 5 MB mixed-precision. LDR storage scales as $x$9 scalars per class; for $t$0, 1000 classes, and $t$1, the paper gives

$t$2

Fitting spherical GMMs with $t$3 takes <5 s on CPU for \sim100 classes, and TTPA adds only 34.7% overhead relative to zero-shot inference, compared with about 6.05× for TENT and about 29.36× for CoTTA (Li et al., 22 Jun 2026).

The stated limitations are equally explicit. Spherical GMMs are acknowledged to be compact approximations rather than exact models of CLIP features. In 16-shot settings, $t$4 can overfit in density terms, though replay performance remains robust. TTPA depends on confident anchors; if a batch has no anchors, no adaptation step is taken. These design choices prioritize stability and low latency under strict black-box constraints (Li et al., 22 Jun 2026).

6. Reproducibility, implementation details, and terminological ambiguity

The primary implementation uses CLIP ViT-B/16-224, AdamW for SPA with weight decay $t$5, batch size $t$6, learning rate $t$7, and 200 epochs by default. LDR uses spherical GMM with $t$8 components per class and replay coefficient 64. TTPA uses confidence threshold $t$9, learning rate $z = E_I(x), \qquad u = E_T(t).$0, label smoothing $z = E_I(x), \qquad u = E_T(t).$1, and SGD with momentum 0.9, taking at most one gradient step per batch if anchors exist. Standard CLIP preprocessing is used, namely RandomResizedCrop and RandomHorizontalFlip for training, and Resize+CenterCrop for evaluation (Li et al., 22 Jun 2026). The authors state: “The code will be made available soon.”

The string “Black-CL,” however, is not unique across arXiv-adjacent literatures. In person re-identification, it has been used to denote the black-clothes regime studied by “Black Re-ID,” where clothing cues collapse and head–shoulder features are emphasized through the Head-Shoulder Adaptive Attention framework (Xu et al., 2020). In extragalactic astronomy, “CL” often denotes coronal lines or changing-look, producing unrelated usages such as infrared coronal-line diagnostics of accreting black holes in low-metallicity dwarfs and intermediate-mass black-hole searches with JWST (Cann et al., 2021, Cann et al., 2018), as well as changing-look AGN and changing-look blazar studies (Wang et al., 2023, Zeltyn et al., 10 Nov 2025, Sniegowska et al., 2021, Kang et al., 2024). A further multimodal recommendation paper discusses “does it come in black?” style retrieval with CLIP-like models, but explicitly notes that “Black-CL” is not the term used in that work (Chia et al., 2022).

For current VLM continual-learning research, the precise referent is therefore the benchmark “Black-Box Continual Learning for Vision-LLMs,” in which Black-CL names a constrained continual-learning protocol and BETA serves as its prototype-only baseline (Li et al., 22 Jun 2026).