Black-Box Continual Learning for Vision-Language Models

Abstract: The rapid deployment of Vision-LLMs (VLMs) in dynamic environments necessitates the ability to learn continuously without forgetting. However, traditional continual learning (CL) settings often rely on white-box paradigms, which is increasingly invalidated by the shift toward cloud-hosted models. In this paper, we introduce Black-CL, a more realistic benchmark for VLMs that enforces three primary real-world challenges: weight and architecture inaccessibility, constrained computation, and task-agnostic inference. The learner can query only output embeddings or logits, with no gradient flow through or structural modification of the backbone. Current CL methodologies, which rely on backbone backpropagation or complex parameter expansion, are fundamentally incompatible with these constraints. Under this setting, we propose BETA, a simple yet effective baseline built on the key insight that solely optimizing textual prototypes can navigate the complexities of CL. BETA integrates three core components: Semantic Projection Accumulation (SPA) for incremental knowledge acquisition, Latent Distribution Replay (LDR) for anchoring the embedding space against catastrophic forgetting, and Test-Time Prototype Adaptation (TTPA) for dynamic, instance-aware boundary refinement. Extensive experiments across ten diverse datasets and various backbones demonstrate that BETA significantly outperforms existing black-box tuners. Remarkably, with only 0.05 M trainable parameters, a 180--3000$\times$ reduction compared to competitive methods, BETA achieves performance on par with or even exceeding white-box CL methods. We believe Black-CL and BETA provide a foundational framework for future advancements in continual learning and accelerates the transition of continual learning from academia to real-world systems.

• The paper introduces BETA, a method that optimizes external textual prototypes and employs latent distribution replay to enable continual learning in black-box VLMs.
• The paper demonstrates that BETA achieves up to 85.4% Last Accuracy and significant parameter efficiency compared to traditional white-box methods.
• The paper validates BETA’s robustness across diverse datasets and VLM backbones while incurring minimal inference latency.

Black-Box Continual Learning for Vision-LLMs: An Expert Essay

Motivation and Problem Formulation

The deployment of Vision-LLMs (VLMs) in dynamic, real-world environments increasingly requires continual learning (CL), where models assimilate new knowledge incrementally while retaining prior capabilities. However, the prevailing CL methodologies operate under "white-box" assumptions, allowing full access to model weights, gradients, and architecture. Such assumptions are progressively untenable, given the trend toward proprietary foundation models (cloud-hosted or edge-deployed) that expose only output embeddings or logits, with no access to internal structure or gradient flow. The paper "Black-Box Continual Learning for Vision-LLMs" (2606.22999) formalizes these practical constraints into the Black-CL benchmark, enforcing:

• Weight and Architecture Inaccessibility: No access to backbone internals, no gradient propagation, and no insertion of architectural modifications.
• Constrained Computation: Minimal parameter footprints and update efficiency.
• Task-Agnostic Inference: Global label space classification without task or domain identifiers at inference.

This benchmark is stricter than merely freezing a locally accessible backbone, as it bars even prompt-tuning or adapter insertion through gradient-based methods. The study evidently bridges the gap between academic CL settings and industrial deployment realities.

Proposed Method: BETA

BETA (Black-box Embedding Tuning and Adaptation) is introduced as a conceptually simple yet highly effective baseline for Black-CL. The central insight is that optimizing external textual prototypes, rather than modifying model weights, suffices for robust incremental learning. BETA comprises three synergistic modules:

• Semantic Projection Accumulation (SPA): Textual prototypes are sequentially optimized for each incremental task by querying the text encoder and tuning class-specific external vectors. These prototypes serve as persistent semantic anchors and are extremely lightweight relative to backbone parameters.
• Latent Distribution Replay (LDR): Catastrophic forgetting is mitigated by modeling the statistical distribution of visual embedding space via parametric spherical Gaussian Mixture Models (GMMs) per class. Sampled pseudo-features from stored GMMs enable replay in the embedding space, enforcing semantic boundaries with minimal storage overhead.
• Test-Time Prototype Adaptation (TTPA): Instance-aware adaptation occurs during inference, dynamically recalibrating prototypes based on local test-batch distributions. High-confidence anchors trigger lightweight gradient updates per batch, ensuring efficient and robust adaptation with negligible latency overhead.

Notably, all learning happens strictly outside the backbone, using no gradient flow or architectural modification, fully adhering to Black-CL constraints.

Empirical Evaluation: Numerical Results and Analysis

BETA was evaluated across ten diverse datasets (Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101) and multiple VLM backbones (CLIP, SigLIP, SigLIP2). The critical findings are:

• Parameter Efficiency: BETA uses only 0.05M trainable parameters, a 180-3000x reduction compared to strong baselines (e.g., LADA with 9M, ZSCL with 149.6M).
• Performance Metrics: On the full-shot stream, BETA achieved an 85.4% Last Accuracy and 74.2% Average Accuracy, outperforming existing black-box tuners. Transfer Accuracy remained at 62.9%, on par or superior to white-box methods, indicating near-optimal preservation of zero-shot capabilities.
• Comparison with White-Box Methods: Despite strict black-box constraints, BETA's accuracy closely matches or surpasses white-box SOTA methods (e.g., LADA, MoE-Adapters), demonstrating minimal trade-off for computational efficiency.
• Generalization Across Backbones: Accuracy gains were consistent across all tested architectures; for instance, BETA elevated CLIP-B/16 backbone performance from 61.0% to 85.4% (+24.4%), and raised SigLIP2-So/14 backbone from 76.1% to 92.1%. Gains are particularly pronounced in specialized domains, e.g., >50% improvement on EuroSAT.
• Robustness to Data Scarcity and Task Sequence: Both 16-shot and full-shot settings, and alternative task sequences, confirm that BETA's efficacy is unaffected by training regime or data order.
• Inference Latency: TTPA adds only 34.7% overhead relative to the zero-shot baseline, far less than classic test-time adaptation methods (CoTTA: +2736.4%).

Technical Ablations and Component Analysis

Ablation studies demonstrate the indispensable role of each BETA component:

• SPA delivers task-specialized adaptation.
• LDR stabilizes the embedding space, contributing a 6.9% boost in Last Accuracy under full-shot.
• TTPA ensures robust calibration against distribution shifts at test time, yielding highest overall accuracy.

Empirical analysis of GMM fitting shows that spherical GMMs offer a compact, stable local approximation to CLIP features with high Q-Q radial diagnostic correlations, favoring accuracy-storage-stability trade-offs over diagonal GMMs. LDR’s replay effectiveness demonstrably relies on the structural information within the GMM mixture, not merely random noise.

Implications, Practical and Theoretical

BETA establishes a paradigm shift in continual learning for restricted-access VLMs. By eschewing architectural interventions and weight updates, it democratizes incremental adaptation for foundation models in cloud/edge environments. Its negligible parameter and storage footprints are practical for deployment on resource-constrained hardware, and its task-agnostic inference aligns with real-world operational requirements. The strong empirical results indicate that prototype-based optimization can rival weight-tuned SOTA, challenging conventional wisdom regarding CL in multimodal systems.

Theoretically, the findings highlight the sufficiency of semantic anchor accumulation and distribution replay for embedding space stability, raising questions about the necessity of elaborate parameter-expansion or gradient-based adaptation in future continual learning research.

Future Directions

BETA's framework paves the way for efficient, scalable continual learning under black-box constraints. Future directions include:

• Incorporation of richer replay distributions or capsule representations for finer-grained control.
• Extension to more complex multimodal streams (video, audio, text).
• Integration with privacy-preserving mechanisms and federated continual learning.
• Exploration of reinforcement-driven adaptation policies for online prototype refinement.
• Potential convergence with evolutionary black-box optimization in “learning-as-a-service” scenarios.

Conclusion

"Black-Box Continual Learning for Vision-LLMs" (2606.22999) rigorously addresses the practical challenges of continual learning in VLMs with inaccessible backbones. The BETA method achieves significant performance gains with strict parameter efficiency, outperforming both black-box and white-box baselines. Its modular architecture—focusing on textual prototype optimization, latent distribution replay, and test-time adaptation—demonstrates robust capability across data domains and operational regimes. The research sets the stage for new CL techniques that are deployable, efficient, and theoretically sound, driving the field toward scalable real-world adaptation of large multimodal systems.