LargeMonitor: Monitoring Online Task-Free Continual Learning via Large Pretrained Models

Abstract: Online task-free continual learning (TFCL) requires intelligent agents to sequentially accumulate knowledge from an unbounded, non-stationary data stream under strict single-pass constraints and without any explicit task identifiers. Existing online TFCL paradigms primarily rely on parameter-efficient prompt tuning or dynamic structure expansion driven by training-coupled optimization dynamics, such as empirical loss fluctuations or evolving latent distances. As a result, these training-coupled solvers remain agnostic to the structural origins of distribution drift, mechanically enforcing a fixed strategy across fundamentally distinct streaming variations. To address this gap, we propose LargeMonitor, a framework that leverages large pretrained foundation models to autonomously orchestrate task-free continuous adaptation. Specifically, LargeMonitor introduces a decoupled detection module utilizing the frozen, stable representation space of large vision models (LVMs) to achieve robust, zero-shot drift detection without training-dependent interference or brittle threshold tuning. Upon a confirmed drift, the framework activates a context-aware diagnostic module driven by large multimodal models (LMMs) to interpret the precise semantic etiologies of the stream variation (e.g., novel class emergence vs. environmental domain shift). This dual-stage capability empowers the continuous learner to dynamically deploy adaptive and shift-specific optimization strategies. Extensive experiments across multiple TFCL settings and benchmarks demonstrate that LargeMonitor achieves precise, robust detection and diagnosis of complex data streams while consistently improving the performance of existing online TFCL algorithms.

• The paper introduces a decoupled monitoring pipeline that uses frozen vision models with CKA and CUSUM for zero-shot drift detection, then uses multimodal models to classify shifts and select targeted adaptation policies.
• The method improves continual-learning performance across multiple benchmarks, including a 3.43-point ImageNet-R gain without replay and an AUC increase from 80.51 to 82.14 on the HS-Incremental benchmark.
• LargeMonitor detects abrupt shifts reliably with moderate-sized encoders, but subtle gradual drift may require billion-parameter models, creating practical challenges around memory, latency, and edge deployment.

Motivation and problem setting

Online task-free continual learning (TFCL) requires a learner to process an unbounded, non-stationary stream of samples in a strict single-pass manner, with no task identifiers available at training or inference time. The learner must maintain anytime-inference capability while adapting to arbitrary distribution changes. Existing online TFCL methods fall into two broad families: prompt-based parameter-efficient tuning (e.g., MVP [(Ikeda et al., 2023)-style online prompt methods]) and dynamic structure expansion driven by internal optimization signals (e.g., ODDL, DEMD, Online-LoRA). Both families share a structural weakness: their shift-detection cues—gradient-matching coefficients, empirical loss values, evolving latent distances—are training-coupled. They drift as the learner's representations evolve, require per-dataset threshold tuning, and provide only a binary indication that "something changed," with no capacity to characterize what changed. Consequently, these solvers apply a fixed adaptation strategy to fundamentally heterogeneous shifts such as novel class emergence, environmental domain drift, or sensor corruption.

LargeMonitor addresses this gap by decoupling monitoring from the training loop and delegating it to large pretrained foundation models: frozen large vision models (LVMs) perform zero-shot drift detection in a stable representation space, and large multimodal models (LMMs) diagnose the semantic cause of each detected shift, enabling shift-specific adaptation policies.

Method

Zero-shot drift detection

Each incoming mini-batch $\mathcal{B}_t$ is embedded by a frozen LVM backbone (DINOv3 or CLIP vision encoder), producing features $\bm{F}_t \in \mathbb{R}^{B \times d}$. A FIFO sliding buffer $\mathcal{M}$ of size $M$ maintains recent historical representations. Rather than prototype distances, which collapse variance information, the framework computes linear centered kernel alignment (CKA) between the batch features and the buffered baseline:

$$\mathrm{CKA}(\bm{F}_t, \bm{F}_{\mathcal{M}}) = \frac{\|\bar{\bm{F}}_t^\top \bar{\bm{F}}_{\mathcal{M}}\|_F^2}{\|\bar{\bm{F}}_t^\top \bar{\bm{F}}_t\|_F \|\bar{\bm{F}}_{\mathcal{M}}^\top \bar{\bm{F}}_{\mathcal{M}}\|_F}.$$

A one-sided CUSUM procedure accumulates evidence of sustained downward deviation of the CKA score from its rolling median/MAD baseline:

$$S_t = \max\left(0,\; S_{t-1} + (\tilde{\mu}_t - C_t) - \kappa_d \tilde{\sigma}_t\right),$$

with a dynamic alarm threshold $h_t = \kappa_h \tilde{\sigma}_t$. Because both the encoder and the CUSUM parameters are decoupled from the learner's training dynamics, detection is zero-shot and requires no per-dataset threshold tuning; per-batch cost is $\mathcal{O}(1)$.

Diagnosis via LMMs

Upon a confirmed alarm, an LMM (Qwen3.6-Flash primarily; GPT-4o and other Qwen variants ablated) receives a small subset of images from the current batch—those most distant from the buffer—and classifies the shift into categories such as new class emergence, domain shift, data corruption, or noise/false alarm, entirely zero-shot. The diagnosis then conditions a shift-specific policy: novel classes trigger aggressive plasticity (full replay coefficient, gradient selective forgetting); domain shifts invoke moderate replay plus adaptive feature scaling; corruption triggers conservative replay with a 50% memory skip rate to avoid buffer contamination; false alarms incur no action.

Experimental evaluation

The evaluation spans disjoint class-incremental, Si-Blurry class-incremental, and domain-incremental settings over CIFAR-100, Tiny-ImageNet, ImageNet-R, ImageNet-Sketch, CUB-200, and CORe50, plus a newly introduced heterogeneous shift-incremental (HS-Incremental) benchmark on ImageNet-R that interleaves ten tasks containing new-class, domain-shift, and corruption transitions with ground-truth labels, enabling quantitative evaluation of both detection and diagnosis.

Detection quality. Across all settings and datasets, the CKA trajectory remains stable within tasks and drops sharply at boundaries, including under extreme inter-task imbalance (ImageNet-Sketch under Si-Blurry). Buffer-size ablations show 1024 yields the lowest cumulative regret, with 512 adequate except in the Domain setting; buffers of 64 substantially degrade detection. Encoder-scale ablations reveal a notable asymmetry: DINOv3-ViT-S/16 (21M) suffices for abrupt semantic shifts, but on CORe50—where gradual domain shifts preserve class semantics while altering low-level statistics—only ViT-7B/16 (6,716M) achieves zero regret. This is an important dependency: sensitivity to subtle continuous drift is contingent on very large encoders, which the authors acknowledge as a limitation.

Downstream algorithm enhancement. Augmenting MVP/MVP-R with boundary-triggered modulation of update intensity consistently improves accuracy across buffer sizes. Representative results:

Setting	Method	CIFAR-100 $A_\text{Last}$	Tiny-ImageNet $A_\text{Last}$	ImageNet-R $\bm{F}_t \in \mathbb{R}^{B \times d}$0
Buffer 0	MVP	64.70	59.35	31.71
Buffer 0	MVP+L.M.	63.06	61.36	33.99
Buffer 2000	MVP-R	84.10	76.45	47.29
Buffer 2000	MVP-R+L.M.	84.17	76.53	49.28

Gains are largest where baselines are weakest (e.g., +3.43 points on ImageNet-R at buffer 0), though at buffer 0 on CIFAR-100 the last-accuracy slightly decreases (64.70 → 63.06) even as AUC improves—an instance where the enhancement is not uniformly beneficial.

Diagnosis-guided adaptation on HS-Incremental. With ground-truth-free shift-type-aware policies (MVP-Shift), MVP-R improves from 80.51/82.06 (AUC/last) to 82.14/82.24 on ImageNet-HS, outperforming uniform-response baselines. The qualitative examples show the LMM correctly distinguishing stylistic domain change from category addition.

Limitations and open questions

The authors identify two principal limitations. First, reliance on multi-billion-parameter foundation models introduces substantial memory footprint and inference latency, limiting deployment on edge devices or real-time robotic systems; the paper does not quantify end-to-end latency budgets. Second, CUSUM-based detection is effective for abrupt transitions but its sensitivity to extremely gradual or continuous drift depends heavily on encoder scale, with smaller models exhibiting delayed or missed alarms. Open questions include whether multi-scale historical tracking or self-supervised representation calibration can restore small-model sensitivity, whether the diagnosis taxonomy (class/domain/corruption/noise) covers realistic compound shifts, and how robust LMM diagnoses are when the two compared batches differ along multiple axes simultaneously—the HS-Incremental benchmark evaluates single-type shifts per boundary.

Conclusion

LargeMonitor reframes shift handling in online TFCL as a decoupled detect-and-diagnose pipeline built on frozen LVM representations and zero-shot LMM reasoning, replacing training-coupled triggers and uniform responses with semantically informed, shift-specific adaptation. Empirically it delivers sharp boundary detection across standard benchmarks and consistent—if sometimes modest—downstream gains, with the strongest results under mixed shift types. Its dependence on large encoders for subtle drift and its computational overhead remain the key constraints on practical adoption.