BayesTTA: Bayesian CT-TTA for Vision-Language

• BayesTTA is a Bayesian test-time adaptation framework that adapts vision-language models to continuous temporal shifts through incremental estimation and calibrated Gaussian discriminant analysis.
• It incrementally updates class-conditional Gaussian mixtures while employing adaptive covariance selection via statistical hypothesis testing to address evolving distributions.
• BayesTTA fuses zero-shot CLIP semantics with distribution-aware calibration to supervise self-paced refinement of normalization layers, achieving state-of-the-art performance.

Searching arXiv for BayesTTA and closely related continual test-time adaptation work. BayesTTA is a Bayesian test-time adaptation framework for vision-LLMs, introduced for Continual-Temporal Test-Time Adaptation (CT-TTA), a setting in which test distributions evolve gradually over time rather than shifting abruptly (Cui et al., 11 Jul 2025). The method is designed for vision-LLMs such as CLIP, whose zero-shot recognition performance degrades under temporally evolving distribution shifts such as gradual illumination change or seasonal variation (Cui et al., 11 Jul 2025). BayesTTA addresses this setting by incrementally estimating class-conditional Gaussian mixture distributions without storing raw data, adaptively selecting covariance structures through statistical hypothesis testing, performing calibrated inference using Gaussian discriminant analysis (GDA), and using these calibrated predictions to supervise self-paced adaptation of normalization layers in the visual encoder (Cui et al., 11 Jul 2025). The paper positions this combination as a response to three defects in prior continual test-time adaptation methods under temporal drift: limited memory cache, unreliable entropy-based confidence, and static visual representations (Cui et al., 11 Jul 2025).

1. Problem formulation

BayesTTA is situated within the broader progression from test-time adaptation (TTA) to continual test-time adaptation (CTTA), and then to CT-TTA (Cui et al., 11 Jul 2025). In ordinary TTA, a model trained on a source distribution is deployed on an unlabeled target distribution and is allowed to adapt online at test time without source data or labels (Cui et al., 11 Jul 2025). CTTA generalizes this to a stream of changing target distributions, with no reset between batches or domains (Cui et al., 11 Jul 2025). BayesTTA further specializes this scenario by formalizing temporally smooth drift, where the distribution at time $t+1$ remains close to the distribution at time $t$ in the sense that

$$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$

At time step $t$, the model receives an unlabeled batch

$$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$

and updates parameters according to the online rule

$$\boldsymbol{\theta}_t = \mathcal{A}(\boldsymbol{\theta}_{t-1}, \boldsymbol{X}^t).$$

This formulation enforces causal online adaptation: only the current unlabeled batch and previously accumulated model state may be used (Cui et al., 11 Jul 2025).

The paper argues that temporally evolving shifts differ materially from the abrupt corruption or discrete domain-switch patterns often used in CTTA benchmarks (Cui et al., 11 Jul 2025). Examples given include daily illumination changes, satellite imagery across years or seasons, hardware-induced sensor drift, and gradual weather dynamics (Cui et al., 11 Jul 2025). In this setting, temporal continuity is an exploitable structure rather than a nuisance. A plausible implication is that methods designed for abrupt, weakly correlated shifts may fail to exploit long-range continuity and may amplify early pseudo-label errors over long streams.

2. Methodological structure

BayesTTA combines four components: zero-shot semantic initialization from CLIP, incremental Gaussian distribution estimation, covariance-structure selection via hypothesis testing, and lightweight visual adaptation through normalization layers (Cui et al., 11 Jul 2025). The method uses CLIP’s text prototypes as semantic anchors and models test-time image features as evolving class-conditional Gaussian distributions (Cui et al., 11 Jul 2025).

For an image $\boldsymbol{x}$, the visual encoder produces an embedding

$$\boldsymbol{z} = \boldsymbol{f}_v(\boldsymbol{x}) \in \mathbb{R}^D,$$

while the text encoder provides class prototypes $\boldsymbol{w}_k$ (Cui et al., 11 Jul 2025). BayesTTA first computes zero-shot sketch logits

$$\ell_k^{\mathrm{sketch}}(\boldsymbol{z}_i^t) = \cos(\boldsymbol{z}_i^t, \boldsymbol{w}_k),$$

followed by sketch probabilities

$t$0

with $t$1 (Cui et al., 11 Jul 2025). These sketch probabilities provide an initial semantic estimate without labels.

The class-conditional feature model is

$t$2

with mixture marginal

$t$3

Initialization uses the text prototypes and identity covariance:

$t$4

Although the paper refers to “class-conditional Gaussian mixture distributions,” the explicit equations instantiate one Gaussian per class (Cui et al., 11 Jul 2025). This suggests that the practical model is a class-indexed Gaussian mixture over the known label space rather than a multi-component latent mixture within each class.

A central design choice is that BayesTTA does not store raw past data. Instead, it stores sufficient statistics and model parameters, namely soft counts, class means, covariances, and priors (Cui et al., 11 Jul 2025). This is presented as an alternative to finite replay caches and as a mechanism for long-range temporal memory.

3. Incremental Gaussian estimation and discriminant inference

BayesTTA updates its generative model through an online EM-style procedure (Cui et al., 11 Jul 2025). Given current parameters at time $t$5, the responsibility of class $t$6 for feature $t$7 is

$t$8

Cumulative soft counts are updated as

$t$9

with initialization $$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$0 (Cui et al., 11 Jul 2025). Class means are updated by weighted accumulation:

$$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$1

When class-specific covariances are used, the update is

$$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$2

where $$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$3 (Cui et al., 11 Jul 2025). When shared covariance is used, the update becomes

$$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$4

with $$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$5 (Cui et al., 11 Jul 2025). Priors are normalized soft counts:

$$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$6

Covariance matrices are regularized through shrinkage,

$$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$7

and the Moore-Penrose pseudo-inverse is used for inversion when needed (Cui et al., 11 Jul 2025).

Prediction uses Gaussian discriminant analysis. The class posterior is

$$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$8

and the corresponding discriminant score is

$$\mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta.$$9

If the homogeneous-covariance hypothesis is accepted, the model reduces to an LDA-style shared-covariance classifier; otherwise it uses class-specific covariances in a QDA-style form (Cui et al., 11 Jul 2025).

4. Covariance selection and calibrated adaptation

A distinctive feature of BayesTTA is adaptive selection between shared and class-specific covariance structures through a statistical test (Cui et al., 11 Jul 2025). The method first projects features to a PCA subspace of dimension $t$0, producing $t$1 (Cui et al., 11 Jul 2025). Using sketch probabilities as soft assignments, it computes sketch-weighted class counts, means, and covariances:

$t$2

$t$3

$t$4

The null hypothesis is covariance homogeneity,

$t$5

against the alternative that at least one class covariance differs (Cui et al., 11 Jul 2025). Under $t$6, the pooled covariance is

$t$7

BayesTTA then computes Box’s $t$8 statistic,

$t$9

followed by the corrected form $$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$0, an approximate $$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$1-statistic, and rejection at significance level $$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$2 if

$$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$3

(Cui et al., 11 Jul 2025). If $$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$4 is rejected, BayesTTA uses heterogeneous covariances; otherwise it uses a shared covariance (Cui et al., 11 Jul 2025).

The calibrated inference from GDA is not used in isolation. BayesTTA fuses the semantic CLIP sketch and the distribution-aware GDA score as

$$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$5

and predicts

$$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$6

This fusion serves as the effective calibration mechanism of the framework (Cui et al., 11 Jul 2025). The paper explicitly contrasts this with entropy-based confidence, arguing that entropy becomes unreliable under continual temporal drift (Cui et al., 11 Jul 2025). A plausible implication is that BayesTTA treats posterior structure induced by the evolving feature distribution as more trustworthy than instantaneous output entropy.

These adapted logits then supervise self-paced refinement of the visual encoder by minimizing

$$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$7

Only normalization layers are updated: LayerNorm for ViT and BatchNorm for ResNet (Cui et al., 11 Jul 2025). The text encoder and the rest of the image encoder remain frozen, and the paper states that no more than $$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$8 of the total parameters are adapted for ViT-B/16 (Cui et al., 11 Jul 2025). Parameter stabilization uses an exponential moving average,

$$\boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t},$$9

5. Evaluation protocol and empirical performance

BayesTTA is evaluated on a CT-TTA benchmark comprising four temporally evolving datasets: fMoW, Yearbook, RMNIST, and CIFAR10-Light (Cui et al., 11 Jul 2025). The benchmark covers annual satellite imagery, long-term portrait changes, smooth rotation drift, and gradual illumination change, respectively (Cui et al., 11 Jul 2025). The paper also reports generalization on ten standard TTA datasets: Aircraft, Caltech101, StanfordCars, DTD, EuroSAT, Flower102, Food101, OxfordPets, SUN397, and UCF101 (Cui et al., 11 Jul 2025).

Experiments use CLIP-ResNet-50 and CLIP-ViT-B/16 with handcrafted prompts, a single NVIDIA RTX 3090 GPU, and batch size $$\boldsymbol{\theta}_t = \mathcal{A}(\boldsymbol{\theta}_{t-1}, \boldsymbol{X}^t).$$0 (Cui et al., 11 Jul 2025). Compared baselines include TENT, CoTTA, RoTTA, TPT, Adaprompt, TDA, WATT, DPE, DynaPrompt, and on the standard TTA setting also DOTA, HisTPT, and BCA (Cui et al., 11 Jul 2025).

The reported CT-TTA results show BayesTTA outperforming the listed baselines on all four temporally evolving datasets (Cui et al., 11 Jul 2025). For example, on fMoW the method reaches 15.39% with ResNet-50 and 23.63% with ViT-B/16; on Yearbook it reaches 91.73% and 96.29%; on RMNIST it reaches 75.07% and 70.67%; and on CIFAR10-Light it reaches 81.59% and 93.68% (Cui et al., 11 Jul 2025). The introduction highlights +14.27% over prior state of the art on RMNIST with ViT-B/16 under CT-TTA (Cui et al., 11 Jul 2025). On the ten-dataset standard TTA benchmark, BayesTTA reports 65.11% average with ResNet-50 and 71.35% average with ViT-B/16 (Cui et al., 11 Jul 2025).

The paper also reports efficiency comparisons on CIFAR10-Light with CLIP-ViT-B/16, where BayesTTA achieves 93.68% in 285 s, compared with CLIP at 85.7% in 150 s, TPT at 86.65% in 9078 s, TDA at 88.4% in 593 s, DPE at 87.1% in 2730 s, and DynaPrompt at 81.92% in 13747 s (Cui et al., 11 Jul 2025). These numbers support the paper’s claim that BayesTTA maintains efficiency while outperforming state-of-the-art methods (Cui et al., 11 Jul 2025).

Ablation studies isolate the roles of the main components (Cui et al., 11 Jul 2025). Removing the covariance hypothesis test, incremental EM, logit fusion, self-paced optimization, or the continual mechanism all reduce performance, with particularly large degradations on RMNIST when the covariance test or self-paced optimization is removed (Cui et al., 11 Jul 2025). This suggests that BayesTTA’s gains do not arise from a single ingredient alone but from the interaction of temporal distribution estimation, covariance adaptation, calibrated fusion, and lightweight encoder refinement.

6. Position within related work and interpretation

BayesTTA is positioned against several strands of prior adaptation research (Cui et al., 11 Jul 2025). Relative to entropy-minimization methods such as TENT and CoTTA, it replaces heuristic entropy confidence with explicit probabilistic modeling (Cui et al., 11 Jul 2025). Relative to cache-based methods such as TDA, HisTPT, and DynaPrompt, it stores only sufficient statistics rather than raw examples or features (Cui et al., 11 Jul 2025). Relative to prototype-evolution methods such as DPE, it models both first-order and second-order class structure through means and covariances (Cui et al., 11 Jul 2025). Relative to earlier GDA-style methods such as DOTA, it introduces adaptive covariance selection, online EM under temporal streams, fusion with CLIP semantics, and normalization-layer refinement driven by calibrated GDA predictions (Cui et al., 11 Jul 2025).

The paper emphasizes that BayesTTA is not merely offline GDA on CLIP features (Cui et al., 11 Jul 2025). It is unlabeled, online, continual, initialized by text prototypes, updated by soft assignments over an evolving stream, and coupled to selective encoder adaptation (Cui et al., 11 Jul 2025). This suggests that its central contribution lies in linking generative calibration and continual visual alignment under temporal drift.

The method also carries several assumptions and limitations. The paper notes that hyperparameter tuning is still required and that gradient-based updates introduce moderate overhead relative to plain zero-shot CLIP (Cui et al., 11 Jul 2025). It additionally assumes a fixed known label space with class text prompts and relies on the approximation that CLIP features remain reasonably Gaussian within class under gradual drift (Cui et al., 11 Jul 2025). A plausible implication is that abrupt concept shifts, strong multimodality within classes, or open-set label-space mismatch may challenge the modeling assumptions, although these cases are not analyzed in detail in the paper.

In summary, BayesTTA defines CT-TTA as a practically motivated adaptation regime for temporally evolving test streams and proposes a Bayesian framework that integrates zero-shot semantics, online Gaussian estimation, covariance-structure testing, GDA-based calibration, and lightweight normalization adaptation (Cui et al., 11 Jul 2025). Its reported results indicate consistent gains over contemporary TTA and CTTA baselines while preserving relatively low adaptation cost (Cui et al., 11 Jul 2025).