--- title: 'BayesTTA: Bayesian CT-TTA for Vision-Language' url: https://www.emergentmind.com/topics/bayestta type: topic --- # BayesTTA: Bayesian CT-TTA for Vision-Language Searching arXiv for BayesTTA and closely related continual test-time adaptation work. BayesTTA is a Bayesian test-time adaptation framework for vision-language models, introduced for **Continual-Temporal Test-Time Adaptation (CT-TTA)**, a setting in which test distributions evolve gradually over time rather than shifting abruptly [2507.08607]. The method is designed for vision-language models such as CLIP, whose zero-shot recognition performance degrades under temporally evolving distribution shifts such as gradual illumination change or seasonal variation [2507.08607]. BayesTTA addresses this setting by incrementally estimating class-conditional Gaussian mixture distributions without storing raw data, adaptively selecting covariance structures through statistical hypothesis testing, performing calibrated inference using Gaussian discriminant analysis (GDA), and using these calibrated predictions to supervise self-paced adaptation of normalization layers in the visual encoder [2507.08607]. The paper positions this combination as a response to three defects in prior continual test-time adaptation methods under temporal drift: limited memory cache, unreliable entropy-based confidence, and static visual representations [2507.08607]. ## 1. Problem formulation BayesTTA is situated within the broader progression from test-time adaptation (TTA) to continual test-time adaptation (CTTA), and then to CT-TTA [2507.08607]. In ordinary TTA, a model trained on a source distribution is deployed on an unlabeled target distribution and is allowed to adapt online at test time without source data or labels [2507.08607]. CTTA generalizes this to a stream of changing target distributions, with no reset between batches or domains [2507.08607]. BayesTTA further specializes this scenario by formalizing temporally smooth drift, where the distribution at time \(t+1\) remains close to the distribution at time \(t\) in the sense that $$ \mathbb{E}\left[ D_{\mathrm{KL}}(\mathcal{P}_{t} \,\|\, \mathcal{P}_{t+1}) \right] \leq \delta. $$ At time step \(t\), the model receives an unlabeled batch $$ \boldsymbol{X}^t = \{\boldsymbol{x}_i^t\}_{i=1}^{N_t}, $$ and updates parameters according to the online rule $$ \boldsymbol{\theta}_t = \mathcal{A}(\boldsymbol{\theta}_{t-1}, \boldsymbol{X}^t). $$ This formulation enforces causal online adaptation: only the current unlabeled batch and previously accumulated model state may be used [2507.08607]. The paper argues that temporally evolving shifts differ materially from the abrupt corruption or discrete domain-switch patterns often used in CTTA benchmarks [2507.08607]. Examples given include daily illumination changes, satellite imagery across years or seasons, hardware-induced sensor drift, and gradual weather dynamics [2507.08607]. In this setting, temporal continuity is an exploitable structure rather than a nuisance. A plausible implication is that methods designed for abrupt, weakly correlated shifts may fail to exploit long-range continuity and may amplify early pseudo-label errors over long streams. ## 2. Methodological structure BayesTTA combines four components: zero-shot semantic initialization from CLIP, incremental Gaussian distribution estimation, covariance-structure selection via hypothesis testing, and lightweight visual adaptation through normalization layers [2507.08607]. The method uses CLIP’s text prototypes as semantic anchors and models test-time image features as evolving class-conditional Gaussian distributions [2507.08607]. For an image \(\boldsymbol{x}\), the visual encoder produces an embedding $$ \boldsymbol{z} = \boldsymbol{f}_v(\boldsymbol{x}) \in \mathbb{R}^D, $$ while the text encoder provides class prototypes \(\boldsymbol{w}_k\) [2507.08607]. BayesTTA first computes zero-shot sketch logits $$ \ell_k^{\mathrm{sketch}}(\boldsymbol{z}_i^t) = \cos(\boldsymbol{z}_i^t, \boldsymbol{w}_k), $$ followed by sketch probabilities $$ \mathrm{P}_{\mathrm{sketch}}(Y=k\mid \boldsymbol{x}_i^t) = \frac{\exp\left(\ell_k^{\mathrm{sketch}}(\boldsymbol{z}_i^t)/\tau\right)} {\sum_{j=1}^K \exp\left(\ell_j^{\mathrm{sketch}}(\boldsymbol{z}_i^t)/\tau\right)}, $$ with \(\tau = 0.01\) [2507.08607]. These sketch probabilities provide an initial semantic estimate without labels. The class-conditional feature model is $$ p_t(\boldsymbol{z}\mid Y=k)=\mathcal{N}(\boldsymbol{\mu}_k^t,\boldsymbol{\Sigma}_k^t), $$ with mixture marginal $$ p_t(\boldsymbol{z}) = \sum_{k=1}^K \pi_k^t \mathcal{N}(\boldsymbol{\mu}_k^t,\boldsymbol{\Sigma}_k^t). $$ Initialization uses the text prototypes and identity covariance: $$ \boldsymbol{\mu}_k^{0} = \boldsymbol{w}_k,\quad \boldsymbol{\Sigma}_k^{0} = \boldsymbol{I}. $$ Although the paper refers to “class-conditional Gaussian mixture distributions,” the explicit equations instantiate one Gaussian per class [2507.08607]. This suggests that the practical model is a class-indexed Gaussian mixture over the known label space rather than a multi-component latent mixture within each class. A central design choice is that BayesTTA does not store raw past data. Instead, it stores sufficient statistics and model parameters, namely soft counts, class means, covariances, and priors [2507.08607]. This is presented as an alternative to finite replay caches and as a mechanism for long-range temporal memory. ## 3. Incremental Gaussian estimation and discriminant inference BayesTTA updates its generative model through an online EM-style procedure [2507.08607]. Given current parameters at time \(t-1\), the responsibility of class \(k\) for feature \(\boldsymbol{z}_i^t\) is $$ \gamma_{ik}^t = \frac{\pi_k^{t-1}\mathcal{N}(\boldsymbol{z}_i^{t}\mid \boldsymbol{\mu}_k^{t-1}, \boldsymbol{\Sigma}_k^{t-1})} {\sum_{j=1}^K \pi_j^{t-1}\mathcal{N}(\boldsymbol{z}_i^{t}\mid \boldsymbol{\mu}_j^{t-1}, \boldsymbol{\Sigma}_j^{t-1})}. $$ Cumulative soft counts are updated as $$ s_k^t = s_k^{t-1} + \sum_{i=1}^{N_t} \gamma_{ik}^t, $$ with initialization \(s_k^0 = 1\) [2507.08607]. Class means are updated by weighted accumulation: $$ \boldsymbol{\mu}_k^{t} = \frac{s_k^{t-1}\boldsymbol{\mu}_k^{t-1} + \sum_{i=1}^{N_t}\gamma_{ik}^t \boldsymbol{z}_i^t} {s_k^t}. $$ When class-specific covariances are used, the update is $$ \boldsymbol{\Sigma}_k^t = \frac{ s_k^{t-1}\boldsymbol{\Sigma}_k^{t-1} + \sum_{i=1}^{N_t} \gamma_{ik}^t \Delta \boldsymbol{z}_{ik}^t(\Delta \boldsymbol{z}_{ik}^t)^\top } {s_k^t}, $$ where \(\Delta \boldsymbol{z}_{ik}^t = \boldsymbol{z}_i^t - \boldsymbol{\mu}_k^t\) [2507.08607]. When shared covariance is used, the update becomes $$ \boldsymbol{\Sigma}^{t} = \frac{ s^{t-1}\boldsymbol{\Sigma}^{t-1} + \sum_{i=1}^{N_t}\sum_{k=1}^K \gamma_{ik}^t \Delta \boldsymbol{z}_{ik}^t(\Delta \boldsymbol{z}_{ik}^t)^\top } {s^{t-1}+N_t}, $$ with \(s^{t-1}=\sum_{k=1}^K s_k^{t-1}\) [2507.08607]. Priors are normalized soft counts: $$ \pi_k^t = \frac{s_k^t}{\sum_{j=1}^K s_j^t}. $$ Covariance matrices are regularized through shrinkage, $$ \boldsymbol{\Sigma}_k^{t} \leftarrow (1-\epsilon)\,\boldsymbol{\Sigma}_k^t + \epsilon\,\sigma_{\mathrm{prior}}^2 \boldsymbol{I}, $$ and the Moore-Penrose pseudo-inverse is used for inversion when needed [2507.08607]. Prediction uses Gaussian discriminant analysis. The class posterior is $$ \mathrm{P}_t(Y = k \mid \boldsymbol{z}_i^t) = \frac{\mathcal{N}(\boldsymbol{z}_i^t \mid \boldsymbol{\mu}_k^t, \boldsymbol{\Sigma}_k^t)\pi_k^t} {\sum_{j=1}^K \mathcal{N}(\boldsymbol{z}_i^t \mid \boldsymbol{\mu}_j^t, \boldsymbol{\Sigma}_j^t)\pi_j^t}, $$ and the corresponding discriminant score is $$ \mathcal{D}_k(\boldsymbol{z}_i^t) = \log \pi_k^t - \frac{1}{2}(\Delta \boldsymbol{z}_{ik}^t)^\top (\boldsymbol{\Sigma}_k^t)^{-1}\Delta \boldsymbol{z}_{ik}^t - \frac{1}{2}\log |\boldsymbol{\Sigma}_k^t|. $$ If the homogeneous-covariance hypothesis is accepted, the model reduces to an LDA-style shared-covariance classifier; otherwise it uses class-specific covariances in a QDA-style form [2507.08607]. ## 4. Covariance selection and calibrated adaptation A distinctive feature of BayesTTA is adaptive selection between shared and class-specific covariance structures through a statistical test [2507.08607]. The method first projects features to a PCA subspace of dimension \(d=10\), producing \(\hat{\boldsymbol{z}}_i^t \in \mathbb{R}^d\) [2507.08607]. Using sketch probabilities as soft assignments, it computes sketch-weighted class counts, means, and covariances: $$ \hat{n}_k^t = \sum_{\boldsymbol{x}_i^t\in \boldsymbol{X}^t} \mathrm{P}_{\mathrm{sketch}}(Y=k\mid \boldsymbol{x}_i^t), $$ $$ \hat{\boldsymbol{\mu}}_k^t = \frac{1}{\hat{n}_k^t} \sum_{\boldsymbol{x}_i^t\in \boldsymbol{X}^t} \mathrm{P}_{\mathrm{sketch}}(Y=k\mid \boldsymbol{x}_i^t)\, \hat{\boldsymbol{z}}_i^t, $$ $$ \hat{\boldsymbol{\Sigma}}_k^t = \frac{1}{\hat{n}_k^t} \sum_{\boldsymbol{x}_i^t\in \boldsymbol{X}^t} \mathrm{P}_{\mathrm{sketch}}(Y=k\mid \boldsymbol{x}_i^t)\, (\hat{\boldsymbol{z}}_i^t-\hat{\boldsymbol{\mu}}_k^t) (\hat{\boldsymbol{z}}_i^t-\hat{\boldsymbol{\mu}}_k^t)^\top. $$ The null hypothesis is covariance homogeneity, $$ \mathcal{H}_0:\ \Sigma_1=\Sigma_2=\cdots=\Sigma_K, $$ against the alternative that at least one class covariance differs [2507.08607]. Under \(\mathcal{H}_0\), the pooled covariance is $$ \hat{\boldsymbol{\Sigma}}_{\text{pooled}}^t = \frac{1}{\sum_{k=1}^{K}(\hat{n}_k^t - 1)} \sum_{k=1}^{K} (\hat{n}_k^t - 1)\hat{\boldsymbol{\Sigma}}_k^t. $$ BayesTTA then computes Box’s \(M\) statistic, $$ M = \sum_{k=1}^{K} (\hat{n}_k^t - 1)\ln \left|\hat{\boldsymbol{\Sigma}}_{\text{pooled}}^t\right| - \sum_{k=1}^{K} (\hat{n}_k^t - 1)\ln \left|\hat{\boldsymbol{\Sigma}}_k^t\right|, $$ followed by the corrected form \(M^* = M/c\), an approximate \(F\)-statistic, and rejection at significance level \(\kappa = 0.05\) if $$ F > F_{1-\kappa}(d_1,d_2) $$ [2507.08607]. If \(\mathcal{H}_0\) is rejected, BayesTTA uses heterogeneous covariances; otherwise it uses a shared covariance [2507.08607]. The calibrated inference from GDA is not used in isolation. BayesTTA fuses the semantic CLIP sketch and the distribution-aware GDA score as $$ \ell_k^{\mathrm{adapt}}(\boldsymbol{z}_i^t) = \ell_k^{\mathrm{sketch}}(\boldsymbol{z}_i^t) + \alpha \, \mathcal{D}_k(\boldsymbol{z}_i^t), $$ and predicts $$ \hat{y}_i^t = \arg\max_k \ell_k^{\mathrm{adapt}}(\boldsymbol{z}_i^t). $$ This fusion serves as the effective calibration mechanism of the framework [2507.08607]. The paper explicitly contrasts this with entropy-based confidence, arguing that entropy becomes unreliable under continual temporal drift [2507.08607]. A plausible implication is that BayesTTA treats posterior structure induced by the evolving feature distribution as more trustworthy than instantaneous output entropy. These adapted logits then supervise self-paced refinement of the visual encoder by minimizing $$ \mathcal{L}_{\text{refine}} = - \sum_{k=1}^K \mathrm{softmax}(\boldsymbol{\ell}^{\mathrm{adapt}})_k \cdot \log \mathrm{softmax}(\boldsymbol{\ell}^{\mathrm{sketch}})_k. $$ Only normalization layers are updated: LayerNorm for ViT and BatchNorm for ResNet [2507.08607]. The text encoder and the rest of the image encoder remain frozen, and the paper states that no more than \(0.05\%\) of the total parameters are adapted for ViT-B/16 [2507.08607]. Parameter stabilization uses an exponential moving average, $$ \boldsymbol{\theta}_{\mathrm{ema}}^t = \beta \boldsymbol{\theta}_{\mathrm{ema}}^{t-1} + (1-\beta)\boldsymbol{\theta}^t. $$ ## 5. Evaluation protocol and empirical performance BayesTTA is evaluated on a CT-TTA benchmark comprising four temporally evolving datasets: fMoW, Yearbook, RMNIST, and CIFAR10-Light [2507.08607]. The benchmark covers annual satellite imagery, long-term portrait changes, smooth rotation drift, and gradual illumination change, respectively [2507.08607]. The paper also reports generalization on ten standard TTA datasets: Aircraft, Caltech101, StanfordCars, DTD, EuroSAT, Flower102, Food101, OxfordPets, SUN397, and UCF101 [2507.08607]. Experiments use CLIP-ResNet-50 and CLIP-ViT-B/16 with handcrafted prompts, a single NVIDIA RTX 3090 GPU, and batch size \(128\) [2507.08607]. Compared baselines include TENT, CoTTA, RoTTA, TPT, Adaprompt, TDA, WATT, DPE, DynaPrompt, and on the standard TTA setting also DOTA, HisTPT, and BCA [2507.08607]. The reported CT-TTA results show BayesTTA outperforming the listed baselines on all four temporally evolving datasets [2507.08607]. For example, on fMoW the method reaches **15.39%** with ResNet-50 and **23.63%** with ViT-B/16; on Yearbook it reaches **91.73%** and **96.29%**; on RMNIST it reaches **75.07%** and **70.67%**; and on CIFAR10-Light it reaches **81.59%** and **93.68%** [2507.08607]. The introduction highlights **+14.27%** over prior state of the art on RMNIST with ViT-B/16 under CT-TTA [2507.08607]. On the ten-dataset standard TTA benchmark, BayesTTA reports **65.11%** average with ResNet-50 and **71.35%** average with ViT-B/16 [2507.08607]. The paper also reports efficiency comparisons on CIFAR10-Light with CLIP-ViT-B/16, where BayesTTA achieves **93.68%** in **285 s**, compared with CLIP at **85.7%** in **150 s**, TPT at **86.65%** in **9078 s**, TDA at **88.4%** in **593 s**, DPE at **87.1%** in **2730 s**, and DynaPrompt at **81.92%** in **13747 s** [2507.08607]. These numbers support the paper’s claim that BayesTTA maintains efficiency while outperforming state-of-the-art methods [2507.08607]. Ablation studies isolate the roles of the main components [2507.08607]. Removing the covariance hypothesis test, incremental EM, logit fusion, self-paced optimization, or the continual mechanism all reduce performance, with particularly large degradations on RMNIST when the covariance test or self-paced optimization is removed [2507.08607]. This suggests that BayesTTA’s gains do not arise from a single ingredient alone but from the interaction of temporal distribution estimation, covariance adaptation, calibrated fusion, and lightweight encoder refinement. ## 6. Position within related work and interpretation BayesTTA is positioned against several strands of prior adaptation research [2507.08607]. Relative to entropy-minimization methods such as TENT and CoTTA, it replaces heuristic entropy confidence with explicit probabilistic modeling [2507.08607]. Relative to cache-based methods such as TDA, HisTPT, and DynaPrompt, it stores only sufficient statistics rather than raw examples or features [2507.08607]. Relative to prototype-evolution methods such as DPE, it models both first-order and second-order class structure through means and covariances [2507.08607]. Relative to earlier GDA-style methods such as DOTA, it introduces adaptive covariance selection, online EM under temporal streams, fusion with CLIP semantics, and normalization-layer refinement driven by calibrated GDA predictions [2507.08607]. The paper emphasizes that BayesTTA is not merely offline GDA on CLIP features [2507.08607]. It is unlabeled, online, continual, initialized by text prototypes, updated by soft assignments over an evolving stream, and coupled to selective encoder adaptation [2507.08607]. This suggests that its central contribution lies in linking generative calibration and continual visual alignment under temporal drift. The method also carries several assumptions and limitations. The paper notes that hyperparameter tuning is still required and that gradient-based updates introduce moderate overhead relative to plain zero-shot CLIP [2507.08607]. It additionally assumes a fixed known label space with class text prompts and relies on the approximation that CLIP features remain reasonably Gaussian within class under gradual drift [2507.08607]. A plausible implication is that abrupt concept shifts, strong multimodality within classes, or open-set label-space mismatch may challenge the modeling assumptions, although these cases are not analyzed in detail in the paper. In summary, BayesTTA defines CT-TTA as a practically motivated adaptation regime for temporally evolving test streams and proposes a Bayesian framework that integrates zero-shot semantics, online Gaussian estimation, covariance-structure testing, GDA-based calibration, and lightweight normalization adaptation [2507.08607]. Its reported results indicate consistent gains over contemporary TTA and CTTA baselines while preserving relatively low adaptation cost [2507.08607]. Source: https://www.emergentmind.com/topics/bayestta