FreeTTA: Online Adaptation for Vision-Language Models

• FreeTTA is a training-free test-time adaptation method for vision-language models that adapts online by recursively updating Gaussian distribution parameters without storing past samples or retraining.
• It employs a Bayesian online EM framework to update class means and a shared covariance using VLM text-derived priors, thereby stabilizing predictions under domain shift.
• Empirical results show FreeTTA improves zero-shot recognition accuracy across cross-domain and out-of-distribution benchmarks while maintaining strict deployment constraints.

Searching arXiv for the specified paper and related mentions of FreeTTA. FreeTTA is a test-time adaptation (TTA) method for vision-LLMs (VLMs) such as CLIP that is designed to improve zero-shot recognition under domain shift and distribution shift while preserving strict deployment constraints: no labels, no retraining, no source data, and online prediction one sample at a time. It is introduced in “Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EM” (Dai et al., 9 Jul 2025), which characterizes the method as training-free and universally available, and as the first method in this setting to explicitly model the evolving test distribution while remaining online, training-free, and history-free in the sense that it does not store old samples. A later study, “Ultra-Light Test-Time Adaptation for Vision--LLMs” (Kim, 12 Nov 2025), positions FreeTTA as a Bayesian online-EM baseline for CLIP that estimates means and covariances, thereby providing an additional comparative description of its methodological role.

1. Conceptual position within test-time adaptation

FreeTTA is situated within source-free, label-free test-time adaptation for VLMs. The motivating problem is that zero-shot CLIP-like models often degrade when test data distributions diverge from training data, even though they retain strong open-world recognition capability (Dai et al., 9 Jul 2025). Existing TTA methods for VLMs are described as often relying on assumptions that are inconvenient in deployment: optimization of prompts or other parameters at test time using backpropagation, simultaneous access to batches of test samples, caching of historical test features, or dependence on source-domain statistics (Dai et al., 9 Jul 2025).

The method is formulated around three desired properties: explicit target-distribution modeling, availability, and training-free adaptation (Dai et al., 9 Jul 2025). In this context, “availability” means not needing access to source data, not modifying model weights, and not requiring storage of past test samples or simultaneous access to a batch. “Training-free” means there is no gradient-based optimization or backpropagation during adaptation. The paper’s distinctive claim is that FreeTTA is the first method in this setting to explicitly model the test data distribution while preserving these deployment properties (Dai et al., 9 Jul 2025).

The central intuition is that if target-domain test samples are related, then predictions for an individual sample should benefit from modeling the population of test samples seen so far, even when samples are processed independently upon arrival (Dai et al., 9 Jul 2025). FreeTTA therefore differs from per-sample entropy-minimization approaches and from cache-based training-free methods that rely on stored historical features as instance-level references. Instead, it exploits inter-sample relationships through recursively updated distribution parameters rather than through data retention (Dai et al., 9 Jul 2025).

2. Probabilistic formulation and online EM

FreeTTA is built on Gaussian discriminant analysis (GDA) and a Gaussian mixture model (GMM) in VLM embedding space (Dai et al., 9 Jul 2025). Let $x_t$ denote the feature of the incoming test image at time $t$, obtained from the frozen image encoder $f(\cdot)$, and let there be $K$ classes. The paper assumes that class-conditional test features follow Gaussians with class-specific means and a shared covariance:

$$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$

with

$\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$

where $d$ is the feature dimension, $\mu_y$ is the mean of class $y$, and $\Sigma$ is a covariance matrix shared across classes (Dai et al., 9 Jul 2025).

Under this model, GDA predicts by Bayes’ rule:

$t$0

Substituting the Gaussian likelihood gives

$t$1

and the decision rule is

$t$2

Because test labels are unknown and samples arrive sequentially, the method turns the class-conditional GDA model into an unlabeled Gaussian mixture model,

$t$3

though the actual developed method uses a shared covariance $t$4 across classes (Dai et al., 9 Jul 2025). The latent variable is a class-assignment indicator $t$5, and the posterior responsibility for class $t$6 on sample $t$7 is

$t$8

This makes expectation-maximization (EM) a natural fit: the E-step computes soft assignments of the current sample to classes, and the M-step updates parameter estimates using those soft assignments. FreeTTA uses an online EM variant in which each arriving sample is processed once and the sufficient statistics are updated recursively rather than by revisiting a stored test set (Dai et al., 9 Jul 2025).

3. Initialization from VLM priors and online recursive updates

A defining design choice in FreeTTA is the use of VLM priors for initialization. The frozen text encoder $t$9 produces one feature vector $f(\cdot)$0 for each class prompt $f(\cdot)$1, and these text embeddings initialize the class means:

$f(\cdot)$2

The text-derived class anchors seed the generative model, and the shared covariance is initialized to the identity matrix $f(\cdot)$3, corresponding to an unbiased isotropic initial uncertainty (Dai et al., 9 Jul 2025). The CLIP zero-shot posterior is written as

$f(\cdot)$4

For each incoming test sample $f(\cdot)$5, the online E-step computes the Gaussian-model posterior

$f(\cdot)$6

The paper denotes this posterior by $f(\cdot)$7 (Dai et al., 9 Jul 2025).

The online M-step updates the mixture prior, class means, covariance, and effective counts. The paper gives the following basic recursions:

$f(\cdot)$8

where $f(\cdot)$9 is the effective sample count for class $K$0, initialized to $K$1, and $K$2 is the total number of samples up to time $K$3;

$K$4

$K$5

$K$6

The paper notes that there are typographical issues around normalization constants and count notation, but it describes the intended recursion as maintaining running class responsibilities $K$7, a running total sample count, class means updated by soft assignment, and a shared covariance updated from the new sample’s contribution (Dai et al., 9 Jul 2025). This is the mechanism that allows distribution information to accumulate over time without storing previous features.

4. Confidence-weighted priors and hybrid decision rule

FreeTTA addresses early-stage instability by injecting VLM priors beyond initialization. For the incoming sample $K$8, let the zero-shot CLIP probabilities be $K$9. The paper computes the self-entropy

$$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$0

This entropy is converted into a sample weight

$$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$1

where $$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$2 is a hyperparameter (Dai et al., 9 Jul 2025). Low-entropy samples therefore contribute more strongly to the online sufficient-statistics update, while uncertain samples are downweighted.

With confidence weighting, the online updates become

$$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$3

$$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$4

$$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$5

with count updates

$$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$6

The manuscript again contains some formatting corruption, but the intended mechanism is explicit: replace a hard unit contribution by a confidence-weighted soft contribution (Dai et al., 9 Jul 2025).

After updating the generative model, FreeTTA combines the original zero-shot discriminative branch with logits induced by the Gaussian generative model. Using standard GDA algebra, the Gaussian classifier is written in linear form:

$$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$7

If $$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$8 is the image feature and $$p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),$$9 is the text feature, the final combined logit is

$\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$0

where $\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$1 controls the contribution of the adapted generative branch (Dai et al., 9 Jul 2025). The zero-shot branch preserves semantics, while the Gaussian branch captures target-domain structure.

5. Operational characteristics and comparison with related TTA methods

Operationally, FreeTTA is training-free because the image encoder and text encoder remain frozen and no gradient descent, backpropagation, or VLM parameter tuning occurs at test time (Dai et al., 9 Jul 2025). Adaptation is confined to lightweight statistical parameters: $\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$2, $\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$3, $\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$4, and effective counts. The method assumes access only to the current test sample at inference time and does not require source data, target labels, mini-batches, or explicit storage of historical test samples. Previous information is compressed into current sufficient statistics (Dai et al., 9 Jul 2025).

The paper characterizes the memory cost as roughly $\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$5, arising from $\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$6 class mean vectors of dimension $\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$7, one shared covariance matrix $\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$8, its inverse if used explicitly, and scalar counts and priors (Dai et al., 9 Jul 2025). Per-sample computation is described as dominated by evaluating $\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),$9 Gaussian responsibilities and updating the covariance, with naive full shared-covariance computation on the order of $d$0 because of Mahalanobis operations and covariance updates (Dai et al., 9 Jul 2025). The paper does not present a formal complexity theorem, but frames its practicality as a deployment claim relative to prompt tuning with backpropagation and multiple augmentations.

The method is described as “universally available” in the sense that it can be used broadly across VLMs and datasets so long as image embeddings and class text embeddings are available (Dai et al., 9 Jul 2025). The experiments instantiate it with CLIP, but the method is presented as conceptually model-agnostic over embedding-based VLM classifiers.

Relative to prior TTA methods, the paper draws several contrasts. Optimization-based methods such as TPT and DiffTPT perform prompt learning at test time via entropy or related objectives over multiple augmentations, making them slower and dependent on backpropagation (Dai et al., 9 Jul 2025). Training-based methods such as CoOp and CoCoOp require labeled target-domain or few-shot adaptation data and thus are outside the same strict test-time regime (Dai et al., 9 Jul 2025). Batch-based or source-statistics-based methods such as PromptAlign rely on information unavailable in strict online deployment. Cache-based training-free methods such as TDA are closer in spirit but retain historical test features and use them primarily as instance-level references rather than fitting an explicit target distribution. Methods such as MTA and ZERO are described as training-free and available online, but they do not explicitly model the target distribution (Dai et al., 9 Jul 2025).

A later comparative paper, (Kim, 12 Nov 2025), summarizes FreeTTA as a method that “frames CLIP with a Bayesian EM that estimates means and covariances,” and reports that under a CLIP ViT-B/16 single-pass streaming protocol its “Params updated” are “means + $d$1.” This later characterization is consistent with the original paper’s emphasis on feature-distribution-level adaptation rather than logit-level adaptation alone.

6. Empirical evaluation, ablations, and observed behavior

FreeTTA is evaluated on both cross-domain generalization and natural out-of-distribution robustness using two CLIP backbones, RN50 and ViT-B/16, under batch size 1 and top-1 accuracy (Dai et al., 9 Jul 2025). The cross-domain benchmark includes 10 datasets: FGVCAircraft, Caltech101, StanfordCars, DTD, EuroSAT, Flowers102, Food101, OxfordPets, SUN397, and UCF101. The OOD benchmark uses ImageNet and four ImageNet variants: ImageNet-A, ImageNet-V2, ImageNet-R, and ImageNet-S (Dai et al., 9 Jul 2025). The implementation uses pretrained CLIP, follows TDA’s dataset-specific class templates, sets

$d$2

and runs on an NVIDIA 3090 GPU (Dai et al., 9 Jul 2025).

On the 10-dataset cross-domain benchmark, FreeTTA with RN50 achieves an average accuracy of $d$3, compared with zero-shot CLIP at $d$4, TPT at $d$5, DiffTPT at $d$6, and CoCoOp at $d$7 (Dai et al., 9 Jul 2025). With ViT-B/16, FreeTTA reaches $d$8 average, compared with zero-shot CLIP at $d$9, PromptAlign at $\mu_y$0, TDA at $\mu_y$1, MTA at $\mu_y$2, and ZERO at $\mu_y$3 (Dai et al., 9 Jul 2025). The paper summarizes an average gain of $\mu_y$4 on the cross-domain benchmark relative to zero-shot CLIP.

On the OOD benchmark, FreeTTA with RN50 achieves an overall average of $\mu_y$5 and OOD average of $\mu_y$6, compared with zero-shot CLIP at $\mu_y$7 and $\mu_y$8, DiffTPT at $\mu_y$9 and $y$0, and TPT at $y$1 and $y$2 (Dai et al., 9 Jul 2025). With ViT-B/16, it reaches overall average $y$3 and OOD average $y$4, exceeding zero-shot CLIP ($y$5, $y$6), TDA ($y$7, $y$8), MTA ($y$9, $\Sigma$0), and ZERO ($\Sigma$1, $\Sigma$2) (Dai et al., 9 Jul 2025). The paper reports average gains of $\Sigma$3 on the OOD benchmark over zero-shot CLIP.

Several dataset-level results are explicitly highlighted. For ViT-B/16 on the cross-domain benchmark, FreeTTA performs especially well on EuroSAT ($\Sigma$4), UCF101 ($\Sigma$5), OxfordPets ($\Sigma$6), and SUN397 ($\Sigma$7). On RN50, notable gains are reported on EuroSAT ($\Sigma$8 vs CLIP $\Sigma$9), Flowers102 ($t$00 vs CLIP $t$01), and UCF101 ($t$02 vs CLIP $t$03) (Dai et al., 9 Jul 2025). On the OOD benchmark with ViT-B/16, it reaches $t$04 on ImageNet-A, $t$05 on ImageNet-R, and $t$06 on ImageNet-S, with the last two reported as the best in the table (Dai et al., 9 Jul 2025).

The ablation study clarifies the role of each component. Using CLIP ViT-B/16 on the cross-domain benchmark, zero-shot CLIP starts at $t$07, and full FreeTTA reaches $t$08 (Dai et al., 9 Jul 2025). If mean vectors are not updated and remain fixed at text embeddings, performance becomes $t$09, which the paper interprets as showing that dynamic mean adaptation is crucial. If the covariance matrix is fixed as the identity, performance drops to $t$10, indicating that covariance modeling matters. If VLM priors are removed, performance falls to $t$11, indicating that entropy-based confidence weighting improves robustness to noisy uncertain samples (Dai et al., 9 Jul 2025).

A later comparative study (Kim, 12 Nov 2025) reports FreeTTA as a strong baseline on several additional datasets under identical CLIP ViT-B/16 streaming protocols, with DomainNet $t$12 Top-1 / $t$13 ECE, PACS $t$14, Office-Home $t$15, Terra Incognita $t$16, ImageNet-R $t$17, and ImageNet-A $t$18. The same paper characterizes FreeTTA as heavier than a logit-level alternative, reporting “FLOPs overhead” of $t$19 and “Latency overhead” of $t$20, and describing its long-stream behavior on a 200K-sample DomainNet stream as “mild” drift with “Acc drop” $t$21 (Kim, 12 Nov 2025). These later measurements are comparisons from another paper rather than internal results of the original FreeTTA paper.

7. Limitations, assumptions, and interpretation

FreeTTA assumes that each class distribution in embedding space can be reasonably approximated by a Gaussian with shared covariance (Dai et al., 9 Jul 2025). The original paper identifies this as a limitation when class distributions are highly multimodal or heavy-tailed, or when class semantics are poorly aligned with prompt text features. The method also depends on the zero-shot VLM being at least moderately informative, because text embeddings initialize the means and CLIP confidence governs update reliability (Dai et al., 9 Jul 2025). If the zero-shot model is badly miscalibrated or prompts are poor, the online distribution estimate may drift.

Another practical issue is covariance estimation in high-dimensional feature spaces. The original paper notes that full covariance tracking can be expensive and numerically sensitive, although it does not discuss stabilization beyond initialization and entropy weighting (Dai et al., 9 Jul 2025). Order dependence is another inherent property: early mistakes can influence later estimates, though the confidence weighting is intended to mitigate this (Dai et al., 9 Jul 2025). A later paper reinforces this concern indirectly by describing FreeTTA as exhibiting mild drift in long streams relative to a lighter alternative (Kim, 12 Nov 2025).

The method is likely to work well when the target domain exhibits coherent class-conditional structure in VLM embedding space and when zero-shot CLIP is decent but imperfect (Dai et al., 9 Jul 2025). This suggests a regime of moderate domain shift in which online refinement can leverage evolving target statistics without sacrificing the zero-shot semantic prior. It may struggle when text embeddings are poor initial anchors, when label spaces are highly ambiguous, or when class distributions are strongly non-Gaussian and multimodal. The fact that final prediction retains the zero-shot CLIP branch is presented as a built-in stabilizer against catastrophic drift (Dai et al., 9 Jul 2025).

In broader methodological terms, FreeTTA contributes a specific perspective on VLM test-time adaptation: rather than adapting prompts, calibrating temperatures, or caching target features, it models the test feature distribution as an online-updated Gaussian discriminant model seeded by the VLM’s own semantics (Dai et al., 9 Jul 2025). Its key innovation is the introduction of an online EM formulation into VLM TTA so that unlabeled test samples can inform one another through recursive sufficient statistics without being stored together or optimized over jointly. A plausible implication is that FreeTTA helped establish explicit target-distribution modeling as a viable alternative to both per-sample entropy heuristics and batch-dependent memory mechanisms in online VLM adaptation.