---
title: 'FreeTTA: Online Adaptation for Vision-Language Models'
url: https://www.emergentmind.com/topics/freetta
type: topic
---

# FreeTTA: Online Adaptation for Vision-Language Models

Searching arXiv for the specified paper and related mentions of FreeTTA.
FreeTTA is a test-time adaptation (TTA) method for vision-language models (VLMs) such as CLIP that is designed to improve zero-shot recognition under domain shift and distribution shift while preserving strict deployment constraints: no labels, no retraining, no source data, and online prediction one sample at a time. It is introduced in “Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EM” [2507.06973], which characterizes the method as training-free and universally available, and as the first method in this setting to explicitly model the evolving test distribution while remaining online, training-free, and history-free in the sense that it does not store old samples. A later study, “Ultra-Light Test-Time Adaptation for Vision--Language Models” [2511.09101], positions FreeTTA as a Bayesian online-EM baseline for CLIP that estimates means and covariances, thereby providing an additional comparative description of its methodological role.

## 1. Conceptual position within test-time adaptation

FreeTTA is situated within source-free, label-free test-time adaptation for VLMs. The motivating problem is that zero-shot CLIP-like models often degrade when test data distributions diverge from training data, even though they retain strong open-world recognition capability [2507.06973]. Existing TTA methods for VLMs are described as often relying on assumptions that are inconvenient in deployment: optimization of prompts or other parameters at test time using backpropagation, simultaneous access to batches of test samples, caching of historical test features, or dependence on source-domain statistics [2507.06973].

The method is formulated around three desired properties: explicit target-distribution modeling, availability, and training-free adaptation [2507.06973]. In this context, “availability” means not needing access to source data, not modifying model weights, and not requiring storage of past test samples or simultaneous access to a batch. “Training-free” means there is no gradient-based optimization or backpropagation during adaptation. The paper’s distinctive claim is that FreeTTA is the first method in this setting to explicitly model the test data distribution while preserving these deployment properties [2507.06973].

The central intuition is that if target-domain test samples are related, then predictions for an individual sample should benefit from modeling the population of test samples seen so far, even when samples are processed independently upon arrival [2507.06973]. FreeTTA therefore differs from per-sample entropy-minimization approaches and from cache-based training-free methods that rely on stored historical features as instance-level references. Instead, it exploits inter-sample relationships through recursively updated distribution parameters rather than through data retention [2507.06973].

## 2. Probabilistic formulation and online EM

FreeTTA is built on Gaussian discriminant analysis (GDA) and a Gaussian mixture model (GMM) in VLM embedding space [2507.06973]. Let \(x_t\) denote the feature of the incoming test image at time \(t\), obtained from the frozen image encoder \(f(\cdot)\), and let there be \(K\) classes. The paper assumes that class-conditional test features follow Gaussians with class-specific means and a shared covariance:

\[
p(x_t \mid y) = \mathcal{N}(x_t \mid \mu_y, \Sigma),
\]

with

\[
\mathcal{N}(x_t \mid \mu_y, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2} \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right),
\]

where \(d\) is the feature dimension, \(\mu_y\) is the mean of class \(y\), and \(\Sigma\) is a covariance matrix shared across classes [2507.06973].

Under this model, GDA predicts by Bayes’ rule:

\[
P(y \mid x_t) = \frac{P(y)\,p(x_t \mid y)}{\sum_{y'} P(y')\,p(x_t \mid y')}.
\]

Substituting the Gaussian likelihood gives

\[
P(y \mid x_t) = \frac{ \exp\left( -\frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right) }{ \sum_{y'} \exp\left( -\frac{1}{2}(x_t-\mu_{y'})^\top \Sigma^{-1}(x_t-\mu_{y'}) \right) },
\]

and the decision rule is

\[
y^* = \arg\max_y \left( \log P(y) - \frac{1}{2}(x_t-\mu_y)^\top \Sigma^{-1}(x_t-\mu_y) \right).
\]

Because test labels are unknown and samples arrive sequentially, the method turns the class-conditional GDA model into an unlabeled Gaussian mixture model,

\[
p(x) = \sum_{i=1}^{I} \pi_i \,\mathcal{N}(x \mid \mu_i, \Sigma_i),
\]

though the actual developed method uses a shared covariance \(\Sigma\) across classes [2507.06973]. The latent variable is a class-assignment indicator \(z\), and the posterior responsibility for class \(y\) on sample \(x_t\) is

\[
\gamma_{y,t} = P(z_y = 1 \mid x_t).
\]

This makes expectation-maximization (EM) a natural fit: the E-step computes soft assignments of the current sample to classes, and the M-step updates parameter estimates using those soft assignments. FreeTTA uses an online EM variant in which each arriving sample is processed once and the sufficient statistics are updated recursively rather than by revisiting a stored test set [2507.06973].

## 3. Initialization from VLM priors and online recursive updates

A defining design choice in FreeTTA is the use of VLM priors for initialization. The frozen text encoder \(g(\cdot)\) produces one feature vector \(g(t_y)\) for each class prompt \(t_y\), and these text embeddings initialize the class means:

\[
\mu_y = g(t_y), \qquad \Sigma = I.
\]

The text-derived class anchors seed the generative model, and the shared covariance is initialized to the identity matrix \(I\), corresponding to an unbiased isotropic initial uncertainty [2507.06973]. The CLIP zero-shot posterior is written as

\[
P_{\text{CLIP}(y=i \mid x) = \frac{ \exp\left(\cos(f(x), g(t_i))\right) }{ \sum_{k=1}^{K} \exp\left(\cos(f(x), g(t_k))\right) }.
\]

For each incoming test sample \(x_t\), the online E-step computes the Gaussian-model posterior

\[
P_{\text{GAUS}(z_y = 1 \mid x_t) = \frac{ \pi_y \cdot \mathcal{N}(x_t \mid \mu_y, \Sigma) }{ \sum_j \pi_j \cdot \mathcal{N}(x_t \mid \mu_j, \Sigma) }.
\]

The paper denotes this posterior by \(\gamma_{y,t}\) [2507.06973].

The online M-step updates the mixture prior, class means, covariance, and effective counts. The paper gives the following basic recursions:

\[
\pi_y' = \frac{N_y + \gamma_{y,t}}{n_t},
\]

where \(N_y\) is the effective sample count for class \(y\), initialized to \(1/\text{number of classes}\), and \(n_t\) is the total number of samples up to time \(t\);

\[
\mu_y' = \frac{N_y \cdot \mu_y + \gamma_{y,t} \cdot x_t}{N_y + \gamma_{y,t}},
\]

\[
\Sigma' = \frac{ (n_t - 1)\Sigma + \sum_y \gamma_{y,t}(x_t-\mu_y')(x_t-\mu_y')^\top }{ n_t - 1 },
\]

\[
N_y' = N_y + \gamma_{y,t}.
\]

The paper notes that there are typographical issues around normalization constants and count notation, but it describes the intended recursion as maintaining running class responsibilities \(N_y\), a running total sample count, class means updated by soft assignment, and a shared covariance updated from the new sample’s contribution [2507.06973]. This is the mechanism that allows distribution information to accumulate over time without storing previous features.

## 4. Confidence-weighted priors and hybrid decision rule

FreeTTA addresses early-stage instability by injecting VLM priors beyond initialization. For the incoming sample \(x_t\), let the zero-shot CLIP probabilities be \(\{P_{\text{CLIP}(z_y=1\mid x_t)\}_{y=1}^K\). The paper computes the self-entropy

\[
H(x_t) = -\sum_{y=1}^{K} P_{\text{CLIP}(z_y = 1 \mid x_t)} \log P_{\text{CLIP}(z_y = 1 \mid x_t)}.
\]

This entropy is converted into a sample weight

\[
w(h) = e^{-\beta h},
\]

where \(\beta\) is a hyperparameter [2507.06973]. Low-entropy samples therefore contribute more strongly to the online sufficient-statistics update, while uncertain samples are downweighted.

With confidence weighting, the online updates become

\[
\pi_y' = \frac{ N_y + w(H(x_t)) \cdot \gamma_{y,t} }{ n_t' + w(H(x_t)) },
\]

\[
\mu_y' = \frac{ N_y \cdot \mu_y + w(H(x_t)) \cdot \gamma_{y,t} \cdot x_t }{ N_y + w(H(x_t)) \cdot \gamma_{y,t} },
\]

\[
\Sigma' = \frac{ (n_t' - 1)\Sigma + w(H(x_t))\sum_y \gamma_{y,t}(x_t-\mu_y')(x_t-\mu_y')^\top }{ n_t' - 1 },
\]

with count updates

\[
N_y' = N_y + w(H(x_t)) \cdot \gamma_{y,t}, \qquad n_t' = n_{t-1}' + w(H(x_t)).
\]

The manuscript again contains some formatting corruption, but the intended mechanism is explicit: replace a hard unit contribution by a confidence-weighted soft contribution [2507.06973].

After updating the generative model, FreeTTA combines the original zero-shot discriminative branch with logits induced by the Gaussian generative model. Using standard GDA algebra, the Gaussian classifier is written in linear form:

\[
w_y = \Sigma^{-1}\mu_y, \qquad b_y = \log P(y) - \frac{1}{2}\mu_y^\top \Sigma^{-1}\mu_y.
\]

If \(F = f(x_t)\) is the image feature and \(T_y = g(t_y)\) is the text feature, the final combined logit is

\[
\text{logits}_y = F T_y^\top + \alpha \left( w_y^\top F + b_y \right),
\]

where \(\alpha\) controls the contribution of the adapted generative branch [2507.06973]. The zero-shot branch preserves semantics, while the Gaussian branch captures target-domain structure.

## 5. Operational characteristics and comparison with related TTA methods

Operationally, FreeTTA is training-free because the image encoder and text encoder remain frozen and no gradient descent, backpropagation, or VLM parameter tuning occurs at test time [2507.06973]. Adaptation is confined to lightweight statistical parameters: \(\mu_y\), \(\Sigma\), \(\pi_y\), and effective counts. The method assumes access only to the current test sample at inference time and does not require source data, target labels, mini-batches, or explicit storage of historical test samples. Previous information is compressed into current sufficient statistics [2507.06973].

The paper characterizes the memory cost as roughly \(O(Kd + d^2)\), arising from \(K\) class mean vectors of dimension \(d\), one shared covariance matrix \(\Sigma \in \mathbb{R}^{d \times d}\), its inverse if used explicitly, and scalar counts and priors [2507.06973]. Per-sample computation is described as dominated by evaluating \(K\) Gaussian responsibilities and updating the covariance, with naive full shared-covariance computation on the order of \(O(Kd^2)\) because of Mahalanobis operations and covariance updates [2507.06973]. The paper does not present a formal complexity theorem, but frames its practicality as a deployment claim relative to prompt tuning with backpropagation and multiple augmentations.

The method is described as “universally available” in the sense that it can be used broadly across VLMs and datasets so long as image embeddings and class text embeddings are available [2507.06973]. The experiments instantiate it with CLIP, but the method is presented as conceptually model-agnostic over embedding-based VLM classifiers.

Relative to prior TTA methods, the paper draws several contrasts. Optimization-based methods such as TPT and DiffTPT perform prompt learning at test time via entropy or related objectives over multiple augmentations, making them slower and dependent on backpropagation [2507.06973]. Training-based methods such as CoOp and CoCoOp require labeled target-domain or few-shot adaptation data and thus are outside the same strict test-time regime [2507.06973]. Batch-based or source-statistics-based methods such as PromptAlign rely on information unavailable in strict online deployment. Cache-based training-free methods such as TDA are closer in spirit but retain historical test features and use them primarily as instance-level references rather than fitting an explicit target distribution. Methods such as MTA and ZERO are described as training-free and available online, but they do not explicitly model the target distribution [2507.06973].

A later comparative paper, [2511.09101], summarizes FreeTTA as a method that “frames CLIP with a Bayesian EM that estimates means and covariances,” and reports that under a CLIP ViT-B/16 single-pass streaming protocol its “Params updated” are “means + \(\Sigma\).” This later characterization is consistent with the original paper’s emphasis on feature-distribution-level adaptation rather than logit-level adaptation alone.

## 6. Empirical evaluation, ablations, and observed behavior

FreeTTA is evaluated on both cross-domain generalization and natural out-of-distribution robustness using two CLIP backbones, RN50 and ViT-B/16, under batch size 1 and top-1 accuracy [2507.06973]. The cross-domain benchmark includes 10 datasets: FGVCAircraft, Caltech101, StanfordCars, DTD, EuroSAT, Flowers102, Food101, OxfordPets, SUN397, and UCF101. The OOD benchmark uses ImageNet and four ImageNet variants: ImageNet-A, ImageNet-V2, ImageNet-R, and ImageNet-S [2507.06973]. The implementation uses pretrained CLIP, follows TDA’s dataset-specific class templates, sets

\[
\alpha = 0.2, \qquad \beta = 4.5,
\]

and runs on an NVIDIA 3090 GPU [2507.06973].

On the 10-dataset cross-domain benchmark, FreeTTA with RN50 achieves an average accuracy of \(61.33\), compared with zero-shot CLIP at \(56.63\), TPT at \(57.66\), DiffTPT at \(59.85\), and CoCoOp at \(57.23\) [2507.06973]. With ViT-B/16, FreeTTA reaches \(68.42\) average, compared with zero-shot CLIP at \(64.59\), PromptAlign at \(66.92\), TDA at \(67.53\), MTA at \(64.63\), and ZERO at \(64.66\) [2507.06973]. The paper summarizes an average gain of \(3.76\%\) on the cross-domain benchmark relative to zero-shot CLIP.

On the OOD benchmark, FreeTTA with RN50 achieves an overall average of \(49.81\) and OOD average of \(46.88\), compared with zero-shot CLIP at \(46.43\) and \(43.09\), DiffTPT at \(48.71\) and \(45.69\), and TPT at \(47.26\) and \(43.89\) [2507.06973]. With ViT-B/16, it reaches overall average \(65.58\) and OOD average \(64.42\), exceeding zero-shot CLIP (\(61.20\), \(59.42\)), TDA (\(65.01\), \(63.89\)), MTA (\(63.16\), \(61.63\)), and ZERO (\(64.02\), \(62.76\)) [2507.06973]. The paper reports average gains of \(1.66\%\) on the OOD benchmark over zero-shot CLIP.

Several dataset-level results are explicitly highlighted. For ViT-B/16 on the cross-domain benchmark, FreeTTA performs especially well on EuroSAT (\(62.93\)), UCF101 (\(71.16\)), OxfordPets (\(90.11\)), and SUN397 (\(67.76\)). On RN50, notable gains are reported on EuroSAT (\(43.64\) vs CLIP \(25.79\)), Flowers102 (\(68.26\) vs CLIP \(62.77\)), and UCF101 (\(63.97\) vs CLIP \(59.48\)) [2507.06973]. On the OOD benchmark with ViT-B/16, it reaches \(61.41\) on ImageNet-A, \(80.49\) on ImageNet-R, and \(50.88\) on ImageNet-S, with the last two reported as the best in the table [2507.06973].

The ablation study clarifies the role of each component. Using CLIP ViT-B/16 on the cross-domain benchmark, zero-shot CLIP starts at \(64.59\), and full FreeTTA reaches \(68.42\) [2507.06973]. If mean vectors are not updated and remain fixed at text embeddings, performance becomes \(64.64\), which the paper interprets as showing that dynamic mean adaptation is crucial. If the covariance matrix is fixed as the identity, performance drops to \(67.07\), indicating that covariance modeling matters. If VLM priors are removed, performance falls to \(67.78\), indicating that entropy-based confidence weighting improves robustness to noisy uncertain samples [2507.06973].

A later comparative study [2511.09101] reports FreeTTA as a strong baseline on several additional datasets under identical CLIP ViT-B/16 streaming protocols, with DomainNet \(59.3\) Top-1 / \(7.9\) ECE, PACS \(90.7 / 4.1\), Office-Home \(80.4 / 5.5\), Terra Incognita \(49.1 / 9.2\), ImageNet-R \(73.0 / 7.1\), and ImageNet-A \(41.1 / 14.3\). The same paper characterizes FreeTTA as heavier than a logit-level alternative, reporting “FLOPs overhead” of \(+180\%\) and “Latency overhead” of \(+325\%\), and describing its long-stream behavior on a 200K-sample DomainNet stream as “mild” drift with “Acc drop” \(-3.4\%\) [2511.09101]. These later measurements are comparisons from another paper rather than internal results of the original FreeTTA paper.

## 7. Limitations, assumptions, and interpretation

FreeTTA assumes that each class distribution in embedding space can be reasonably approximated by a Gaussian with shared covariance [2507.06973]. The original paper identifies this as a limitation when class distributions are highly multimodal or heavy-tailed, or when class semantics are poorly aligned with prompt text features. The method also depends on the zero-shot VLM being at least moderately informative, because text embeddings initialize the means and CLIP confidence governs update reliability [2507.06973]. If the zero-shot model is badly miscalibrated or prompts are poor, the online distribution estimate may drift.

Another practical issue is covariance estimation in high-dimensional feature spaces. The original paper notes that full covariance tracking can be expensive and numerically sensitive, although it does not discuss stabilization beyond initialization and entropy weighting [2507.06973]. Order dependence is another inherent property: early mistakes can influence later estimates, though the confidence weighting is intended to mitigate this [2507.06973]. A later paper reinforces this concern indirectly by describing FreeTTA as exhibiting mild drift in long streams relative to a lighter alternative [2511.09101].

The method is likely to work well when the target domain exhibits coherent class-conditional structure in VLM embedding space and when zero-shot CLIP is decent but imperfect [2507.06973]. This suggests a regime of moderate domain shift in which online refinement can leverage evolving target statistics without sacrificing the zero-shot semantic prior. It may struggle when text embeddings are poor initial anchors, when label spaces are highly ambiguous, or when class distributions are strongly non-Gaussian and multimodal. The fact that final prediction retains the zero-shot CLIP branch is presented as a built-in stabilizer against catastrophic drift [2507.06973].

In broader methodological terms, FreeTTA contributes a specific perspective on VLM test-time adaptation: rather than adapting prompts, calibrating temperatures, or caching target features, it models the test feature distribution as an online-updated Gaussian discriminant model seeded by the VLM’s own semantics [2507.06973]. Its key innovation is the introduction of an online EM formulation into VLM TTA so that unlabeled test samples can inform one another through recursive sufficient statistics without being stored together or optimized over jointly. A plausible implication is that FreeTTA helped establish explicit target-distribution modeling as a viable alternative to both per-sample entropy heuristics and batch-dependent memory mechanisms in online VLM adaptation.

Source: https://www.emergentmind.com/topics/freetta