REFEREE: Frequency-Aware Dynamic Prompt Collaborations

• The paper introduces REFEREE, a rehearsal-free SFCDCL approach that combines a source-pretrained ViT with CLIP, achieving superior performance over baselines.
• It employs frequency-aware prompting to enhance low-frequency semantic content and uses uncertainty-aware weighting to mitigate noisy pseudo labels during adaptation.
• The method leverages a frozen backbone with KLDA using random Fourier features, ensuring robust continual classification without accessing source data.

Rehearsal-Free Frequency-Aware Dynamic Prompt Collaborations (REFEREE) is a method for source-free cross-domain continual learning in which source-domain samples are completely prohibited during adaptation, only a pre-trained source model and unlabeled target-domain samples are available, and both source and target domains feature class-incremental task sequences, with labels only on the source side. The method combines a source-pre-trained model with a large-scale vision-LLM, uses a frequency-aware prompting technique that encourages low-frequency components while suppressing high-frequency components, introduces uncertainty-aware weighting to mitigate noisy pseudo labels, and performs continual classification through kernel linear discriminant analysis (KLDA) with a frozen backbone and a random kernel method (Furqon et al., 2 Oct 2025).

1. Problem setting and design objectives

REFEREE is formulated for Source-Free Cross-Domain Continual Learning (SFCDCL). In this setting, the use of source-domain samples is completely prohibited, which distinguishes the method from cross-domain continual learning approaches that still assume a fully labeled source domain. The paper identifies three central difficulties: over-dependence on the source model, noisy pseudo-labels induced by domain shift on unlabeled target data, and double catastrophic forgetting (DCF), meaning that catastrophic forgetting occurs both in the source and the target continual streams (Furqon et al., 2 Oct 2025).

The method addresses these difficulties with four linked ideas. First, it uses a dual-branch architecture comprising a source-pretrained branch and a vision-language branch. Second, it applies frequency-aware prompting so that low-frequency image components are encouraged and high-frequency components are suppressed. Third, it performs uncertainty-aware weighting of class statistics to reduce the influence of unreliable pseudo labels. Fourth, it avoids updating the feature extractor and instead relies on KLDA with random Fourier features (RFF) for classification, so that the backbone remains frozen throughout target adaptation (Furqon et al., 2 Oct 2025).

A common misunderstanding is to read the term “rehearsal-free” as implying an absence of persistent state. REFEREE is rehearsal-free in the sense that it does not retain source-domain samples for adaptation, but it does maintain continually updated class statistics, specifically means and covariance matrices in the KLDA classifier. The paper’s claim is therefore not that no information persists, but that information persists in summary statistics rather than stored exemplars (Furqon et al., 2 Oct 2025).

2. Dual-branch architecture and prediction fusion

The architecture consists of two branches. The source-pretrained branch uses a ViT encoder, pretrained and frozen during target adaptation, and produces logits via a KLDA classifier. The vision-LLM branch uses CLIP, with a frozen visual encoder and text encoder, and produces zero-shot class probabilities by matching visual features against class text prompts in the joint embedding space (Furqon et al., 2 Oct 2025).

For an input $x_i$, the two branches yield probability vectors that are fused: $$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$ The details state that $\alpha$ and $\beta$ are typically defined dynamically based on each branch’s confidence. Pseudo labels are then obtained from the maximum component of $\hat{p}_i$ (Furqon et al., 2 Oct 2025).

The CLIP branch computes class probabilities with a temperature-scaled similarity between visual and textual embeddings: $$p_{\mathrm{VLM}}(y \mid x)= \frac{\exp(\mathrm{sim}(g_{\theta_v}(x), g_{\theta_{tx}}(\omega_k))/\tau)} {\sum_{k=1}^{M}\exp(\mathrm{sim}(g_{\theta_v}(x), g_{\theta_{tx}}(\omega_k))/\tau)}.$$ Here $g_{\theta_v}$ and $g_{\theta_{tx}}$ are the frozen visual and text encoders, and $\omega_k$ denotes class prompts (Furqon et al., 2 Oct 2025).

This dual-branch construction is central to REFEREE’s handling of sub-optimal generalization. The source-pretrained branch supplies domain-specific structure learned from the source model, whereas the CLIP branch contributes domain-invariant semantic information. The paper explicitly presents this as a synergy between a source-pre-trained model and a large-scale vision-LLM (Furqon et al., 2 Oct 2025).

3. Frequency-aware prompting and pseudo-label curation

REFEREE handles domain shift through a frequency-aware prompting technique. Each input image is transformed to the frequency domain using the Discrete Wavelet Transform (DWT), which splits the image into four sub-bands:

• $f_{LL}$: low-low, associated with low-frequency semantic content,
• $$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$0, $$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$1, $$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$2: higher-frequency components, which the paper describes as more noise-sensitive (Furqon et al., 2 Oct 2025).

Two augmented views are then constructed by manipulating the high-frequency bands while keeping $$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$3 unchanged:

• $$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$4: high-frequency parts are set to zero,
• $$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$5: high-frequency parts are randomized with a normal distribution.

The augmented batch is written as

$$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$6

and the training-set notation given in the details is

$$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$7

The stated purpose is to generate frequency-aware augmented samples that are more robust against noisy pseudo labels (Furqon et al., 2 Oct 2025).

Pseudo-label quality is further controlled through an uncertainty-aware weighting scheme based on the entropy of the fused prediction: $$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$8

$$\hat{p}_i = \alpha\, p(y_i \mid x_i) + \beta\, p_{\mathrm{VLM}}(y_i \mid x_i).$$9

Low-entropy predictions receive weights near $\alpha$0, while high-entropy predictions receive weights near $\alpha$1. The paper describes this as weighting the mean and covariance matrix by prediction uncertainties, thereby mitigating the adverse effects of noisy pseudo labels (Furqon et al., 2 Oct 2025).

This design assigns complementary roles to the two curation mechanisms. Frequency manipulation addresses the input side of the noise problem by altering the spectrum of the image, whereas entropy weighting addresses the label side by reducing the contribution of uncertain pseudo-labeled samples. A plausible implication is that REFEREE treats pseudo-label noise as both a representation-level and a statistics-estimation problem.

4. KLDA, random Fourier features, and forgetting mitigation

REFEREE overcomes catastrophic forgetting by freezing the backbone network and shifting continual adaptation into a KLDA classifier guided by random Fourier features. Given a backbone feature $\alpha$2, the random Fourier mapping is

$\alpha$3

where $\alpha$4 and $\alpha$5 (Furqon et al., 2 Oct 2025).

Class statistics are updated online. For class $\alpha$6, the weighted mean is

$\alpha$7

and the shared covariance update is given as

$\alpha$8

The resulting linear discriminant parameters are

$\alpha$9

and the final score is

$\beta$0

Classification is performed by choosing the class with the highest score (Furqon et al., 2 Oct 2025).

The paper emphasizes that the backbone is frozen throughout, and the target adaptation phase uses no gradient descent after the initial source model. It also states that no source data is accessed after initial source model training. In this formulation, forgetting is controlled not by constraining parameter updates in the encoder, but by preventing such updates altogether and updating only the KLDA statistics (Furqon et al., 2 Oct 2025).

The source phase and target phase are separated. In the source phase, labeled source tasks are used to initialize KLDA statistics. In the target phase, each unlabeled target task is processed by obtaining fused pseudo labels, generating frequency-aware augmentations, computing uncertainty weights, and updating KLDA means and covariance with the weighted augmented data (Furqon et al., 2 Oct 2025).

5. Experimental evaluation and ablation evidence

The experiments are reported on four datasets with class-incremental task sequences: VisDA with 12 classes and 4 tasks, Office-31 with 31 classes and 5 tasks, Office-Home with 65 classes and 13 tasks, and DomainNet with 345 classes and 15 tasks (Furqon et al., 2 Oct 2025). The paper compares against up to 13 baselines spanning continual learning, continual domain adaptation, and source-free domain adaptation, including CLAMP, CDCL, AGLA, ICaRL, EWC, and LwF (Furqon et al., 2 Oct 2025).

The main result reported is that REFEREE consistently outperforms all baselines, including methods that still use source-domain data. The paper states that REFEREE is the only approach that does not use any source data during adaptation and still surpasses methods with access to source data (Furqon et al., 2 Oct 2025).

Dataset	REFEREE Avg. (%)	Next best baseline
VisDA	85.12	CLAMP 30.71
Office-31	90.72	CLAMP 79.74
Office-Home	62.97	CLAMP 50.49
DomainNet	32.66	CLAMP 13.27

The ablation study on VisDA isolates the contribution of several components. Removing the CLIP branch reduces performance from 85.12 to 68.60, removing frequency-aware prompting reduces it to 83.79, and removing uncertainty weighting reduces it to 84.44 (Furqon et al., 2 Oct 2025).

Variant	VisDA Avg. (%)
REFEREE	85.12
w/o CLIP branch	68.60
w/o FAP	83.79
w/o Uncertainty Weighting	84.44

The paper further describes the method as single linear pass over data and gradient-free after initial source model, with runtime $\beta$1 in the number of samples. It also reports low sensitivity to the random Fourier feature dimension $\beta$2 and the kernel width (Furqon et al., 2 Oct 2025). Taken together, these observations suggest that the gains are not solely attributable to one module: the CLIP branch is critical for semantic robustness, while frequency-aware augmentation and uncertainty-aware weighting contribute additional robustness under noisy pseudo labeling.

6. Position within rehearsal-free prompting literature

REFEREE belongs to a broader research trajectory in which rehearsal-free continual learning is achieved by freezing large pretrained backbones and moving adaptation into prompts, prompt statistics, or lightweight classifiers. CODA-Prompt learns prompt components that are assembled with input-conditioned weights through an attention-based end-to-end key-query scheme (Smith et al., 2022). INCPrompt combines adaptive key-learners with task-aware prompts, and its prompt-selection histograms are described as dynamic, frequency-, and task-aware (Wang et al., 2024). RefFiL uses collaborative prompt sharing and clustering in rehearsal-free federated domain-incremental learning (Sun et al., 2024). PromptDSI employs a topic-aware prompt pool with neural topic embeddings as fixed keys for rehearsal-free continual document retrieval (Huynh et al., 2024). APT abandons prompt pools and concatenation, instead adding shared prompts directly to the CLS token’s attention computation (2503.07979).

Against this background, REFEREE’s use of the phrase frequency-aware dynamic prompt collaborations has a distinct operational meaning. The paper does not center continual adaptation on prompt-pool querying of the kind emphasized in CODA-Prompt, INCPrompt, or PromptDSI. Instead, the prompt mechanism is tied to frequency-aware augmentation, while catastrophic forgetting is handled through frozen features plus KLDA, and semantic robustness is strengthened by collaboration between the source-pretrained branch and the CLIP branch (Furqon et al., 2 Oct 2025). A plausible implication is that REFEREE shifts the principal burden of continual retention away from prompt optimization and toward uncertainty-weighted statistical classification.

This also clarifies a second misconception: REFEREE is not a source-only or prompt-only method. Its reported advantage arises from the joint use of a source-pretrained model, a large-scale vision-LLM, frequency-aware prompting, uncertainty-aware statistics, and a frozen-backbone KLDA classifier. In the paper’s own formulation, this combination enables REFEREE to beat prior arts having access to source domain samples with significant margins (Furqon et al., 2 Oct 2025).