---
title: Dual Active Learning with Multimodal (DAM)
url: https://www.emergentmind.com/topics/dual-active-learning-with-multimodal-dam
type: topic
---

# Dual Active Learning with Multimodal (DAM)

Searching arXiv for the specified DAM paper and closely related multimodal active learning work.
I’ll look up the DAM paper and the two related multimodal active learning papers to ground the article in current arXiv records.
Dual Active Learning with Multimodal, abbreviated DAM, denotes a framework for **source-free active domain adaptation (SFADA)** that integrates sparse human annotations with supervision from a **Vision-and-Language (ViL) foundation model** into a **dual supervisory signal**, then couples the two through **bidirectional distillation** during target-domain adaptation [2509.24896]. In the DAM formulation, a pretrained source model is adapted to an unlabeled target domain without access to source data, under a limited labeling budget, and active querying is used once to select target samples for annotation. DAM is situated within a broader multimodal active learning literature that has emphasized modality imbalance, dominant-modality bias, and the limits of modality-agnostic acquisition, but its mechanism is distinct: rather than balancing multiple raw input modalities within a pool-based classifier, it uses a multimodal foundation model to provide an auxiliary supervision pathway during adaptation [2306.08306] [2603.29677].

## 1. Problem formulation and motivation

DAM is defined in the setting where a **source model** \(S\) has already been pretrained on a labeled source dataset
\[
\{(x_i,y_i)\}_{i=1}^{m}
\]
spanning \(C\) classes, while the **source data are unavailable** during adaptation. The available data consist of an unlabeled target set
\[
\mathcal T=\{x'_i\}_{i=1}^{n}
\]
and a limited labeling budget \(B\). An active query strategy \(\pi\) selects a subset \(\mathcal T_q\subset\mathcal T\) for oracle annotation, and the objective is to minimize the target risk of the adapted target model \(T\) on \(\mathcal T\) [2509.24896].

The paper adopts the **one-shot query setting from MHPL**, in which all queried samples are selected at once using the raw source model \(S\), under
\[
|\mathcal{T}_q|=\rho|\mathcal{T}|
\]
with \(\rho\in(0,1)\). This choice places DAM in a regime where pseudo-label quality and supervision efficiency are critical, because SFDA already lacks source data and therefore depends on source-model predictions, self-training or pseudo-labeling, entropy or diversity regularization, and a small number of target labels [2509.24896].

The motivation for DAM is that recent domain adaptation work has introduced **ViL models** to improve pseudo-label quality or feature alignment, but has often treated **ViL-based** supervision and **human-labeled active samples** as separate sources. DAM is proposed to overcome this separation by “integrat[ing] multimodal supervision from a ViL model to complement sparse human annotations,” thereby forming a **dual supervisory signal** [2509.24896]. The framework is therefore aimed at coordinated fusion rather than a simple additive combination of active labels and foundation-model predictions.

Within the broader multimodal active learning literature, this motivation aligns with a recurring observation: naïvely transferring active learning methods from unimodal settings can produce biased selection toward a dominant information source, and multimodal strategies do not automatically resolve that problem [2306.08306] [2603.29677]. DAM addresses that general concern in a different application setting—SFADA rather than pool-based multimodal classification—by placing the multimodal component in the supervisory loop.

## 2. Architecture of the dual supervisory signal

DAM organizes its method around two named components: **Dual-Focused active Supervision (DFS)** and **Alternating Distillation (ADL)** [2509.24896]. The duality comes from combining:

1. **Sparse human annotations** on actively queried target samples \(\mathcal T_q\).
2. **ViL-based supervision** generated by a prompt-tuned CLIP-like model.

The ViL model \(V\) is based on **CLIP** with a **frozen image encoder**, **frozen text encoder**, and **learnable category prompts only**. For each class \(k\), the prompt \(\mathbb{V}_k\) is structured as
\[
\{v_1, v_2, \cdots, v_m, c_k\}
\]
and initialized as
\[
\mathbb{V}^0_k:\text{``a photo of a }[c_k]\text{''}.
\]
Text embeddings are computed by
\[
\mathbf{w}_k = \mathrm{Enc}_t(\mathbb{V}_k),
\]
while \(f^I\) denotes the image feature from the frozen image encoder [2509.24896].

The ViL prediction is defined as
\[
p_V^{(k)}(x)= \frac{\exp(\text{sim}(f^I,\mathbf w_k)/\tau)}{\sum_{i=1}^C\exp(\text{sim}(f^I,\mathbf w_i)/\tau)},
\]
where \(\text{sim}(\cdot,\cdot)\) is cosine similarity and \(\tau\) is a learned temperature parameter. The paper attributes to this construction **robust semantic priors**, **smoother predictions than a purely target-trained classifier early in adaptation**, and **better class separation under low-label regimes**, while also emphasizing that the ViL model is not task-specific until it is adapted on the actively labeled target samples [2509.24896].

DFS is the mechanism that turns these ingredients into a stable supervisory source. After one-shot active querying, the prompts \(\mathbb V\) are optimized on \(\mathcal T_q\) using
\[
\mathcal L_{C}= \mathcal L_{ce} + \mathcal L_{kg} = \mathbb E_{x\in\mathcal T_q} \text{CE}\big(\mathbf e_y\,\|\, p_V(x)\big) + \frac{1}{C}\sum_{k=1}^{C} ||\mathbf w_k-\mathbf w_k^0||_2^2.
\]
Here, \(\mathbf e_y\) is the one-hot label vector, and \(\mathbf w_k^0=\text{Enc}_t(\mathbb V_k^0)\) is the original prompt embedding. The term \(\mathcal L_{kg}\) is described as anchoring prompts to their pretrained general-purpose semantics, so that prompt tuning remains target-aware without discarding the ViL model’s prior structure [2509.24896].

A central misconception addressed by the paper is that DAM is merely “active learning + CLIP.” The method is defined instead by the fact that active labels **specialize the ViL model itself**, and that this specialized ViL model is then used as an additional teacher throughout adaptation [2509.24896].

## 3. Alternating distillation and optimization

ADL is the mechanism by which DAM turns the dual supervisory signal into a coupled training process. The framework treats the target model \(T\) and the ViL model \(V\) as **mutual teachers**. For a student model \(A\in\{T,V\}\) taught by teacher \(B\in\{V,T\}\), teacher outputs and weights are defined by
\[
\big(\tilde p_{A\leftarrow B}, \mathcal W\big)=
\begin{cases}
\big(e_{y},\beta_q\big), & x\in \mathcal T_q\ \text{with label }y,\\
\big(\mathrm{F}(B(x)/\tau_B),\beta\big), & x\in \mathcal T\setminus \mathcal T_q,
\end{cases}
\]
where \(\mathrm{F}(\cdot)\) is Softmax, and \(\beta_q\ge \beta>0\) are weighting hyperparameters [2509.24896]. The student prediction is
\[
p_A^{\tau_A}(x)=\mathrm{F}(A(x)/\tau_A),
\]
and the paper notes that **hard pseudo-labeling** is a special case when \(\tau_B\to 0\).

In the direction \(T \rightarrow V\), the target model provides confident samples to refine the ViL model. During each epoch, DAM selects the **top-\(N\) most confident samples per class** from target predictions, forms \(X_{\text{top}}\), and uses these together with \(\mathcal T_q\) to train the ViL model through
\[
\mathcal L_{dist}^{V\leftarrow T} = -\mathbb{E}_{x \in X_{\text{top}}\cup\mathcal T_q}\mathcal W(x)\text{CE}\big(\tilde p_{V\leftarrow T}(x)||p_V^{\tau_V}(x)\big).
\]
The paper specifies \(\tau_V=\tau_T=1\) in this direction and states that \(X_{\text{top}}\) carries the target model’s most reliable discriminative knowledge [2509.24896].

In the reverse direction \(V \rightarrow T\), the ViL model supervises the target model on unlabeled target data:
\[
\mathcal L_{dist}^{T\leftarrow V}= -\mathbb E_{x\in\mathcal{T}}\mathcal W(x)\text{CE}\big(\tilde p_{T\leftarrow V}(x)||p_T^{\tau_T}(x)\big),
\]
with \(\tau_T=1\) and \(\tau_V\rightarrow 0\) for hard ViL pseudo-labels. This gives the target network a pseudo-label source that is intended to be semantically grounded and more stable than target-only self-training early in adaptation [2509.24896].

DAM augments this distillation with two regularizers used for the target model. The first is **entropy minimization**,
\[
\mathcal L_{ent} =\mathbb E_{x\in \mathcal T}\text{CE}\Big(p_T(x)\,\|\,p_T(x)\Big),
\]
and the second is **diversity regularization**,
\[
\mathcal{L}_{div} = \sum_{k=1}^{C}\hat p_k\log(\hat p_k) =\text {KL}(\hat p,\frac{1}{C}\mathds{1}(C)) - \log C,
\]
where
\[
\hat p = \mathbb{E}_{x\in \hat{X}_t}[p_T(x)].
\]
The overall objectives are
\[
\mathcal L_T = \mathcal L_{\mathrm{dist}^{T\leftarrow V}}+ \mathcal L_{ent} + \mathcal{L}_{div},
\]
\[
\mathcal L_V=\mathcal L_{\mathrm{dist}^{V\leftarrow T}}+\mathcal L_{kg}.
\]
After convergence, the ViL model is discarded and only the adapted target model is used for inference [2509.24896].

A plausible implication is that DAM treats the multimodal foundation model primarily as a **training-time supervisory instrument** rather than as part of the deployed inference stack. This is consistent with the paper’s explicit report of **zero inference FLOPs** from the ViL component after training [2509.24896].

## 4. Active querying, multimodality, and relation to adjacent research

DAM is **strategy-agnostic** with respect to the active query rule. The framework uses one-shot sample selection based on the raw source model \(S\), and the paper evaluates DAM as a plug-in with several active selection strategies, including **CoreSet**, **MHPL**, **ELPT**, and **BIAS-augmented pipelines** [2509.24896]. In this design, actively selected samples play two roles: they provide supervised target labels, and they also “activate and specialize the ViL prompt learner” [2509.24896].

This role of active samples distinguishes DAM from earlier multimodal active learning methods developed for pool-based multimodal classification. In **Balanced Multimodal Active Learning (BMMAL)**, for example, the objective is to avoid biased acquisition toward the dominant modality by modulating BADGE-style gradient embeddings with modality contributions estimated from **Shapley values** [2306.08306]. BMMAL operates on multimodal inputs directly, with modality-specific encoders, late fusion by concatenation,
\[
z_{mm} = z_{m_1} \oplus z_{m_2},
\]
and a joint training objective
\[
\mathcal{L}_{final} =  \frac{1}{3} \big[\mathcal{L}_{CE}(f_{m_1}, y)+\mathcal{L}_{CE}(f_{m_2}, y)+\mathcal{L}_{CE}(f_{mm}, y)\big],
\]
so its balancing mechanism is located in the **acquisition geometry** itself [2306.08306].

The benchmark study “Mind the Gap: A Framework for Assessing Pitfalls in Multimodal Active Learning” formalizes a related but broader critique. It identifies **missing modalities**, **modality imbalance**, and different **interaction structures**—shared, unique, and complementary/synergistic information—as multimodal pitfalls absent in the unimodal case, and reports that “models consistently develop imbalanced representations, relying primarily on one modality while neglecting others,” while “multimodal strategies do not provide clear advantages over unimodal ones in our experiments” [2603.29677]. That analysis concerns multimodal active learning in the standard pool-based sense, whereas DAM uses multimodality through a foundation-model supervision path.

This comparison clarifies a common ambiguity around the acronym. In DAM, “dual” refers to the fusion of **human labels** and **ViL-based supervision**, followed by mutual distillation; it does not denote a classical two-view disagreement-based active learner. This suggests that DAM should be read as a framework for **dual supervision in active adaptation**, not as a synonym for all modality-balanced query methods.

## 5. Experimental evaluation and reported performance

DAM is evaluated on **Office**, **Office-Home**, and **VisDA-C**, with **ResNet-50** used for Office and Office-Home, **ResNet-101** for VisDA-C, and CLIP with **ViT-B/32** image encoder for the ViL component [2509.24896]. The default labeling budget is \(\rho = 5\%\), unless otherwise noted, and the reported hyperparameters include \(\beta_q = 3\), \(\beta = 0.3\), prompt length \(m = 16\), and top-confidence set size \(N = 16\) [2509.24896].

The paper compares against five categories of methods: **Source-only / zero-shot CLIP**, **UDA methods**, **SFDA methods**, **ADA methods**, and **SFADA methods**. Examples listed include Source-only, ZS-C-B32, RADA-prompt, DAPL, CPGA, SHOT, A2Net, GRAPE, DIFO-C-B32, CLUE, EADA, DUC, CA-ADA, SQAdapt, ELPT, CoreSet, BIAS, and MHPL [2509.24896].

Under a **5% labeling budget**, the paper reports representative results such as:

- **DAM + MHPL**: Office **94.1**, Office-Home **82.0**, VisDA-C **92.0**.
- **DAM + CoreSet**: Office **93.4**, Office-Home **83.9**, VisDA-C **92.0**.
- **DAM + ELPT**: Office **93.5**, Office-Home **81.9**, VisDA-C **91.9** [2509.24896].

The comparison against BIAS-augmented baselines is explicit. The paper highlights that **CoreSet + DAM** improves over **CoreSet + BIAS** by **2.4% / 7.1% / 3.8%** on Office / Office-Home / VisDA-C, and **MHPL + DAM** improves over **MHPL + BIAS** by **0.5% / 1.7% / 0.9%** on the same benchmarks [2509.24896].

The ablation study on Office-Home isolates the contributions of the two main components. With **Source+CLIP** as baseline, the reported results are **69.8 / 85.2** on two transfer tasks; with only \(\mathcal L_C\), **72.7 / 89.3**; with only \(\mathcal L_V\), **70.5 / 85.8**; and with both, **75.2 / 89.3** [2509.24896]. The paper interprets this as showing that DFS contributes strongly, ADL alone helps modestly, and the combination yields the best result.

The computational-cost analysis reports that the ViL prompt-learning module adds only **0.002M** trainable parameters, about **0.48 GB** extra peak GPU memory, and **zero inference FLOPs** because the ViL model is discarded after training [2509.24896]. This makes the training-time supervisory expansion lightweight at deployment.

## 6. Interpretation, scope, and limitations

The DAM paper explains its gains through five factors: **better supervision quality**, **task-aware activation**, **mutual reinforcement**, **complementarity of human labels and multimodal priors**, and **plug-and-play compatibility** with multiple active learning backbones [2509.24896]. In this interpretation, active labels are sparse but accurate, ViL supervision is broad but requires calibration, and the benefit arises from coupling them in an iterative loop rather than using either source in isolation.

The broader multimodal active learning literature provides a useful caution against overgeneralizing this result. The balanced multimodal classification literature argues that standard active learning can over-select samples favored by a dominant modality, and proposes explicit balancing rules to avoid greedy sample selection from that modality [2306.08306]. The 2026 multimodal benchmark study goes further, showing that across synthetic settings for Missing, Share, Unique, and Synergy, modality imbalance consistently emerges and multimodal query methods do not reliably improve the weak modality [2603.29677]. These findings do not contradict DAM, but they delimit what it establishes. DAM demonstrates the value of multimodal **supervision** in SFADA; it does not by itself solve the general problem of modality-aware **acquisition** in multimodal classification.

The paper explicitly notes that DAM is compatible with various active sampling strategies and that performance depends on the chosen sampler. It also states that future work should explore **optimal active sampling methods** within this collaborative framework, since the present focus is on **ViL activation** rather than advanced sampling design [2509.24896]. This suggests that the quality of the initial active query remains structurally important, even though the dual-supervision loop can improve several existing query backbones.

In summary, DAM occupies a specific position within active learning research. It is a **source-free active domain adaptation framework** in which multimodality enters through a **foundation-model teacher**, not through direct multi-input balancing at query time. Its principal technical contribution is the fusion of active labels and prompt-tuned ViL supervision into a coupled distillation process, while its main conceptual significance lies in showing that multimodal priors can be “carefully activated” to strengthen low-budget adaptation when source data are unavailable [2509.24896].

Source: https://www.emergentmind.com/topics/dual-active-learning-with-multimodal-dam