Dual Adaptive Refinement Transfer (DART)

• DART is a framework for open-vocabulary multi-label recognition that refines patch features and transfers relational cues from seen to unseen classes.
• It employs an Adaptive Refinement Module that leverages LoRA-augmented attention and depth-wise convolution to enhance intra-class localization under weak supervision.
• The Adaptive Transfer Module uses LLM-derived class graphs and graph attention networks to propagate structured inter-class relationships for improved prediction.

Searching arXiv for the cited DART papers to ground the article in current metadata. Dual Adaptive Refinement Transfer (DART) is a framework for Open-Vocabulary Multi-Label Recognition (OV-MLR) that enhances a frozen Vision-Language Pre-training (VLP) backbone with two complementary adaptive modules: an Adaptive Refinement Module (ARM) for intra-class localization and an Adaptive Transfer Module (ATM) for inter-class reasoning. In the formulation introduced for OV-MLR, DART is designed to assign an image a set of labels drawn from both seen and unseen classes, using only image-level supervision while combining patch-feature refinement, graph-based relational transfer, and a unified objective over classification and weakly supervised localization (Liu et al., 7 Aug 2025). The acronym “DART” is also used by a distinct 2025 framework for articulated dexterous manipulation, so the term is context-dependent across recent arXiv literature (Zhang et al., 18 Sep 2025).

1. Problem setting and conceptual scope

DART is situated in the OV-MLR setting, where a model must identify multiple object categories in a single image, including categories never encountered during training. The problem formulation emphasizes two simultaneous requirements: generalization to unseen categories via text embeddings, and multi-label prediction under weak supervision, specifically image-level labels rather than patch-level annotations (Liu et al., 7 Aug 2025).

The motivating analysis isolates two core difficulties. The first is fine-grained localization under weak supervision. VLP models such as CLIP provide strong global image-text alignment, but direct patch-text similarity is described as noisy and diffuse, which limits the discriminability of class-specific visual features. The second is relational reasoning for unseen classes. In conventional multi-label recognition, class dependencies can be estimated from training-set statistics, but those statistics do not exist for unseen classes. Off-the-shelf VLP embeddings are characterized as capturing only basic semantic similarity, without explicitly encoding broader structured knowledge or an adaptive mechanism for transferring relational cues (Liu et al., 7 Aug 2025).

Within this problem decomposition, DART consists of two coordinated components. ARM targets the intra-class side by refining patch representations to improve localization. ATM targets the inter-class side by using a Class Relationship Graph (CRG), mined from a LLM, together with Graph Attention Networks (GATs) to transfer structured relational information between class representations. This suggests that the framework’s “dual” structure is not merely architectural duplication, but a partition of the OV-MLR problem into spatial discrimination and relational transfer.

2. Adaptive Refinement Module (ARM)

ARM attaches to the late layers of a frozen Vision Transformer image encoder and produces an input-dependent residual $\Delta \tilde{\mathbf{x}}$ that refines the original patch features $\mathbf{x}_\text{orig}$. The refined patch representation is

$$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$

The module is composed of three sub-modules within each selected transformer layer (Liu et al., 7 Aug 2025).

The first sub-module is LoRA-augmented attention adaptation. Low-rank adapters are injected into the query and output projections. Internally, ARM computes

$$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$

The second sub-module is local context encoding. After excluding the $[CLS]$ token and reshaping to $H \times W \times d$, ARM applies a lightweight depth-wise convolution:

$$\mathbf{x}_\text{arm}^\text{dw} = \mathrm{DWConv}(\mathbf{x}_\text{arm}^\text{lora}).$$

The third sub-module is cross-attention feature integration, where the depth-wise-convolved features serve as queries and the original VLP features act as keys and values:

$$\mathbf{x}_\text{arm}^l = \mathrm{Attn}\bigl( Q=\mathbf{x}_\text{arm}^\text{dw}, K=\mathbf{x}_\text{orig}^l, V=\mathbf{x}_\text{orig}^l \bigr).$$

After $L$ ARM layers, a small FFN generates the residual: $$\Delta \tilde{\mathbf{x}} = \mathrm{FFN}(\mathbf{x}_\text{arm}^L).$$

Architecturally, ARM is designed to refine patch features without replacing the frozen VLP representation. The explicit residual form, together with the later use of an $\mathbf{x}_\text{orig}$0 penalty on $\mathbf{x}_\text{orig}$1, indicates that refinement is intended to be conservative rather than fully reconstructive.

3. Weakly Supervised Patch Selecting (WPS) loss

The supervisory mechanism for ARM is the Weakly Supervised Patch Selecting (WPS) loss, which operates when only image-level labels are available. For image $\mathbf{x}_\text{orig}$2, the positive and negative class sets are

$\mathbf{x}_\text{orig}$3

For each patch $\mathbf{x}_\text{orig}$4 and class $\mathbf{x}_\text{orig}$5, DART defines a refined score and an original-score baseline: $\mathbf{x}_\text{orig}$6

The loss is structured as an E-step and an M-step (Liu et al., 7 Aug 2025). In the E-step, for positive classes, DART estimates a posterior “responsibility” over patches using a Softmax over refined scores: $\mathbf{x}_\text{orig}$7

It also defines a prior responsibility from the original patch scores: $\mathbf{x}_\text{orig}$8

These are smoothed as

$\mathbf{x}_\text{orig}$9

In the M-step, DART optimizes a weighted positive term together with hard-negative mining:

$$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$0

Here $$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$1 is the index set of the top-$$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$2 patches with highest $$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$3, selected via stop-gradient. According to the formulation, minimizing $$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$4 drives ARM to learn discriminative patch features from image-level supervision alone. A common misconception is that open-vocabulary recognition can rely solely on global alignment from a VLP encoder; the WPS construction is introduced precisely because patch-level localization remains underdetermined in that regime.

4. Class Relationship Graph and Adaptive Transfer Module

The inter-class component of DART is built around a Class Relationship Graph

$$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$5

Edges are mined offline through prompt-based LLM queries that request multiple relation types, specifically synonymy, hypernymy, functional, co-occurrence, and part-whole relations, together with justifications and scores. After aggregating multiple LLM runs, each class retains its top-$$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$6 most strongly scored neighbors as edges, and the graph is treated as unweighted during training (Liu et al., 7 Aug 2025).

This construction is intended to compensate for the absence of reliable training-set co-occurrence statistics for unseen classes. The CRG therefore serves as an explicit external knowledge structure rather than a graph estimated from the recognition dataset itself.

ATM applies GATs over this graph in two stages. The first stage, Text-ATM, performs linguistic enhancement. It initializes each node with the class text embedding,

$$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$7

and applies $$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$8 layers of GATv2 message passing. For each layer $$\tilde{\mathbf{x}} = \mathbf{x}_\text{orig} + \Delta \tilde{\mathbf{x}}.$$9 and neighbor $$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$0, DART computes

$$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$1

$$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$2

$$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$3

After $$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$4 layers, the output is the relationally enhanced text feature $$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$5.

The second stage, MM-ATM, performs multi-modal interaction. DART first aggregates class-conditioned visual evidence from refined patches and concatenates it with the original image representation: $$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$6 It then fuses this with the relationally enhanced text feature: $$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$7 Setting $$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$8, DART runs another $$\begin{aligned} \mathbf{Q} &= (\mathbf{W}_q + \mathbf{B}_q\mathbf{A}_q)\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{K} &= \mathbf{W}_k\,\mathbf{x}_\text{orig}^{\,l-1}, \quad \mathbf{V} = \mathbf{W}_v\,\mathbf{x}_\text{orig}^{\,l-1},\ \mathbf{x}_\text{arm}^\text{lora} &= (\mathbf{W}_\text{out} + \mathbf{B}_\text{out}\mathbf{A}_\text{out}) \,\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}) + \mathbf{x}_\text{orig}^{\,l-1}. \end{aligned}$$9 GAT layers to obtain the final multi-modal embedding $[CLS]$0.

Predictions are produced by similarity to the original class text embedding: $[CLS]$1

The paper states that these predictions are trained with a ranking loss $[CLS]$2 as in prior OV-MLR work. Taken together, Text-ATM and MM-ATM separate two transfer operations: relational enhancement in the textual semantic space, followed by graph-based propagation in a fused visual-textual space.

5. Unified objective and training behavior

DART combines its classification, localization, and regularization terms in a single objective: $[CLS]$3

The $[CLS]$4 penalty on the ARM residual is explicitly introduced to prevent catastrophic forgetting of the frozen VLP features (Liu et al., 7 Aug 2025). This is an important design constraint: the VLP encoder remains frozen, and adaptation is concentrated in the residual branch and the graph-based transfer pathway.

The overall training design reflects the paper’s decomposition of OV-MLR into two failure modes. $[CLS]$5 addresses discriminative localization when only image-level labels are available, while ATM addresses the absence of structured inter-class relations for unseen categories. A plausible implication is that DART treats open-vocabulary generalization not as a single embedding-alignment problem, but as a coupled optimization over patch discrimination and class-graph propagation.

The abstract characterizes this joint design as a frozen-VLP enhancement strategy with “two synergistic adaptive modules.” It further describes DART as explicitly integrating external LLM-derived relational knowledge for adaptive inter-class transfer while simultaneously performing adaptive intra-class refinement under weak supervision (Liu et al., 7 Aug 2025).

6. Experimental validation, reported gains, and terminological ambiguity

The empirical evaluation uses three OV-MLR benchmarks: NUS-WIDE with 925 seen and 81 unseen classes, MS-COCO with 48 seen and 17 unseen classes, and Open Images v4 with 7,186 seen and 400 unseen classes (Liu et al., 7 Aug 2025). Metrics include Precision@$[CLS]$6, Recall@$[CLS]$7, F1@$[CLS]$8 for $[CLS]$9, Mean Average Precision (mAP), Zero-Shot Learning (ZSL) on unseen classes only, and Generalized Zero-Shot Learning (GZSL) on all classes.

Dataset	Setting	Reported result
NUS-WIDE	ZSL @$H \times W \times d$0	F1 = 36.1%, mAP = 43.9%
NUS-WIDE	GZSL	F1 = 23.8%, mAP = 22.2%
MS-COCO	ZSL @$H \times W \times d$1	F1 = 52.2%
MS-COCO	GZSL	F1 = 67.8%
Open Images	ZSL @$H \times W \times d$2	F1 = 21.9%, mAP = 70.0%
Open Images	GZSL @$H \times W \times d$3	F1 = 37.7%, mAP = 83.5%

These results are reported as improvements over prior methods: on NUS-WIDE, ZSL F1@$H \times W \times d$4 improves by 2.0% over MKT and mAP by 6.3%, while GZSL F1 improves by 1.8% and mAP by 4.6%; on MS-COCO, ZSL F1@$H \times W \times d$5 improves by 3.3% and GZSL F1 by 2.8%; on Open Images, ZSL F1@$H \times W \times d$6 improves by 2.2% and mAP by 1.9%, while GZSL F1@$H \times W \times d$7 improves by 2.3% and mAP by 2.1% (Liu et al., 7 Aug 2025). The abstract summarizes these outcomes as “new state-of-the-art performance.”

Two contextual clarifications are important. First, the reported gains are specifically tied to the paper’s dual-adaptation hypothesis: precise intra-class refinement under weak supervision and adaptive inter-class transfer of LLM-mined knowledge. Second, the acronym “DART” is not unique in the 2025 arXiv literature. A separate paper, “A Novel Task-Driven Diffusion-Based Policy with Affordance Learning for Generalizable Manipulation of Articulated Objects,” also introduces a framework called DART, but that system concerns diffusion-based control, affordance learning, and linear temporal logic for dexterous manipulation rather than OV-MLR (Zhang et al., 18 Sep 2025). In encyclopedia and bibliographic contexts, disambiguation by full title or task domain is therefore necessary.