Diversified Knowledge Divergence (DKD)

• DKD is an active learning query metric that selects unlabeled target samples based on how much their representations change from pre-trained to fine-tuned encoders.
• It combines Prior and Adaptive Knowledge Divergence (PAKD) with Pair-wise Dissimilarity (PD) to jointly measure the source-target knowledge gap and intra-domain diversity.
• Empirical evidence shows that high DKD scores significantly improve segmentation performance under limited annotation budgets in volumetric medical imaging.

Searching arXiv for the cited paper to ground the article in the primary source. Diversified Knowledge Divergence (DKD) is an active learning query metric introduced in “Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning” (Yang et al., 13 Sep 2025). It is designed for target-domain sample selection in Active Source-Free Domain Adaptation (ASFDA) for volumetric medical image segmentation. Within that framework, DKD measures two properties simultaneously: the source-target knowledge gap and intra-domain diversity. Formally, it combines Prior and Adaptive Knowledge Divergence (PAKD), which quantifies how much a target sample’s representation changes from a pre-trained encoder to its current fine-tuned version, with Pair-wise Dissimilarity (PD), which discourages redundant selection among already high-PAKD samples. The resulting score is used inside an Active Test Time Sample Query strategy to prioritize unlabeled target samples for annotation (Yang et al., 13 Sep 2025).

1. Definition and problem setting

DKD is defined for an unlabeled target sample $x_n$ in a setting where a Medical Vision Foundation Model (Med-VFM) is adapted to a target domain without access to source pre-training samples. The metric is part of an ASFDA method proposed to “efficiently adapt Med-VFMs to target domains for volumetric medical image segmentation” by selecting informative samples from the target domain for fine-tuning (Yang et al., 13 Sep 2025).

The metric is explicitly defined as the product of two terms:

1. Prior and Adaptive Knowledge Divergence (PAKD)
2. Pair-wise Dissimilarity (PD)

Using the notation of the paper, the formal equations are:

$$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$

$$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$

Here, the samples $\{x_n^1,x_n^2,\dots\}$ are sorted in descending order by their PAKD scores, and $c$ is the rank of $x_n$ in that ordering.

The final metric is:

$$\text{DKD}(x_n)=\text{PAKD}(x_n)\times \text{PD}(x_n)$$

In this formulation, $E^{(0)},E^{(i)}\in \mathbb{R}^d$ denote embeddings of depth $d$, produced by the pre-trained encoder and its current fine-tuned version, respectively, and $\text{CosDis}(u,v)=1-\frac{u\cdot v}{\|u\|\|v\|}$ is cosine-distance in $$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$0 (Yang et al., 13 Sep 2025).

2. Constituent terms and notation

The first component, PAKD, measures what the paper describes as the “domainness” of a target sample. It computes how much the sample’s feature representation has changed from the pre-trained encoder $$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$1 to the current fine-tuned encoder $$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$2. A large cosine distance indicates that the sample carries “new, unlearned target-domain knowledge” (Yang et al., 13 Sep 2025).

The second component, PD, is defined after sorting the unlabeled pool by descending PAKD. For a sample at rank $$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$3, PD averages cosine distances in the current feature space between that sample and higher-ranked samples, with linear weighting by $$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$4. In the paper’s description, this weighting means that “larger $$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$5 gives more emphasis to dissimilarity from the very top PAKD samples” (Yang et al., 13 Sep 2025).

The notation is compact but specific:

Symbol	Meaning
$$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$6	Pre-trained encoder embedding
$$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$7	Current fine-tuned encoder embedding after $$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$8 rounds of AL
$$\text{PAKD}(x_n)=\text{CosDis}(E^{(0)}(x_n),E^{(i)}(x_n)) =1-\frac{E^{(0)}(x_n)\cdot E^{(i)}(x_n)^T}{\|E^{(0)}(x_n)\|\cdot \|E^{(i)}(x_n)\|}$$9	Index of sample in descending PAKD order
$$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$0	Weighting factor in PD

A notable design property is that “no additional scalar hyperparameters are introduced in DKD beyond the implicit rank weights $$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$1” (Yang et al., 13 Sep 2025). The paper further states that the only design choice in PD is the linear weighting by $$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$2.

This structure makes DKD a multiplicative criterion rather than an additive one. The paper explicitly motivates this by stating that only samples that are both novel relative to the source and diverse relative to other novel samples will rank highly. This suggests that the metric is intended to suppress two failure modes at once: source-similar samples with low adaptation value, and target samples that are informative individually but redundant as a set.

3. Theoretical motivation

The theoretical motivation of DKD is presented in three parts in the source description (Yang et al., 13 Sep 2025).

First, PAKD captures the source-target knowledge gap. Because it compares the embedding of a target sample under the pre-trained encoder and the current fine-tuned encoder, it operationalizes how strongly the model’s representation of that sample has shifted during adaptation. In the authors’ wording, a large value indicates that the sample contains “new, unlearned target-domain knowledge.”

Second, PD captures intra-domain diversity. It is intended to “prevent redundant queries and spread annotation effort across different subregions of the target distribution” by penalizing samples that are highly similar in the current feature space to already high-PAKD samples.

Third, the multiplicative combination yields a joint criterion. The paper states that multiplying PAKD by PD ensures that only samples that are both “novel relative to the source” and “diverse relative to other novel samples” receive high DKD scores (Yang et al., 13 Sep 2025).

A common misconception would be to treat DKD as only a novelty score. The formal construction does not support that reading. PAKD alone measures source-target representation change, but DKD additionally includes PD, which is explicitly designed to encode semantic diversity among high-PAKD candidates. Conversely, treating DKD as only a diversity criterion would also be incomplete, because PD is computed after ranking by PAKD and is multiplied by PAKD rather than used independently.

4. Computation procedure

The source description provides a step-by-step computation procedure for DKD (Yang et al., 13 Sep 2025). Given an unlabeled batch $$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$3, a pre-trained encoder $$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$4, and a current encoder $$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$5, DKD scores are computed as follows:

$\{x_n^1,x_n^2,\dots\}$7

Two implementation properties are explicitly stated. First, “the only design choice in PD is the linear weighting by $$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$6.” Second, “no temperature or scaling factor is applied inside DKD itself; normalization and quantile transforms happen downstream when DKD is fused with other metrics” (Yang et al., 13 Sep 2025).

The paper also notes that, in practice, the authors found the simple “weight = rank index” effective, while uniform weights $$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$7 or more strongly convex weights such as $$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$8 could be experimented with if greater emphasis on top-rank diversity is desired. That statement is presented as hyperparameter guidance rather than as a reported experimental result.

5. Role in Active Test Time Sample Query

DKD is not used in isolation in the full query policy. It is integrated into the Active Test Time Sample Query strategy together with Anatomical Segmentation Difficulty (ASD), another metric in the ASFDA framework (Yang et al., 13 Sep 2025).

After computing DKD for every sample in the unlabeled pool, the score is min-max normalized to $$\text{PD}(x_n)= \frac{ \sum_{k=1}^{c-1} k\cdot \text{CosDis}(E^{(i)}(x_n^c),E^{(i)}(x_n^{c-k})) }{ \sum_{k=1}^{c-1} k }$$9 and then passed through a quantile transform:

$\{x_n^1,x_n^2,\dots\}$0

In parallel, ASD is computed “via foreground-masked entropy and dynamic temperature-scaled background tolerance,” normalized, and quantile-transformed to yield $\{x_n^1,x_n^2,\dots\}$1 (Yang et al., 13 Sep 2025).

The final query score is:

$\{x_n^1,x_n^2,\dots\}$2

Unlabeled samples are then sorted by descending $\{x_n^1,x_n^2,\dots\}$3, and the top $\{x_n^1,x_n^2,\dots\}$4 are selected for annotation in that active round. The paper states that this joint strategy picks samples that are: (a) new relative to source knowledge, (b) not redundant with each other, and (c) uncertain in anatomy segmentation (Yang et al., 13 Sep 2025).

In this larger pipeline, DKD contributes the first two properties, while ASD contributes the third. A plausible implication is that DKD functions as the representation-space component of the querying policy, whereas ASD functions as the task-difficulty component.

6. Empirical evidence and ablation results

The paper reports ablation evidence for DKD on AMOS2022-CT and AMOS2022-MRI, comparing three sampling regimes at 5% and 10% annotation budgets: Random sampling, Top-DKD, and Bottom-DKD (Yang et al., 13 Sep 2025).

The specific results quoted in the source description for 5% budget are as follows:

Dataset	Sampling regime	Mean Dice ± std
AMOS2022-CT	Random	73.31 ± 15.50
AMOS2022-CT	High-DKD	76.14 ± 13.67
AMOS2022-CT	Low-DKD	64.48 ± 14.72
AMOS2022-MRI	Random	35.84 ± 27.14
AMOS2022-MRI	High-DKD	45.23 ± 20.89
AMOS2022-MRI	Low-DKD	26.86 ± 22.67

The source description states that “in every case, sampling by highest DKD significantly outperformed random sampling (Δ ≈ +3 to +9 Dice points) and was far superior to sampling by lowest DKD” (Yang et al., 13 Sep 2025). It further interprets these results as confirmation that DKD successfully identifies “the most informative, novel, and diverse target samples for annotation.”

The comparison between Top-DKD and Bottom-DKD is particularly informative because it shows that the ranking direction matters. The result is not merely that DKD is correlated with downstream utility; rather, high-scoring and low-scoring samples produce materially different adaptation outcomes under the same budgeted annotation regime. This suggests that DKD is being used as a discriminative acquisition signal rather than as a passive descriptive statistic.

7. Position within Med-VFM adaptation

DKD is embedded in a broader attempt to adapt Med-VFMs efficiently for downstream segmentation under limited annotation budgets. The motivating problem is that Med-VFMs have “superior capabilities of interpreting medical images due to the knowledge learned from self-supervised pre-training with extensive unannotated images,” but prior practice often fine-tunes them with a few randomly selected target-domain samples (Yang et al., 13 Sep 2025). The paper argues that there is a need for “an efficient way of fine-tuning Med-VFMs by selecting informative samples to maximize their adaptation performance on target domains.”

Within that context, DKD provides a formal mechanism for selecting target samples without access to source pre-training samples. It does so by using the pre-training knowledge encoded in $\{x_n^1,x_n^2,\dots\}$5 and the current adapted knowledge encoded in $\{x_n^1,x_n^2,\dots\}$6 to guide querying toward source-dissimilar and semantic-diverse target samples. The metric therefore sits at the intersection of source-free adaptation, active learning, and representation-based sample acquisition.

DKD should also be understood as only one component of the complete ASFDA method. The same framework additionally employs Selective Semi-supervised Fine-tuning, which “improve[s] the performance and efficiency of fine-tuning by identifying samples with high reliability from unqueried ones” (Yang et al., 13 Sep 2025). DKD is therefore a querying criterion inside a larger adaptation workflow rather than a standalone segmentation objective.

A plausible implication is that DKD is most appropriately interpreted as a budget-aware acquisition metric for iterative annotation rounds, especially when one wants to exploit the discrepancy between pre-trained and adapted representations. The source description does not claim that DKD replaces segmentation losses, uncertainty estimation generally, or semi-supervised learning; instead, it is integrated with ASD and selective fine-tuning inside a specific active adaptation pipeline (Yang et al., 13 Sep 2025).