PU-ADKA: Expert-Driven LLM Adaptation

• PU-ADKA is a budget-aware, expert-in-the-loop framework that improves domain-specific LLMs in fields such as drug discovery and rare disease research.
• It leverages positive–unlabeled question–expert matching combined with multi-agent reinforcement learning to optimize expert interactions under strict budget constraints.
• Empirical evaluations on the CKAD benchmark show notable improvements in Win Rate and Length-Controlled Win Rate compared to baseline methods.

PU-ADKA, short for Positive Unlabeled Active Domain Knowledge Acquisition, is a budget-aware, expert-in-the-loop framework for improving domain-specific LLMs in highly specialized and cost-sensitive domains such as drug discovery and rare disease research. It treats expert interaction as a constrained resource and jointly optimizes which questions to annotate and which expert should answer each question, under a fixed monetary budget such as $100. The framework combines positive–unlabeled question–expert matching with multi-agent reinforcement learning and uses the acquired expert answers as supervised fine-tuning data for a base LLM, rather than as preference data in an RLHF pipeline (Wu et al., 24 Aug 2025).

1. Problem formulation and motivating setting

PU-ADKA is designed for settings in which base LLMs, including biomedical LLMs, have a fixed knowledge cutoff and often miss dynamic, mechanistic, and tacit expert knowledge. The paper instantiates this setting with Llama2-7B as a base “predated” biomedical model whose knowledge stops around early 2023, while the target domain knowledge comes from 2024 PubMed papers and expert expertise (Wu et al., 24 Aug 2025).

The framework targets domains where knowledge is rapidly evolving, true experts are few and busy, and each consultation is costly. The motivating examples include cancer drug development, including NK cells, CAR-T, and mRNA vaccines, as well as sepsis and rare disease mechanisms. In these regimes, standard strategies are characterized as suboptimal under tight budgets: large-corpus fine-tuning captures static textual knowledge but not nuanced expert judgment; generic human-feedback pipelines do not account for heterogeneous expert competence and cost; and naive expert querying wastes budget on easy, redundant, or poorly routed questions (Wu et al., 24 Aug 2025).

Formally, PU-ADKA assumes an unlabeled question pool $D_{tr} = \{q_i\}_{i=1}^n$, an expert set $\mathcal{E} = \{e_j\}_{j=1}^m$, a fixed budget $B$, and a base LLM $\theta$. It seeks a selected set of question–expert pairs $S \subseteq D_{tr} \times \mathcal{E}$ that maximizes downstream performance after fine-tuning:

$$S^* = \arg\max_{S \subseteq D_{tr} \times \mathcal{E}} F(\theta_S, D_{te})$$

subject to

$\sum_{(q_i, e_j) \in S} c(q_i, e_j) \le B.$

Here, $\theta_S$ is the LLM fine-tuned on annotations from $S$, $F(\theta_S, D_{te})$ is an evaluation metric such as Win Rate or LC_WR, and $\mathcal{E} = \{e_j\}_{j=1}^m$0 is the cost of having expert $\mathcal{E} = \{e_j\}_{j=1}^m$1 annotate question $\mathcal{E} = \{e_j\}_{j=1}^m$2 (Wu et al., 24 Aug 2025).

This formulation makes PU-ADKA a dual active-learning system: it performs active question selection and active expert selection simultaneously.

2. Expert modeling and the positive–unlabeled matching layer

A central design choice in PU-ADKA is to model expert assignment as a positive–unlabeled (PU) classification problem. The paper’s rationale is that if a question is extracted from a paper authored by expert $\mathcal{E} = \{e_j\}_{j=1}^m$3, then $\mathcal{E} = \{e_j\}_{j=1}^m$4 is a positive pair. For other experts, however, the corresponding pairs are not reliable negatives, because another expert may still be capable of answering the question. The unlabeled region therefore represents uncertainty in competence, not absence of competence (Wu et al., 24 Aug 2025).

Experts are represented through several structures. The paper defines an expert capability matrix $\mathcal{E} = \{e_j\}_{j=1}^m$5, where $\mathcal{E} = \{e_j\}_{j=1}^m$6 if expert $\mathcal{E} = \{e_j\}_{j=1}^m$7 is capable of annotating question $\mathcal{E} = \{e_j\}_{j=1}^m$8. In simulation, the top 20 PubMed authors by publication count are treated as proxy experts, and questions are mechanism-focused QA pairs extracted from PubMed papers. Expert cost is heterogeneous: in simulation, per-question costs are proportional to cumulative journal impact factors, with example prices $\mathcal{E} = \{e_j\}_{j=1}^m$9; in the human study, costs are [$B$00.2, $B$10.1, $0.1] to reflect doctor versus senior and junior PhD cost ratios (Wu et al., 24 Aug 2025).

Question and expert representations are derived from Llama2-7B embeddings. The question embedding $B$2 comes from the last hidden layer applied to question text, and the expert embedding $B$3 is the average embedding of the expert’s publication texts. An expert-wise attention network then forms an expert-aware question representation:

$B$4

$B$5

$B$6

The model then computes

$B$7

where $B$8 is an MLP producing the probability that expert $B$9 is a good match for question $\theta$0 (Wu et al., 24 Aug 2025).

The “PU” in PU-ADKA therefore refers specifically to positive–unlabeled question–expert matching. This differs from soft-label PU formulations that assign real-valued positivity scores to unlabeled instances; a related but distinct example is “Soft Label PU Learning” (Zhao et al., 2024), which operates on instance labels rather than expert-routing decisions.

3. PU risk minimization and suitability scoring

The question–expert matcher is trained with a non-negative PU risk estimator. Let $\theta$1 denote positive pairs and $\theta$2 unlabeled pairs, with $\theta$3 positive samples, $\theta$4 unlabeled samples, and prior probability $\theta$5. The training objective is

$\theta$6

The first term evaluates risk on known positives, while the second approximates the negative risk from unlabeled and positive distributions and clips it at zero to avoid negative estimates (Wu et al., 24 Aug 2025).

This yields a suitability score $\theta$$7 used throughout the rest of the framework. In operational terms, the PU learner estimates which expert is likely to provide high-value supervision for a given question even when explicit negative supervision is unavailable. A plausible implication is that PU-ADKA’s matching layer is most valuable in domains where expertise overlaps partially rather than partitioning cleanly, because unlabeled question–expert pairs remain semantically ambiguous rather than truly negative.</p> <p>The paper reports an encoder ablation showing that <strong>Llama2-7B</strong> embeddings outperform <strong>BERT-base</strong> embeddings for this stage: with BERT-base, performance is <strong>WR 16.3<h2 class='paper-heading' id='multi-agent-reinforcement-learning-under-a-fixed-budget'>4. Multi-agent reinforcement learning under a fixed budget</h2> <p>Once the framework has question–expert suitability scores, it treats actual data acquisition as a <strong><a href="https://www.emergentmind.com/topics/multi-agent-reinforcement-learning-marl" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">multi-agent reinforcement learning</a></strong> problem. The RL state includes three variables: the matching score$$\theta$8, the remaining budget $\theta$9, and an expert sampling weight

$S \subseteq D_{tr} \times \mathcal{E}$0

where $S \subseteq D_{tr} \times \mathcal{E}$1 is the number of times expert $S \subseteq D_{tr} \times \mathcal{E}$2 has already been selected and $S \subseteq D_{tr} \times \mathcal{E}$3 is a decay factor encouraging less frequent expert reuse (Wu et al., 24 Aug 2025).

Because multiple agents may choose the same question, PU-ADKA uses a competition rule:

$S \subseteq D_{tr} \times \mathcal{E}$4

The expert with the highest PU score receives the question; other agents must reselect until all chosen assignments are unique (Wu et al., 24 Aug 2025).

Reward design explicitly encodes performance gain per unit cost while favoring question diversity:

$S \subseteq D_{tr} \times \mathcal{E}$5

with diversity score

$S \subseteq D_{tr} \times \mathcal{E}$6

where $S \subseteq D_{tr} \times \mathcal{E}$7 is Euclidean distance between question embeddings (Wu et al., 24 Aug 2025). The use of the minimum distance means that a question is rewarded for being far from the entire previously labeled set, not merely from one representative element.

Training uses Double DQN with TD target

$S \subseteq D_{tr} \times \mathcal{E}$8

The reported default configuration uses 10 agents and samples 5 experts per iteration, together with standard DQN components such as experience replay and target-network updates (Wu et al., 24 Aug 2025).

A recurrent misconception is to treat PU-ADKA as an RLHF method. The paper instead positions RL strictly at the selection layer: RL decides which questions to ask and which experts to route them to, while the acquired answers are used later in ordinary supervised fine-tuning.

5. Integration with LLM fine-tuning and the CKAD benchmark

PU-ADKA uses the selected expert responses as labeled QA pairs for LoRA-based supervised fine-tuning of the base LLM. The reported configuration freezes base weights and applies LoRA only to attention modules, with rank 16, $S \subseteq D_{tr} \times \mathcal{E}$9, dropout 0.1, and AdamW at learning rate $S^* = \arg\max_{S \subseteq D_{tr} \times \mathcal{E}} F(\theta_S, D_{te})$0 (Wu et al., 24 Aug 2025).

The benchmark introduced alongside the framework is CKAD, the Cost-effective Knowledge Acquisition Dataset. CKAD is constructed from 2024 PubMed Central papers in sepsis and cancer NK cell research. Questions are generated by GPT-4o-2024-08-06 with prompts emphasizing mechanisms and processes rather than generic summaries, then manually validated. The dataset is filtered to remove any QA pairs that the base Llama2-7B can already answer correctly, leaving a final collection of 48,219 QA pairs for which the base answerable rate is 0 (Wu et al., 24 Aug 2025).

The train/dev/test split is 38,575 / 4,722 / 4,722. In a quality check on 100 random QA pairs, two PhD biomedical researchers assign a mean score of 3.85 with Cohen’s kappa 0.73, which the paper characterizes as substantial agreement (Wu et al., 24 Aug 2025).

The training loop is iterative. RL selects question–expert pairs within the remaining budget; experts provide answers, either in simulation or through human interaction; the LLM is fine-tuned on the accumulated QA pairs; validation performance is measured by Win Rate (WR) and Length-Controlled Win Rate (LC_WR); and the resulting performance delta feeds the reward for subsequent RL updates (Wu et al., 24 Aug 2025).

6. Empirical results, ablations, and interpretive boundaries

The main experiments use budget $$S^* = \arg\max_{S \subseteq D_{tr} \times \mathcal{E}} F(\theta_S, D_{te})$$1. On CKAD, under a GPT-4o judge, the best baselines achieve approximately WR 12–14% and LC_WR 23–24%, whereas PU-ADKA reaches WR 18.2% (±0.6) and LC_WR 25.6% (±1.0). The fully annotated upper bound reaches WR 22.1% and LC_WR 27.8%. Under a GPT-4-Turbo judge, PU-ADKA reports WR 16.7% (±0.4) and LC_WR 26.5% (±0.9), compared with a best-baseline range of roughly WR 10–12% and LC_WR 22–23%, and a full-data upper bound of WR 19.3% and LC_WR 28.1% (Wu et al., 24 Aug 2025).

In a real-world deployment with a cancer drug development team, including 3 sepsis specialists, 2 cancer specialists, one medical doctor, and four PhD-level researchers, PU-ADKA again outperforms the strongest reported baseline. Under the human annotation setting, Random obtains WR 7.5% and LC_WR 20.3%, LESS + Match-Greedy obtains WR 12.5% and LC_WR 21.2%, and PU-ADKA obtains WR 15.2% (±0.8) and LC_WR 24.3% (±0.9) (Wu et al., 24 Aug 2025).

A notable ablation separates the contribution of the PU module from that of multi-agent RL. Removing PU and using unsupervised embedding similarity while retaining multi-agent RL yields WR 13.3% and LC_WR 23.2%. Retaining PU learning but using single-agent RL yields WR 14.2% and LC_WR 23.0%. Full PU-ADKA reaches WR 16.7% and LC_WR 26.5% under the reported setting, indicating that both components are materially contributory (Wu et al., 24 Aug 2025).

The paper also reports that Cost-Greedy can yield the largest number of annotations without delivering the best downstream performance. This directly counters the assumption that annotation count alone is the relevant optimization target; the reported evidence supports the view that quality, routing, and diversity matter more than raw annotation volume (Wu et al., 24 Aug 2025).

The framework’s stated limitations are computational and methodological rather than conceptual. The paper identifies scalability issues as the number of questions and experts grows, notes that it fixes the agent count at 10 without systematic exploration of that axis, restricts experiments to biomedical domains, and acknowledges the nontrivial overhead of combining PU learning, multi-agent RL, and repeated LoRA fine-tuning. It also notes the limitations of LLM-as-a-judge evaluation in expert-knowledge tasks and therefore reports results with both GPT-4o and GPT-4-Turbo judges (Wu et al., 24 Aug 2025).

Taken together, PU-ADKA occupies a specific position in the LLM adaptation landscape: it is neither corpus-scale domain-adaptive pretraining nor preference-based RLHF, but rather a budget-constrained acquisition controller for expert supervision. Its defining contribution is to cast expert time as an explicitly priced resource and to optimize its use through positive–unlabeled matching, budget-aware multi-agent RL, and parameter-efficient supervised LLM updating (Wu et al., 24 Aug 2025).