Expertise-aware Multi-LLM Collaboration

• EMRC is a framework for selecting and coordinating heterogeneous LLM agents based on estimated expertise and task fit.
• It employs both precomputed domain alignment and dynamic state-aware routing to match agent capabilities with evolving task requirements.
• The system leverages structured workflows, self-assessment, and feedback loops to optimize collaboration in applications like medical decision-making and candidate evaluation.

Expertise-aware Multi-LLM Recruitment and Collaboration (EMRC) denotes a class of multi-agent LLM systems in which agent selection is conditioned on estimated task–expertise fit, and collaboration is subsequently organized through delegation, critique, fusion, adjudication, or ranking. Recent work instantiates this pattern in several closely related ways: a two-stage medical decision-making framework that first recruits models from an expertise table and then performs confidence- and adversarial-driven collaboration; an expertise-domain alignment formulation for collaborative reasoning; a state-aware router that matches evolving interaction history to agent knowledge; and a metacognitive framework in which agents explicitly assess their own competence boundaries before execution (Bao et al., 19 Aug 2025, Xu et al., 12 May 2025, Wang et al., 4 Nov 2025, Wang et al., 17 May 2026). Across these variants, the common premise is that heterogeneous LLMs and personas exhibit non-uniform capability profiles, so multi-agent performance depends not only on collaboration itself but also on recruitment, calibration, and re-routing.

1. Formal characterizations of expertise-aware collaboration

One influential formalization models EMRC as an expertise–domain alignment problem. Xu et al. define a pool of candidate agents $$\mathcal{M}_n=\{\mathcal{A}_1,\dots,\mathcal{A}_n\}$$, a set of domains $D=\{d_1,\dots,d_k\}$, and a relevance matrix $R\in\mathbb{R}^{k\times k}$ whose entry $R_{p,q}$ records how often expertise in domain $q$ was judged relevant to tasks in primary domain $p$. The alignment score for a candidate agent with expertise domain $q$ on a task in domain $p$ is $s(p,q)=R_{p,q}$, and the average team alignment for an $m$-agent team is

$D=\{d_1,\dots,d_k\}$0

A practical recruitment rule chooses agents whose $D=\{d_1,\dots,d_k\}$1, with $D=\{d_1,\dots,d_k\}$2 given as an example; empirically, teams with higher $D=\{d_1,\dots,d_k\}$3 tend to yield higher accuracy, especially in contextual reasoning domains such as Health and Law (Xu et al., 12 May 2025).

A second formalization treats EMRC as sequential state-aware routing. In STRMAC, the system state at step $D=\{d_1,\dots,d_k\}$4 is $D=\{d_1,\dots,d_k\}$5, where $D=\{d_1,\dots,d_k\}$6 is the original query and $D=\{d_1,\dots,d_k\}$7 is the history of agent outputs through step $D=\{d_1,\dots,d_k\}$8. Each agent $D=\{d_1,\dots,d_k\}$9 carries private context $R\in\mathbb{R}^{k\times k}$0, and the router computes a distribution $R\in\mathbb{R}^{k\times k}$1 over agents before selecting $R\in\mathbb{R}^{k\times k}$2. State and expertise are encoded separately: $R\in\mathbb{R}^{k\times k}$3 and $R\in\mathbb{R}^{k\times k}$4, with routing by cosine similarity,

$R\in\mathbb{R}^{k\times k}$5

This formulation shifts EMRC from static domain matching to dynamic state-to-expertise matching (Wang et al., 4 Nov 2025).

These two formulations are complementary rather than mutually exclusive. The first emphasizes precomputed domain relevance and team composition; the second emphasizes online adaptation as interaction history evolves. Together they define the core design space of EMRC.

2. Recruitment mechanisms

In medical decision-making, EMRC is explicitly organized around an offline expertise table and an online selection procedure. Let $R\in\mathbb{R}^{k\times k}$6 be the LLM pool, $R\in\mathbb{R}^{k\times k}$7 the set of medical-department categories, and $R\in\mathbb{R}^{k\times k}$8 the difficulty levels. The expertise table $R\in\mathbb{R}^{k\times k}$9 stores an expertise score $R_{p,q}$0 for model $R_{p,q}$1 on department $R_{p,q}$2 and difficulty $R_{p,q}$3, filled by

$R_{p,q}$4

At inference, a single LLM is first chosen as query classifier by

$R_{p,q}$5

and the system then selects the top-$R_{p,q}$6 models according to $R_{p,q}$7 for the predicted department–difficulty pair $R_{p,q}$8. The reported per-query selection complexity is $R_{p,q}$9 (Bao et al., 19 Aug 2025).

In human-resources assessment, recruitment is framed as task-to-model routing over a registry of available LLMs, such as a “code-LLM,” a “biz-LLM,” or a “multimodal-LLM.” An Expertise-Awareness Module computes a task embedding $q$0 from the job description, evaluates similarity scores $q$1 for each LLM, and routes the prompt to either the top-$q$2 models as an ensemble or the single best model. Here the recruited entities are not only answer generators but also rubric generators, CV analyzers, transcript examiners, multimodal interview agents, aggregation agents, ranking agents, and feedback integration agents (Yuksel et al., 17 Mar 2026).

Other EMRC variants use simpler recruitment rules. In the senior-design framework, the Task Agent decomposes a proposal into subtasks, and the Coordinator Agent maps each subtask to specialized personas through role labels, key phrases, and a predefined domain-expertise mapping. Reassignment occurs when an output fails its rubric, either by refining the prompt for the same agent or by routing the subtask to a different agent whose persona better matches the missing criteria. The paper states explicitly that recruitment is rule-based and that it does not introduce explicit LaTeX-formulated utility or similarity functions (Mushtaq et al., 2 Jan 2025).

Taken together, these mechanisms span a spectrum from keyword-based persona assignment to embedding-based routing to offline expertise tables and online state-aware routers. This suggests that EMRC is not tied to a single selection algorithm; rather, it is defined by the attempt to make recruitment sensitive to heterogeneity in competence.

3. Collaboration architectures and execution protocols

Once agents are recruited, EMRC systems differ sharply in how collaboration is orchestrated. Xu et al. distinguish a structured workflow from diversity-driven integration. In the structured workflow, a Solver generates an initial draft, a Critic inspects and annotates logical gaps, and a Coordinator integrates the result into a final answer. In the diversity-driven paradigm, agents are assigned to different fine-grained sub-domains and generate independent outputs that are fused either by majority vote,

$q$3

or by embedding-based consensus using the mean sentence-BERT embedding of agent outputs (Xu et al., 12 May 2025).

MetaCogAgent replaces fixed-role execution with self-aware delegation. Each initially assigned agent computes a confidence score before execution; if confidence is above threshold, the agent executes the task directly. Otherwise, the framework invokes cross-agent evaluation, in which every other agent recomputes its own confidence on the same task. If the best alternative exceeds threshold, the task is delegated; if no agent is sufficiently confident, the system enters “Collaborative Mode” and aggregates outputs through weighted voting,

$q$4

The reported delegation precision, defined as the fraction of delegated tasks that the recipient solves correctly, is $q$5 (Wang et al., 17 May 2026).

The medical EMRC framework uses a different collaboration protocol centered on confidence fusion and adversarial validation. Each recruited medical agent returns an answer $q$6 and self-reported confidence $q$7, which is calibrated by the agent’s expertise score:

$q$8

These calibrated confidences are normalized to $q$9, the agent with highest $p$0 becomes Judge, and each answer is compared with the Judge’s response through an adversarial consistency metric $p$1. Answers with $p$2 are dropped, the surviving responses are revised over $p$3 layers, and an Aggregator LLM produces the final answer (Bao et al., 19 Aug 2025).

These collaboration protocols correspond to different assumptions about coordination. Structured workflows assume role decomposition; diversity-driven integration assumes complementary independent perspectives; metacognitive delegation assumes agents can estimate when not to act; adversarial validation assumes that inconsistency with a high-expertise judge is diagnostically useful.

4. Self-assessment, calibration, and feedback loops

A defining development in EMRC is the move from static specialization to explicit self-modeling. In MetaCogAgent, each agent is equipped with a Metacognitive Self-Assessment Unit that produces a task-specific confidence score

$p$4

combining verbalized confidence with a profile-based confidence derived from the agent’s historical capability profile $p$5. The framework also computes a metacognitive conflict signal

$p$6

If $p$7, the delegation threshold is tightened to

$p$8

After execution, the agent’s capability profile is updated by exponential moving average,

$p$9

where $q$0 is the binary correctness reward. Ablations report that removing Self-Assessment reduces accuracy from $q$1 to $q$2, removing Adaptive Delegation reduces it to $q$3, removing Boundary Learning reduces it to $q$4, removing Cross-Agent Eval reduces it to $q$5, and removing Verbalized Confidences reduces it to $q$6; the paper interprets Self-Assessment as the largest single contributor and Adaptive Delegation as the next largest (Wang et al., 17 May 2026).

STRMAC addresses adaptation at the routing level rather than at the agent self-assessment level. Given labeled pairs $q$7, it trains the router with a contrastive cross-entropy objective,

$q$8

To avoid exhaustive enumeration of execution paths, the framework uses solution-aware pruning and router-guided exploration in a self-evolving loop. The reported effect is a reduction of training-path construction to approximately $q$9 of exhaustive search for PDDP and approximately $p$0 for EBFC, corresponding to a $p$1 reduction in data collection overhead compared to exhaustive search (Wang et al., 4 Nov 2025).

In HR assessment, calibration and feedback are attached to rubric-based scoring and ranking. For criterion $p$2 and candidate $p$3, the system defines $p$4, normalizes scores via logistic calibration on a calibration set, and aggregates criterion scores by

$p$5

Ranking then proceeds through listwise mini-tournaments aggregated with a Plackett–Luce model, while a Feedback Integration Agent uses post-hire outcomes, retention, and manager feedback to update prompt templates and refine rubric-generation hyperparameters. The framework also logs prompt templates, inputs, raw outputs, evidence pointers, and ranking rationale for auditability (Yuksel et al., 17 Mar 2026).

5. Domain-specific instantiations and reported results

EMRC has been instantiated in collaborative reasoning, medical QA, engineering evaluation, and candidate assessment. The reported benchmarks and metrics differ substantially, which reflects the breadth of the paradigm rather than a single canonical evaluation protocol.

Domain / setting	Framework	Reported results
MetaCog-Eval benchmark	MetaCogAgent (Wang et al., 17 May 2026)	700 tasks across 5 cognitive dimensions; 82.4% accuracy; 1382 API calls; 8.7 pp above the best routing baseline; 5.1% fewer calls than AutoGen; 34% fewer than voting; ECE = 0.087
Collaborative reasoning benchmarks	STRMAC (Wang et al., 4 Nov 2025)	Up to +23.8% absolute Acc gain on EBFC vs Random-Chain; up to +8.1% Acc gain on PDDP; CAS improvements of 2–4×
Medical decision-making	EMRC (Bao et al., 19 Aug 2025)	MMLU-Pro-Health ACC = 74.45%; +2.69 pp over GPT-4-0613; outperforms Debate, MoA, and SelfMoA
Senior design proposal evaluation	Multi-agent engineering framework (Mushtaq et al., 2 Jan 2025)	Overall MAE: MAS = 0.205 vs Single = 0.388; 89.3% accuracy improvement
Candidate assessment and ranking	Agentic AI for HR (Yuksel et al., 17 Mar 2026)	87% of system ratings within one level of human experts across 20 dimensions; NDCG@10% from 0.40 to 0.5134; NDCG@25% to 0.5703 after ~20 iterations

The benchmark structure itself is often part of the contribution. MetaCog-Eval contains 700 tasks spanning Logical Reasoning, Knowledge Retrieval, Code Generation, Mathematical Computation, Commonsense Inference, and a 100-task cross-domain partition, with metrics including Task Accuracy, Delegation Precision, Expected Calibration Error, and API-call count (Wang et al., 17 May 2026). Xu et al. evaluate on four MMLU-Pro domains—Math, Business, Health, and Law—and report that aligned teams outperform misaligned teams in 75% of cases, with contextual domains showing the largest gains (Xu et al., 12 May 2025). The medical EMRC work uses MedQA, NEJMQA, and MMLU-Pro-Health, while STRMAC reports on PDDP and EBFC, and the HR framework evaluates ranking quality with NDCG, Kendall-$p$6 stabilization, and query efficiency (Bao et al., 19 Aug 2025, Wang et al., 4 Nov 2025, Yuksel et al., 17 Mar 2026).

This distribution of results suggests that EMRC is applicable wherever performance depends on matching heterogeneous subtasks to heterogeneous competencies: diagnosis, reasoning, rubric-based assessment, and rank aggregation all fit that template.

6. Limitations, trade-offs, and recurrent misconceptions

A recurrent misconception is that specialization always dominates broader or more holistic reasoning. The engineering study complicates that view: although specialized multi-agent evaluation improved most categories, Breadth & Depth was the one category in which the single-agent baseline had lower MAE, $p$7 versus $p$8 for the multi-agent system (Mushtaq et al., 2 Jan 2025). Another misconception is that adding more agents yields monotonic practical gains. Xu et al. report that performance gain $p$9 grows sub-linearly, that gains are marginal in Math, and that the Token-Performance ratio is greater than $s(p,q)=R_{p,q}$0 in contextual domains but less than $s(p,q)=R_{p,q}$1 in Math; they also note that sequential rationale passing induces $s(p,q)=R_{p,q}$2 context growth (Xu et al., 12 May 2025).

The literature also makes clear that EMRC is not uniformly dynamic. Some frameworks rely on explicit optimization or learned routers, but others retain centralized rule-based coordinators. The senior-design system states that it lacks explicit dynamic recruitment via embedding similarity or formal utility functions and lacks advanced multi-agent negotiation protocols such as auctions or voting (Mushtaq et al., 2 Jan 2025). By contrast, the medical framework depends on an initial pseudo-labeler for department and difficulty classification, which may introduce bias if that model errs, and requires tuning of $s(p,q)=R_{p,q}$3 and $s(p,q)=R_{p,q}$4; it also incurs additional latency from multiple LLM calls (Bao et al., 19 Aug 2025).

Open research problems cluster around scale, synergy, and cost control. STRMAC identifies the combinatorial explosion that arises when moving from single-agent routing to group selection, along with the need to model synergistic versus redundant expertise and to balance cost against accuracy, possibly through a utility analogous to CAS; it also notes that agent embeddings can remain private because the router needs only black-box vector access (Wang et al., 4 Nov 2025). A plausible implication is that future EMRC systems will combine offline expertise estimation, online metacognitive calibration, sparse communication, and cost-aware routing rather than relying on any single mechanism.

EMRC is therefore best understood not as one architecture but as a research program. Its unifying concern is the allocation and coordination of heterogeneous model expertise under uncertainty, with recruitment, collaboration, calibration, and feedback treated as coupled design variables rather than independent modules.