HieraMAS: Optimizing Intra-Node LLM Mixtures and Inter-Node Topology for Multi-Agent Systems

Abstract: Multi-agent systems (MAS) built on LLMs have shown strong performance across many tasks. Most existing approaches improve only one aspect at a time, such as the communication topology, role assignment, or LLM routing, while treating each agent as a single, indivisible unit. This misses the opportunity to use mixtures of LLMs within an agent to strengthen role-specific abilities. We propose HieraMAS, a hierarchical collaboration framework that combines intra-node LLM mixtures with an inter-node communication topology. HieraMAS introduces supernodes, where each functional role is implemented by multiple heterogeneous LLMs using a propose-synthesis structure. Optimizing HieraMAS creates unique credit-assignment challenges: final task performance depends heavily on the underlying LLMs' capabilities, which can lead reinforcement methods to incorrectly reward suboptimal configurations. To address this, we use a two-stage algorithm: (1) multi-level reward attribution, which provides fine-grained feedback at both the node level and the overall system level; (2) graph classification for topology selection, which treats choosing the communication structure as a holistic decision rather than optimizing edges one by one. Experiments on reasoning and coding benchmarks show that HieraMAS substantially outperforms existing methods while also delivering better cost-performance trade-offs.

• The paper introduces HieraMAS, which jointly assigns heterogeneous LLMs inside role-based supernodes, prunes agents, and learns sparse communication topologies for multi-agent systems.
• The two-stage method uses multi-level rewards for node-level credit assignment and graph classification for topology selection, avoiding unreliable per-edge policy gradients.
• HieraMAS achieves 94.61% average accuracy, including 96.67% on MATH, while costing 3.27× less than Full-Graph on MMLU-Redux and 18.41× less to train than AFlow on HumanEval++.

Motivation and core idea

Existing LLM-based multi-agent systems (MAS) typically optimize one dimension at a time—communication topology (GPTSwarm, G-Designer, AgentPrune), role assignment (DyLAN, MetaGPT), or LLM routing (MASRouter)—while treating each agent as a monolithic unit backed by a single model. HieraMAS departs from this by exploiting two distinct forms of collaboration simultaneously: intra-node collaboration, where each agent role is realized as a "supernode" containing multiple heterogeneous LLMs in a propose–synthesize structure (building on the Mixture-of-Agents finding that LLMs improve when conditioned on other models' outputs (Wang et al., 2024)), and inter-node collaboration, where supernodes exchange information through a learned communication graph. The framework jointly optimizes three coupled dimensions: the inter-supernode topology, role retention (pruning), and LLM assignment within each supernode (2602.20229).

Problem formulation

The system is cast as an MDP over a search space $\mathbb{S} = (\mathbb{M}, \mathbb{R}, \mathbb{G})$, where $\mathbb{M}$ is a pool of $N_m$ LLM backbones (including a special skip token), $\mathbb{R}$ is a set of predefined roles, and $\mathbb{G}$ is the space of DAG topologies. A supernode $$S_i = (r_i, \{m_{i,j}^{(w)}\}_{j=1}^{W}, m_i^{(a)})$$ consists of $W$ proposer LLMs and one synthesizer; the internal propose–synthesize structure is fixed and not optimized. The reward balances correctness against token cost, and the objective is to learn a policy maximizing expected reward over a labeled query dataset.

A key design choice is the skip token: selecting skip for a proposer position removes that proposer at zero cost, and selecting it for a synthesizer deactivates the entire supernode. This unified mechanism handles role pruning, proposer pruning, and LLM selection within a single action space, without a separate pruning module.

Two-stage optimization for credit assignment

Joint optimization raises two credit assignment problems that the paper identifies as the central technical obstacles:

1. Per-node masking: with final-reward-only training, a supernode that introduces errors can still receive positive gradient updates when other agents compensate for the failure. HieraMAS addresses this with multi-level rewards, $$R_i^{\mathrm{eff}} = \alpha R_i^{\mathrm{node}} + (1-\alpha) R^{\mathrm{final}}$$, where $R_i^{\mathrm{node}}$ evaluates the supernode's synthesizer output against the ground truth. A proposition in the appendix proves that when a failing node is compensated by others, multi-level rewards produce correctly signed policy gradients provided $$\alpha > R^{\mathrm{final}}/(R^{\mathrm{final}} - R_j^{\mathrm{node}})$$; a corollary shows $\mathbb{M}$0 suffices under normalized rewards, and this is the value used in practice. Roles whose outputs are not directly checkable (planners, critics) fall back to $\mathbb{M}$1.
2. Per-edge entanglement: after Stage 1 converges, most configurations succeed regardless of topology, so per-edge policy gradients reinforce non-beneficial edges with probability roughly $\mathbb{M}$2 (with $\mathbb{M}$3 the fraction of optimal edges, $\mathbb{M}$4 the edge sampling probability, and $\mathbb{M}$5 the success probability). The paper proves this error is irreducible—it does not vanish with more samples. HieraMAS therefore abandons per-edge optimization entirely: Stage 2 treats topology selection as holistic graph classification. A pool of $\mathbb{M}$6 random DAGs is pre-generated with edge densities in $\mathbb{M}$7; for each training query, $\mathbb{M}$8 graphs are executed with the frozen Stage-1 selector, top-reward graphs are labeled positive, and a GCN classifier with mean pooling is trained with binary cross-entropy. A generalization theorem shows that under a reward-margin assumption $\mathbb{M}$9, $N_m$0 samples per candidate suffice to identify the optimal topology with high probability, converting an ill-posed credit assignment problem into a well-posed estimation problem with vanishing $N_m$1 error.

Stage 1 itself trains the LLM selector $N_m$2 (a sentence-encoder + MLP scorer over query-role and LLM-profile embeddings, all-MiniLM-L6-v2, frozen) via REINFORCE with entropy regularization, using random graphs from the pool to expose the selector to diverse topologies. Both node-level and final rewards use a cost-sensitive form: correct outputs earn $N_m$3, failures incur $N_m$4, penalizing expensive failures most severely.

Empirical results

Experiments use HumanEval++ (Pass@1), MATH, and MMLU-Redux, with a nine-model LLM pool (Qwen3-8B, Qwen3-Next-80B-A3B, DeepSeek-R1-Distill-14B, Llama-3.1-8B, DeepSeek-V3.2, Gemma-3-27B, GPT-5-Mini, GPT-5-Nano, GPT-4o-Mini) under two settings (GPT-5-Mini as executor; Qwen3-80B as executor with GPT models excluded from selection). Baselines span single-agent (Base, CoT), fixed MAS (Self-Consistency, LLM-Debate, Full-Graph, Random-Graph), and learning-based MAS (AFlow, GDesigner, MASRouter).

Method	Avg. accuracy (%)
Base	83.14
CoT	89.81
Full-Graph	92.78
AFlow	92.69
MASRouter	90.89
GDesigner	90.68
HieraMAS	94.61

HieraMAS achieves the best average accuracy (94.61%) and the best results on HumanEval++ and MATH, with a notable 96.67% on MATH under GPT-5-Mini. Cost results are a central claim: on MMLU-Redux, HieraMAS costs $N_m$54.23 for Full-Graph (3.27× cheaper), and its training cost on HumanEval++ is 18.41× cheaper than AFlow, whose <a href="https://www.emergentmind.com/topics/constrained-monte-carlo-tree-search-mcts" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">MCTS</a> optimization with Claude 3.5-Sonnet costs $N_m$63.92 vs $1.29)—a strong claim that heterogeneous mixture assignment beats uniformly deploying the strongest backbone.

Analysis of learned structures shows topologies that are sparse (density 0.23–0.32), irregular, and query-dependent (pairwise Jaccard similarity of the top-5 graphs ranges 0.11–0.44), with Psychologist/Doctor roles consistently as sink nodes and Critic as source. LLM preferences form task-specific strong–weak combinations: Llama-3.1-8B and DeepSeek variants dominate on HumanEval++, Qwen3 models on MATH, and GPT-5-Nano/Mini on MMLU-Redux. Notably, the skip token is rarely selected, indicating the system prefers cost reduction through sparser topologies rather than role pruning. On out-of-domain generalization to 5 held-out MMLU subjects, HieraMAS reaches 68–72%, outperforming MASRouter (56% under GPT-5-Mini, attributed to that method's preference for weaker Qwen models), though it trails SC+CoT and AFlow under the GPT-5-Mini setting—evidence that heavier learned components can hurt generalization.

Limitations and open questions

Several limitations are conceded or evident. First, the graph classifier operates over a pre-generated candidate pool; the generalization guarantee assumes the optimal (or a near-optimal) topology is present in the pool (assumption B1), and performance depends on the reward-margin assumption (B2) that the paper itself notes would make topology optimization meaningless if violated. Second, the theory's edge-error bound relies on the assumption that strong LLMs compensate for suboptimal topologies but not vice versa, which justifies $N_m$7 but is an empirical posit rather than a proven property. Third, generalization to unseen domains is weaker than for methods that simply execute the strongest model, and the paper's explanation (favorable GPT-model preferences, plausible transfer of learned role patterns) remains a hypothesis. Fourth, supernode size is sensitive: increasing proposers from 2 to 8 degrades MMLU-Redux accuracy from 95.20% to 88.80%, partly due to role pruning during training and the lost-in-the-middle effect in long synthesizer contexts—leaving open how to scale intra-node mixtures without aggregation degradation. Finally, the internal propose–synthesize wiring is fixed and not co-optimized, and evaluation is limited to single-round, text-only reasoning, coding, and QA tasks; whether the two-stage decomposition remains advantageous for multi-round or tool-using MAS is unexamined.

Conclusion

HieraMAS unifies intra-node LLM mixtures and inter-node topology optimization in a single MAS framework, and its principal technical contribution is the treatment of the resulting credit assignment difficulties: multi-level rewards with a theoretically grounded mixing coefficient for per-node attribution, and a reformulation of topology selection as graph classification over a candidate pool, supported by an impossibility-style result for per-edge policy gradients. The empirical record—94.61% average accuracy with substantially lower inference and training costs than comparable learned MAS methods—supports the claim that heterogeneous, jointly optimized configurations outperform both fixed structures and single-dimension optimization, while the generalization and scalability caveats delineate the boundaries of that claim.