GEMMAS: Evaluation Metrics for Multi-Agent Systems

• GEMMAS is a graph-based evaluation framework for multi-agent systems, modeling internal collaboration as directed acyclic graphs.
• It introduces key metrics, Information Diversity Score (IDS) and Unnecessary Path Ratio (UPR), to assess reasoning diversity and redundancy.
• Experiments on math benchmarks reveal that similar accuracy can hide significant differences in process-level collaboration and computational efficiency.

GEMMAS, short for Graph-based Evaluation Metrics for Multi Agent Systems, is an evaluation framework for multi-agent systems built on LLMs. It is designed for collaborative reasoning settings in which several agents exchange messages to solve tasks such as math word problems, symbolic reasoning, and code-based problem solving. Rather than evaluating only whether the final answer is correct, GEMMAS models the internal collaboration process as a directed acyclic graph and introduces process-level diagnostics intended to expose semantic redundancy, unnecessary reasoning paths, and associated computational inefficiency. Its two central metrics are Information Diversity Score (IDS) and Unnecessary Path Ratio (UPR), which are meant to complement, rather than replace, outcome metrics such as accuracy (Lee et al., 17 Jul 2025).

1. Conceptual scope and motivation

GEMMAS was introduced in response to a specific limitation in the evaluation of LLM-based multi-agent systems: conventional assessment typically reports only final-task correctness, leaving internal coordination largely unexamined. In that setting, systems with similar accuracy can still differ substantially in whether agents merely restate each other, whether some branches of reasoning are redundant, and how much token budget is spent on communication that does not materially improve task performance (Lee et al., 17 Jul 2025).

The framework therefore treats collaboration quality as an object of evaluation in its own right. Its central claim is that outcome-only metrics are insufficient for understanding multi-agent performance, because they cannot distinguish between a system that reaches a correct answer through focused, low-redundancy interaction and one that arrives at a similar answer through repeated or unhelpful reasoning paths. GEMMAS is explicitly positioned as a process-level diagnostic framework for designing more interpretable and resource-efficient collaborative AI systems.

A common misconception addressed by the framework is that higher final accuracy necessarily implies better collaboration. The reported experiments show that this implication does not hold in general. Another misconception is that diversity alone is always desirable. GEMMAS does not treat high IDS as intrinsically sufficient; the intended interpretation is joint, using IDS together with UPR and accuracy.

2. Graph representation of multi-agent interaction

GEMMAS models a multi-agent system as a directed acyclic graph,

$G = (V, E, F),$

where $V = \{v_1, \dots, v_N\}$ denotes nodes corresponding to agent instances, $E \subseteq V \times V$ denotes directed communication edges, and $F = \{f_1, \dots, f_N\}$ denotes agent-specific reasoning functions or roles. Each node is associated with an identity or role, a prompt-response pair, and an implicit time step. The DAG assumption encodes forward-only information flow and excludes cyclic interaction within a single execution trace (Lee et al., 17 Jul 2025).

A distinctive feature of GEMMAS is its separation of structural and temporal relations. It constructs two adjacency matrices:

• Spatial adjacency $S \in \{0,1\}^{N \times N}$: $S_{ij} = 1$ if the communication topology allows the output of $v_i$ to be sent to $v_j$.
• Temporal adjacency $T \in \{0,1\}^{N \times N}$: $T_{ij} = 1$ if the output of $V = \{v_1, \dots, v_N\}$0 is causally or temporally prior to and influences $V = \{v_1, \dots, v_N\}$1.

This distinction separates who can communicate from who actually influenced whom during a run. From raw execution logs, GEMMAS constructs the node set $V = \{v_1, \dots, v_N\}$2, the outputs $V = \{v_1, \dots, v_N\}$3, the topology-derived matrix $V = \{v_1, \dots, v_N\}$4, and the log-derived matrix $V = \{v_1, \dots, v_N\}$5. These structures define the substrate on which both IDS and UPR are computed.

3. Information Diversity Score and Unnecessary Path Ratio

IDS is intended to quantify whether connected or causally related agents contribute distinct information or instead produce semantically overlapping messages. To do so, GEMMAS combines syntactic similarity from TF-IDF features with semantic similarity from BERT embeddings. If $V = \{v_1, \dots, v_N\}$6 and $V = \{v_1, \dots, v_N\}$7, then pairwise cosine similarities are computed for both representations and combined as

$V = \{v_1, \dots, v_N\}$8

In the reported experiments, $V = \{v_1, \dots, v_N\}$9. Structural weighting is then defined by

$E \subseteq V \times V$0

and the Information Diversity Score is

$E \subseteq V \times V$1

IDS is therefore a weighted mean dissimilarity over structurally relevant pairs. Values near 0 indicate low diversity and heavy repetition; values near 1 indicate high diversity among related agent messages (Lee et al., 17 Jul 2025).

UPR addresses a different question: how much of the communication graph consists of reasoning paths that do not meaningfully contribute to correct answers. Let $E \subseteq V \times V$2 be the set of all subpaths in the spatial graph and $E \subseteq V \times V$3 the subset deemed necessary. GEMMAS defines

$E \subseteq V \times V$4

A path is classified as necessary by computing a contribution score over its nodes. For a path $E \subseteq V \times V$5, if $E \subseteq V \times V$6 is the number of nodes whose extracted answer matches the correct answer $E \subseteq V \times V$7, and $E \subseteq V \times V$8 is the total number of nodes on the path, then the score is $E \subseteq V \times V$9. A path is necessary if this score is at least $F = \{f_1, \dots, f_N\}$0. Low UPR indicates that most paths contribute substantially; high UPR indicates large fractions of redundant or unhelpful reasoning.

The framework’s interpretive scheme is explicitly joint. High IDS with low UPR is treated as a desirable pattern, because it combines varied reasoning with low redundancy. Low IDS with high UPR indicates poor collaboration. The authors also note that extremely high IDS, if unaccompanied by coordination, could reflect dispersion rather than effective division of labor.

4. Experimental setting and evaluated system families

The empirical study evaluates GEMMAS on five math-focused benchmarks: GSM8K, AQuA, MultiArith, SVAMP, and the mathematics subsets of MMLU. The benchmark sizes reported are 1,000 problems for GSM8K, 254 for AQuA, 180 for MultiArith, 300 for SVAMP, and 748 questions for MMLU. Although all experiments are concentrated in mathematical reasoning, the framework is presented as generalizable in principle to other reasoning domains (Lee et al., 17 Jul 2025).

The multi-agent architecture is adopted from AgentPrune and consists of four specialized agents plus a final aggregator. The four agents are AnalyzeAgent, which decomposes problems and produces structured plans; CodeWritingAgent, which writes code or computational procedures; MathSolverAgent, which performs formal or symbolic solving; and AdversarialAgent, which intentionally proposes plausible but incorrect solutions. A FinalRefer agent collects all outputs and synthesizes the final answer. The collaboration protocol sends the input problem to all four agents concurrently, runs two rounds of communication according to the chosen DAG topology, and then invokes the final aggregator.

Four multi-agent configurations are compared. Vanilla-AD is a fully connected baseline with no structural optimization. AgentDropout dynamically removes underperforming agents. AgentPrune prunes low-impact communication links. G-Designer learns optimized DAG topologies over multiple iterations to reduce communication overhead while maintaining or improving reasoning. The LLMs used are Llama 3.1–8B-Instruct and Qwen 2.5–7B-Instruct. The reported implementation details include learning rate $F = \{f_1, \dots, f_N\}$1, dropout $F = \{f_1, \dots, f_N\}$2, 40 training examples, 10 sampling iterations, two communication rounds per task, and temperature $F = \{f_1, \dots, f_N\}$3.

5. Empirical behavior and diagnostic value

The central empirical result is that systems with similar accuracy can differ sharply in IDS and UPR. On GSM8K, the abstract highlights a case where systems with only a 2.1% difference in accuracy differ by 12.8% in IDS and 80% in UPR, which is presented as evidence that internal collaboration quality can vary substantially even when final outcomes appear close (Lee et al., 17 Jul 2025).

A concrete GSM8K example with Qwen 2.5–7B-Instruct compares Vanilla-AD and G-Designer. Vanilla-AD obtains accuracy $F = \{f_1, \dots, f_N\}$4, IDS $F = \{f_1, \dots, f_N\}$5, and UPR $F = \{f_1, \dots, f_N\}$6, with prompt and completion token counts $F = \{f_1, \dots, f_N\}$7 and $F = \{f_1, \dots, f_N\}$8. G-Designer obtains accuracy $F = \{f_1, \dots, f_N\}$9, IDS $S \in \{0,1\}^{N \times N}$0, and UPR $S \in \{0,1\}^{N \times N}$1, with $S \in \{0,1\}^{N \times N}$2 prompt tokens and $S \in \{0,1\}^{N \times N}$3 completion tokens. The accuracy difference is small relative to the drop in UPR, indicating substantially less redundant internal reasoning.

MultiArith provides an even clearer illustration. With Qwen 2.5–7B-Instruct, Vanilla-AD and AgentPrune both achieve accuracy $S \in \{0,1\}^{N \times N}$4, but their process metrics differ materially: Vanilla-AD has IDS $S \in \{0,1\}^{N \times N}$5 and UPR $S \in \{0,1\}^{N \times N}$6, while AgentPrune has IDS $S \in \{0,1\}^{N \times N}$7 and UPR $S \in \{0,1\}^{N \times N}$8. Outcome-only evaluation would treat them as equivalent, whereas GEMMAS distinguishes a higher-diversity, lower-redundancy collaboration pattern. In the same benchmark, G-Designer reaches accuracy $S \in \{0,1\}^{N \times N}$9, IDS $S_{ij} = 1$0, and UPR $S_{ij} = 1$1.

SVAMP illustrates the opposite pattern. With Llama 3.1–8B-Instruct, AgentPrune records accuracy $S_{ij} = 1$2, IDS $S_{ij} = 1$3, and UPR $S_{ij} = 1$4, while Vanilla-AD records $S_{ij} = 1$5, $S_{ij} = 1$6, and $S_{ij} = 1$7, respectively. This indicates that pruning communication without preserving semantic diversity can be detrimental. MMLU with Qwen 2.5–7B-Instruct shows a more ambiguous trade-off: Vanilla-AD yields accuracy $S_{ij} = 1$8, IDS $S_{ij} = 1$9, and UPR $v_i$0, whereas G-Designer yields $v_i$1, $v_i$2, and $v_i$3. This suggests that higher diversity and improved accuracy may in some settings coexist with more unnecessary paths, rather than with uniformly greater efficiency.

Across these cases, IDS and UPR are described as not strongly correlated with accuracy. Their role is therefore orthogonal: IDS captures diversity of reasoning, while UPR captures redundancy and path-level inefficiency.

6. Computational properties, limitations, and prospective extensions

From an implementation standpoint, GEMMAS is lightweight enough for the evaluated systems because those systems are small DAGs with limited depth and only two communication rounds. Constructing $v_i$4 and $v_i$5 requires $v_i$6 storage, and computing pairwise similarities for IDS requires $v_i$7 comparisons, or $v_i$8 when embedding dimension is made explicit. The main computational caveat lies in UPR: enumerating subpaths in a DAG can be exponential in the worst case, although the paper reports that this remains tractable for the small per-problem graphs studied here (Lee et al., 17 Jul 2025).

The framework also has explicit limitations. First, all reported experiments are on mathematical reasoning tasks; it has not yet been validated on dialogue, code generation, planning, or real-world tool-using agents. Second, the study uses only small open-source instruction-tuned models, specifically Llama 3.1–8B-Instruct and Qwen 2.5–7B-Instruct. Third, IDS depends on TF-IDF and BERT-based similarity estimates, so representation error can propagate into the metric. Fourth, GEMMAS assumes that interaction can be represented as a DAG, which excludes cyclic negotiations unless such cycles are condensed or temporally unfolded. Fifth, UPR adopts a simple threshold-based notion of necessity, defined by whether at least half the nodes on a path produce the correct answer; this does not capture subtler forms of indirect contribution. Sixth, the framework is static: it evaluates completed traces rather than controlling communication online.

The proposed future directions follow directly from those constraints. These include extension to non-math domains, integration with system-level metrics such as latency, memory footprint, and cost per correct answer, evaluation of dynamic DAGs and adaptive topologies, use of IDS and UPR as optimization objectives in topology search, incorporation of human-in-the-loop judgments of interpretability and trustworthiness, and development of additional process-level metrics for consensus dynamics, conflict resolution, or exploration versus exploitation. A plausible implication is that GEMMAS is best understood not as a replacement for accuracy-based benchmarking, but as an additional analytical layer for diagnosing how a multi-agent system reasons, coordinates, and spends computational budget.