Improving Role Consistency in Multi-Agent Collaboration via Quantitative Role Clarity

Abstract: In LLM-driven multi-agent systems, disobey role specification (failure to adhere to the defined responsibilities and constraints of an assigned role, potentially leading to an agent behaving like another) is a major failure mode \cite{DBLP:journals/corr/abs-2503-13657}. To address this issue, in the present paper, we propose a quantitative role clarity to improve role consistency. Firstly, we construct a role assignment matrix $S(φ)=[s_{ij}(φ)]$, where $s_{ij}(φ)$ is the semantic similarity between the $i$-th agent's behavior trajectory and the $j$-th agent's role description. Then we define role clarity matrix $M(φ)$ as $\text{softmax}(S(φ))-I$, where $\text{softmax}(S(φ))$ is a row-wise softmax of $S(φ)$ and $I$ is the identity matrix. The Frobenius norm of $M(φ)$ quantifies the alignment between agents' role descriptions and their behaviors trajectory. Moreover, we employ the role clarity matrix as a regularizer during lightweight fine-tuning to improve role consistency, thereby improving end-to-end task performance. Experiments on the ChatDev multi-agent system show that our method substantially improves role consistency and task performance: with Qwen and Llama, the role overstepping rate decreases from $46.4\%$ to $8.4\%$ and from $43.4\%$ to $0.2\%$, respectively, and the role clarity score increases from $0.5328$ to $0.9097$ and from $0.5007$ to $0.8530$, respectively, the task success rate increases from $0.6769$ to $0.6909$ and from $0.6174$ to $0.6763$, respectively.

• The paper introduces a quantitative role clarity metric to measure and optimize role adherence in multi-agent LLM systems.
• It leverages a LoRA-based fine-tuning framework to integrate role clarity regularization with traditional training, significantly reducing role overstepping.
• Empirical evaluations on benchmarks like SWE-Dev and SRDD demonstrate dramatic improvements in role consistency and overall task quality.

Quantitative Role Clarity for Enhanced Role Consistency in Multi-Agent LLM Collaboration

Introduction and Motivation

LLM-driven multi-agent systems, such as ChatDev and MetaGPT, rely on explicit role assignments among agents to facilitate structured collaboration for complex tasks like automated software development. However, empirical evaluations reveal that such systems frequently suffer from role inconsistency, manifesting as agents overstepping their assigned responsibilities — a critical failure mode that undermines both reliability and end-to-end performance. The studied paper addresses this deficiency by formalizing quantitative role clarity as a structural objective, enabling both the measurement and optimization of role consistency in multi-agent LLM systems (2604.02770).

Formalization of Role Clarity

The proposed approach introduces a quantifiable and differentiable definition of role clarity. Each agent $A_i$ is represented by a triple $(L_i, R_i, B_i)$, covering the LLM instance, a natural-language role description, and the observed behavior trajectory. Both role descriptions and behavior trajectories are embedded into a semantic space via a shared encoder, followed by mean pooling to obtain sequence-level embeddings.

A role assignment matrix $S(\phi)$ is constructed using the cosine similarity between the behavior and role embeddings. The rows of $S(\phi)$ are then normalized using a temperature-controlled softmax, yielding a probabilistic assignment whereby the diagonal entries reflect role adherence, and off-diagonal entries expose overstepping into other roles' responsibilities. The role clarity matrix $M(\phi) = \mathrm{softmax}_\tau(S(\phi)) - I$ captures global deviation from ideal role alignment, with the Frobenius norm as the quantitative metric.

$\|M(\phi)\|_F$ encodes both cross-role overstepping (off-diagonals) and within-role inconsistency (diagonals). A role clarity score $C(M(\phi)) = 1/(1 + \|M(\phi)\|_F)$ provides a normalized, interpretable index of system-wide role consistency.

Figure 1: Overview of the LoRA-tuning framework with role clarity regularization.

LoRA-Based Fine-Tuning With Role Clarity Regularization

To operationalize role clarity, the authors introduce a regularization term into the training objective: the average cross-entropy between each agent's behavior embedding and its correct role embedding under the softmax-assigned one-hot roles. This regularizer is combined with the traditional maximum likelihood loss, enabling a joint optimization framework for both trajectory generation and role adherence.

The key implementation leverages LoRA (Low-Rank Adaptation) for computationally efficient fine-tuning. Only the low-rank update matrices for selected self-attention layers are trainable, with the pretrained backbone kept frozen. Role embeddings are fixed, while behavior embeddings — and thus the role clarity matrix — are differentiable with respect to the adapted parameters.

This design yields a lightweight yet effective gradient signal for minimizing role overstepping and stabilizing collaborative interactions, compatible with both training and inference (via merging LoRA weights).

Empirical Evaluation and Main Results

Experiments are conducted on ChatDev using SWE-Dev (software development simulation) and SRDD (software requirement description) benchmarks, employing state-of-the-art open LLMs (Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct) as agent backbones. Fine-tuning targets the most error-prone agent roles (CEO, CPO).

Role consistency is rigorously tested using the role overstepping rate (proportion of sessions with at least one deviation in ten rounds) and the role clarity score, alongside end-to-end task success metrics (completeness, executability, consistency, and aggregate quality).

Key quantitative findings include:

• With Qwen2.5-7B, the role overstepping rate is reduced from 46.4% to 8.4%, and with Llama-3.1-8B from 43.4% to 0.2% under the full dual-agent fine-tuning regime.
• The role clarity score increases from 0.5328 to 0.9097 (Qwen) and from 0.5007 to 0.8530 (Llama), marking substantial improvements in consistent role adherence.
• End-to-end task completion quality improves: in Qwen, quality increases from 0.6769 to 0.6909; in Llama, from 0.6174 to 0.6763 on SRDD.
• Ablation studies reveal that the majority of improvement in role clarity is directly attributable to the explicit regularization term, as fine-tuning without it does not yield comparable gains.

  Figure 2: Role overstepping rates for DeepSeek Chat, Qwen2.5 7B, and Llama3.1 8B on SWE-Dev under ChatDev.

Theoretical and Practical Implications

The formulation of a continuous, differentiable role clarity metric addresses several theoretical gaps in multi-agent LLM systems. Unlike prompt-based or purely heuristic guardrails, this method allows for:

• Direct, gradient-based optimization of collaborative structure.
• Reduced reliance on brittle prompt engineering or RL-based reward shaping.
• Objective, dataset-independent evaluation and benchmarking of role adherence.

Practically, improved role consistency translates to more dependable multi-agent pipelines in software engineering and other task-oriented applications, with measurable decreases in cross-role interference and semantic drift.

Future Directions

Quantitative role clarity as a loss or evaluation metric opens several promising research trajectories:

• Extending the framework to hierarchical, larger-scale multi-agent ecosystems, including tool-augmented and planning-focused settings.
• Developing automated, role-aware benchmarking infrastructure for LLM-based agentic evaluation.
• Investigating compositional, role-adaptive architectures that dynamically adjust roles based on task context or emergent interaction patterns.
• Combining role-clarity optimization with communication protocol and negotiation learning for maximal robustness.

Conclusion

This work formalizes and demonstrates the utility of quantitative role clarity as an optimizable, differentiable objective for LLM-driven multi-agent systems. The introduced methodology — combining embedding-based role assignment, Frobenius-norm deviation measurement, and LoRA-adapted regularization — delivers strong, reproducible gains in both role consistency and downstream task performance, establishing a new baseline for structured multi-agent collaboration and pointing towards further systematization of role management in agentic AI (2604.02770).