AEGIS: Automated Error Generation and Identification for Multi-Agent Systems (2509.14295v1)

Abstract: As Multi-Agent Systems (MAS) become increasingly autonomous and complex, understanding their error modes is critical for ensuring their reliability and safety. However, research in this area has been severely hampered by the lack of large-scale, diverse datasets with precise, ground-truth error labels. To address this bottleneck, we introduce \textbf{AEGIS}, a novel framework for \textbf{A}utomated \textbf{E}rror \textbf{G}eneration and \textbf{I}dentification for Multi-Agent \textbf{S}ystems. By systematically injecting controllable and traceable errors into initially successful trajectories, we create a rich dataset of realistic failures. This is achieved using a context-aware, LLM-based adaptive manipulator that performs sophisticated attacks like prompt injection and response corruption to induce specific, predefined error modes. We demonstrate the value of our dataset by exploring three distinct learning paradigms for the error identification task: Supervised Fine-Tuning, Reinforcement Learning, and Contrastive Learning. Our comprehensive experiments show that models trained on AEGIS data achieve substantial improvements across all three learning paradigms. Notably, several of our fine-tuned models demonstrate performance competitive with or superior to proprietary systems an order of magnitude larger, validating our automated data generation framework as a crucial resource for developing more robust and interpretable multi-agent systems. Our project website is available at https://kfq20.github.io/AEGIS-Website.

• The paper presents a novel automated pipeline for generating diverse error scenarios in multi-agent systems to address data scarcity.
• It employs a three-stage methodology—baseline collection, LLM-based error injection, and validation—to produce scalable, ground-truth error labels.
• Experimental results show that models trained on AEGIS data outperform larger non-fine-tuned systems, enhancing overall MAS reliability.

AEGIS: Automated Error Generation and Identification for Multi-Agent Systems

Introduction and Objectives

The "AEGIS: Automated Error Generation and Identification for Multi-Agent Systems" framework highlights the progression of Multi-Agent Systems (MAS) and addresses a critical obstacle in their advancement: error identification. As MAS become more intricate, the errors originating from any single agent can have cascading impacts, complicating error diagnosis and root-cause analysis. The AEGIS framework tackles the problem of data scarcity in error identification within MAS by automated synthetic data generation.

Framework Overview

AEGIS introduces a novel pipeline for systematically generating diverse error scenarios in MAS. The framework constructs a dataset by injecting controlled, predetermined errors into successful execution trajectories of MAS, creating synthetic but realistic error data without the need for expensive manual annotations. It consists of three stages:

1. Baseline Collection: Deterministic, error-free trajectories are collected across multiple MAS instantiations and task domains.
2. Error Injection: An LLM-based adaptive manipulator applies sophisticated, context-aware interventions to simulate various error modes, generating multiple faulty versions of each trajectory.
3. Validation and Labeling: The manipulated trajectories are automatically validated, and precise error attributions are recorded, providing a programmatically scalable method to produce ground-truth labels.

Data Utilization and Learning Paradigms

AEGIS supports three learning paradigms to utilize the synthesized data:

• Supervised Fine-Tuning (SFT): Error trajectories are used to train models by forming direct mappings from interactions to error diagnoses.
• Reinforcement Learning (RL): Learning is guided through a hierarchical reward system, offering dense feedback for correct error identifications and penalizing inaccuracies.
• Contrastive Learning (CL): By generating natural positive/negative pairs, models learn robust representations sensitive to subtle error signals.

Experimental Validation

AEGIS achieves substantial improvements across all three paradigms. Experiments indicate that models trained on AEGIS data outmatch both open-source and proprietary systems that lack task-specific fine-tuning. Models fine-tuned on AEGIS data even outperform larger models not fine-tuned on this dataset, confirming the value of the generated error data for MAS reliability.

1. Supervised Fine-Tuning shows the highest performance gains, with models achieving state-of-the-art results in error identification tasks.
2. Reinforcement Learning benefits from a dense and structured feedback system, demonstrating improved learning dynamics and performance stability.
3. Contrastive Learning effectively utilizes the generation process for representation learning, enhancing model sensitivity to error characteristics.

Implications and Future Directions

AEGIS not only provides a scalable method for error generation and identification in MAS but also shifts the methodology towards leveraging programmatically generated data for improving AI reliability. This approach mitigates the limitations of traditional data curation and manual annotation efforts. Moving forward, AEGIS could be extended to simulate more complex error scenarios, including cascading failures across MAS networks, and integrated into self-repairing, adaptive agentic systems to further enhance robustness and interpretability in MAS.

Conclusion

AEGIS pioneers an efficient, automated approach to generating large-scale error datasets for MAS, facilitating the development of robust diagnostic models. Its framework echoes a methodological shift towards automated data generation in AI, setting a foundation for future research in creating reliable, debuggable MAS. By converting an annotation bottleneck into an engineering challenge, AEGIS offers a pathway to enhanced MAS diagnostics and self-repair capabilities.