MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification (2412.04494v2)

Abstract: Extending the capabilities of LLMs with functions or tools for environment interaction has led to the emergence of the agent paradigm. In industry, training an LLM is not always feasible because of the scarcity of domain data, legal holds on proprietary customer data, rapidly changing business requirements, and the need to prototype new assistants. Agents provide an elegant solution to the above by relying on the zero-shot reasoning abilities of the underlying LLM and utilizing tools to explore and reason over customer data and respond to user requests. However, there are two concerns here: (I) acquiring large scale customer queries for agent testing is time-consuming, and (II) high reliance on the tool call sequence (or trajectory) followed by the agent to respond to user queries may lead to unexpected or incorrect behavior. To address this, we propose MAG-V, a multi-agent framework to first generate a dataset of questions that mimic customer queries; and second, reverse-engineer alternate questions from the responses for trajectory verification. Initial results indicate that our synthetic data can improve agent performance on actual customer queries. Furthermore, our trajectory verification methodology, inspired by distant supervision and using traditional ML models, outperforms a GPT-4o judge baseline by 11% accuracy and matches the performance of a GPT-4 judge on our constructed dataset. Overall, our approach is a step towards unifying diverse task agents into a cohesive framework for achieving an aligned objective.

• The paper presents MAG-V, a multi-agent framework designed to generate and verify synthetic data for LLM-driven agents without extensive reliance on LLM feedback.
• MAG-V employs a multi-agent system for realistic data generation and uses classical machine learning techniques like SVMs for deterministic verification of agent tool call trajectories.
• Experiments show MAG-V achieves an 11% accuracy improvement over a GPT-4 judge baseline on generated datasets, demonstrating the viability of deterministic methods for cost-effective and accurate verification.

Multi-Agent Framework for Synthetic Data Generation and Verification

The paper "MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification" presents a novel approach to expand the functionality and applicability of LLMs in generating and verifying synthetic data through the use of a multi-agent system. This research is particularly relevant in the context of creating intelligent customer service agents capable of responding to queries without substantial reliance on often scarce domain-specific data.

Key Contributions

The authors identify two primary issues in deploying LLM-driven agents: the time-intensive and data-hungry nature of creating and testing large-scale customer query datasets and the potential inaccuracies arising from improper tool calls by these agents. To combat these issues, the MAG-V framework undertakes the dual process of synthetic data generation and deterministic trajectory verification without the direct feedback from LLMs.

1. Synthetic Data Generation: MAG-V utilizes a multi-agent setup to generate a realistic dataset that mimics customer queries. The process involves agents generating questions and responses that align with varying customer data requirements, thus enabling stress testing of the assistant with diverse query types.
2. Deterministic Trajectory Verification: The framework employs classical machine learning techniques, specifically distant supervision and discriminative models like SVMs, to verify the sequence of tool calls (trajectories) used by agents. This verification is achieved without defaulting to LLMs as judges, addressing potential variability and inconsistency concerns associated with LLM evaluations.

Experimental Evaluation

The MAG-V framework demonstrates significant improvements in agent accuracy in synthetic data-driven environments. Notably, the approach shows an 11% accuracy improvement over a baseline GPT-4 judge model when applied to generated datasets. This indicates that traditional ML models, with crafted feature engineering, are robust alternatives to expensive LLM evaluations while maintaining comparable accuracy.

Implications and Future Directions

The implications of this research extend to the practical deployment of customer-facing LLM assistants. By reducing dependencies on large, potentially unavailable datasets, MAG-V offers a scalable solution for enterprises to deploy intelligent agents with reduced legal and data privacy concerns.

The theoretical implications highlight the usability of classical ML techniques in an LLM-dominated landscape, reinforcing the potential incorporation of deterministic methods for increased consistency and cost-effectiveness.

Potential future research directions include:

• Scaling Studies: Investigating how MAG-V scales with larger datasets and more complex query scenarios to refine trajectory verification methodologies.
• Enhanced Contextual Grounding: Improving the grounding of trajectory predictions to their associated questions through advanced natural language processing techniques.
• Label Smoothing: Introducing more granularity in trajectory correctness labels (e.g., partial correctness) to address complexities in question interpretation and improve classification fidelity.

Conclusion

The research outlined in "MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification" presents a viable alternative methodology for verifying agent trajectories, reducing reliance on state-of-the-art LLMs. MAG-V stands as a foundational step toward efficient, scalable, and replicable agent deployment, balancing cost-efficiency with accuracy. The proposed framework and its results underscore the value of interdisciplinary approaches, combining elements of classical machine learning with cutting-edge LLM technologies, potentially shaping future developments in AI-powered systems.