Effect of LLM model choice on AgentSynth-generated task properties

Determine how the choice of large language model used for task generation within the AgentSynth pipeline affects the properties of the synthesized tasks, specifically their complexity, realism, and meaningfulness, and characterize the resulting task-generation variance across different LLM architectures to inform diversity and difficulty calibration.

Background

AgentSynth employs GPT-4.1-based agents for task generation due to their robustness and generalization. The pipeline synthesizes tasks by chaining simple subtasks and relies on LLMs to propose and execute these subtasks. The authors note that different LLMs may systematically produce tasks with different characteristics, implicating model choice as a potential factor influencing task properties.

Understanding how the chosen LLM affects generated task properties such as complexity, realism, and meaningfulness is important for controlling benchmark difficulty, expanding task diversity, and ensuring calibrated evaluations. Establishing this relationship would guide the selection and combination of LLMs for more balanced and representative synthetic datasets.