Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems

Abstract: LLM-based automatic Multi-Agent Systems (MAS) generation has become a crucial frontier for tackling complex tasks. However, existing methods face a dilemma between model capability and experience retention. Inference-time MAS leverages frozen frontier LLMs but repeats identical searches without learning from past experience. Conversely, Training-time MAS internalizes experience via gradient updates but is constrained by the low capability ceiling of smaller models, and is hard to scale to large frontier LLMs. To bridge this gap, we propose Skill-MAS, a novel third path that decouples experience retention from parametric updates by conceptualizing the high-level orchestration capability as an evolvable Meta-Skill. Skill-MAS refines this architectural knowledge through a closed optimization loop: (1) Multi-Trajectory Rollout samples a behavioral distribution for each task under the current Meta-Skill; and (2) Selective Reflection adaptively selects priority tasks and applies hierarchical contrastive analysis to distill systemic experience into generalizable, strategy-level principles. Extensive experiments across four complex benchmarks and four distinct LLMs demonstrate that Skill-MAS not only achieves remarkable performance gains but also maintains a favorable cost-performance trade-off. Further analysis reveals that the evolved Meta-Skills are highly robust and exhibit strong transferability across unseen tasks and different LLMs.

• The paper proposes a novel Skill-MAS framework that evolves a Meta-Skill to orchestrate multi-agent systems with enhanced performance and cost-effectiveness.
• The paper details a closed-loop optimization protocol using multi-trajectory rollouts, selective reflection, and targeted skill updates to improve MAS coordination.
• The paper demonstrates robust transferability across LLMs and tasks, outperforming traditional methods while reducing computational overhead.

Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems

Motivation and Positioning

LLMs have catalyzed progress in Automatic Multi-Agent Systems (MAS) generation, but the dichotomy between inference-time and training-time paradigms presents a persistent bottleneck. Inference-time MAS leverages frozen frontier LLMs with strong reasoning ability but no experience retention, whereas training-time MAS incorporates cumulative learning but is restricted to weaker models due to compute and data constraints. The need for a scalable, cost-effective approach that both harnesses high-capability LLMs and achieves experience retention motivates the introduction of Skill-MAS—a “third path” that conceptualizes orchestration not as parametric, model-weight-based learning but as an evolvable Meta-Skill.

Figure 1: Overview of MAS paradigms; Skill-MAS bridges the gap between capability and retention and provides superior cost-performance trade-offs.

Methodological Framework

Skill-MAS formalizes MAS orchestration as a structured, evolvable Meta-Skill $\mathcal{S}$ with three modules: Task Decomposition, Agent Engineering, and Workflow Orchestration. This abstraction allows high-level architectural knowledge to be captured, refined, and retained independently of LLM parameter updates. Skill-MAS operates via a closed-loop optimization protocol:

• Multi-Trajectory Rollout: For each task, the Meta-agent executes $K$ independent rollouts under the current Meta-Skill, generating behavioral statistics (mean, standard deviation) that quantify both execution difficulty and policy uncertainty.
• Selective Reflection: Leveraging these statistics, Skill-MAS adaptively selects priority tasks via an uncertainty--difficulty criterion, then synthesizes evidence through hierarchical contrastive analysis—first within-task, then across tasks—to diagnose systemic failures and strengths.
• Skill Optimization: Reflection evidence informs targeted updates to the Meta-Skill, ensuring revisions are modular, grounded, and abstracted for generalizability. The closed-loop repeats for $R$ rounds, with the best-performing skill adopted for deployment.

  Figure 2: The evolutionary optimization loop of Skill-MAS, delineating skill-guided rollouts and reflection-driven skill refinement.

Empirical Evaluation

Skill-MAS is benchmarked across four challenging domains: DeepResearchBench, Humanity’s Last Exam-Math, BrowseComp-Plus, and VitaBench. The evaluation spans four frontier LLMs (Gemini-3.1-Flash, GPT-5.4-Nano, Qwen3.5-Plus, DeepSeek-V4-Flash), with rigorous comparisons against leading inference-time (EvoAgent, AOrchestra, AFlow) and training-time (MAS$^2$, MAS-Orchestra) MAS baselines.

Key findings:

• Performance: Skill-MAS-optimized achieves the highest average scores across all models and benchmarks, significantly outperforming both baseline categories. Notably, Skill-MAS-init is competitive even without iterative evolution, underscoring the impact of architectural knowledge abstraction.
• Cost-Performance Trade-off: Skill-MAS incurs moderate inference overhead while delivering superior performance, outperforming inference-time MAS in efficiency and training-time MAS in capability.

Skill Evolution and Transferability

Meta-Skills evolved by Skill-MAS encode robust architectural strategies, exhibiting strong transferability across tasks and LLMs. Cross-domain evaluation reveals:

• Maximum gains when source and test settings match.
• Substantial performance retention in cross-LLM and cross-task scenarios due to explicit abstraction from domain specifics during evolution.
• Diminishing transfer effects in simultaneous cross-LLM plus cross-task conditions, aligning with theoretical transfer learning constraints.

  Figure 3: Left: Heatmap of Meta-Skill transferability across tasks and LLMs; Right: Performance scaling with increasing rollout numbers.

Ablation Studies and Skill Evolution Trajectory

Analysis of rollout numbers validates monotonic improvement with increased sampling but with diminishing returns, suggesting a computational trade-off. Selective Reflection ablation demonstrates the criticality of priority-driven task selection; label-free settings degrade performance, highlighting avenues for unsupervised trajectory scoring.

Multi-task learning on aggregated datasets yields domain-dependent results, revealing potential for further optimization via principled multi-task skill evolution.

Skill evolution on BrowseComp-Plus (DeepSeek-V4-Flash) traces a transition from generic decomposition to constraint-aware search topologies, calibrated agent evaluation, and resilient orchestration with dynamic backtracking and evidence recovery.

Figure 4: Illustration of Meta-Skill evolution trajectory on BrowseComp-Plus, emphasizing architectural and epistemic advancements.

Structural Comparison

Structural analysis demonstrates that Skill-MAS-optimized generates MAS with explicit decomposition, parallel evidence retrieval, link verification, and robust integration—qualitatively and quantitatively outperforming baseline MAS workflows, which tend to rely on linear or loosely coordinated search routines lacking structural resilience.

Implications and Future Directions

Skill-MAS establishes a scalable framework for automatizing MAS generation without expensive parametric updates. The abstraction of orchestration into evolvable Meta-Skills decouples architectural learning from LLM weight optimization, enabling high-capability models to achieve progressive improvement and transfer. Practically, this unlocks new opportunities for adaptive agent deployment in cost-sensitive, rapidly shifting environments. Theoretically, Skill-MAS bridges gaps in LLM-based system design by formalizing experience distillation as reusable architectural artifacts rather than ephemeral search traces.

Challenges remain in unsupervised skill evolution and domain-agnostic transfer, motivating future work on label-free trajectory scoring, multi-task learning frameworks, and further formal exploration of skill evolution dynamics in the MAS context.

Conclusion

Skill-MAS presents an authoritative paradigm shift in automatic MAS generation, conceptualizing orchestration as a modular, evolvable Meta-Skill and operationalizing a closed-loop evolution protocol for experience distillation. It achieves superior performance, excellent transferability, and favorable cost-performance balance, substantiating its efficacy for both research and practical AI system engineering.