FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments

Abstract: LLMs are being increasingly deployed as the decision-making core of autonomous agents capable of effecting change in external environments. Yet, in conversational benchmarks, which simulate real-world customer-centric issue resolution scenarios, these agents frequently fail due to the cascading effects of incorrect decision-making. These challenges are particularly pronounced for open-source LLMs with smaller parameter sizes, limited context windows, and constrained inference budgets, which contribute to increased error accumulation in agentic settings. To tackle these challenges, we present the Failure-Aware Meta-Agentic (FAMA) framework. FAMA operates in two stages: first, it analyzes failure trajectories from baseline agents to identify the most prevalent errors; second, it employs an orchestration mechanism that activates a minimal subset of specialized agents tailored to address these failures by injecting a targeted context for the tool-use agent before the decision-making step. Experiments across open-source LLMs demonstrate performance gains up to 27% across evaluation modes over standard baselines. These results highlight that targeted curation of context through specialized agents to address common failures is a valuable design principle for building reliable, multi-turn tool-use LLM agents that simulate real-world conversational scenarios.

• The paper introduces FAMA, a two-stage meta-agentic framework that dynamically mitigates error cascades in open-source LLM agents.
• It employs targeted activation of specialized helper agents to address diverse failure categories while minimizing token overhead.
• Empirical evaluations show up to 27% accuracy improvements and highlight FAMA’s efficiency in resource-constrained environments.

Failure-Aware Orchestration for Open-Source LLM Agents: The FAMA Framework

Motivation and Background

LLMs have rapidly matured as agents capable of structured tool use in multi-turn, goal-oriented dialogue settings central to domains such as customer-service automation, scientific discovery, and enterprise backend manipulation. However, interactive tool-use benchmarks (e.g., $\tau$-bench, $\tau$-trait, ACEBench) continue to expose critical failure points in open-source LLM agentic systems, especially for models with limited parameters and context capacity. These failure cascades typically arise from heterogeneous errors accumulated across long-horizon trajectories, often exacerbated by static orchestration approaches that fail to adapt to model- and task-specific weaknesses.

Traditional mitigation involves parameter-centric approaches—fine-tuning, reinforcement learning, or distillation—whose high compute and supervision cost is especially excessive for resource-constrained deployments. Moreover, these methods optimize for generic correct behavior rather than for correcting dominant, model-specific error patterns, resulting in poor sample efficiency and limited transfer. There remains a significant gap in training-free, dynamic frameworks that enhance open-source LLM agents via modular orchestration explicitly conditioned on observed failure modes.

FAMA: Failure-Aware Meta-Agentic Framework

The Failure-Aware Meta-Agentic (FAMA) framework directly addresses this gap by introducing a two-stage meta-agentic approach to agentic orchestration. Distinct from traditional static setups that activate a fixed suite of helper agents, FAMA first performs empirical and agentic analysis of failure trajectories produced by a baseline agent. These failures are automatically attributed to a set of predefined, broad error categories—domain policy violations, incorrect extraction from tool outputs, contextual misinterpretation/hallucination, and incomplete fulfillment/early stopping—each equipped with a corresponding analysis agent.

The outputs of these analysis agents are aggregated and further evaluated by a higher-level orchestrator agent that performs holistic trajectory-level failure attribution. Conditioned on these outputs, a dedicated mitigation agent selects the minimal subset of specialized agents needed to address the dominant errors in each task instance.

This process constructs the prior conversational context dynamically and task-adaptively, injecting only task-relevant specialized guidance instead of brute-force context amalgamation. As a result, FAMA addresses the primary bottlenecks of open-source, small-parameter LLM agents: their limited context window and strong error heterogeneity.

Figure 1: Comparison of static and dynamic (e.g., FAMA) agentic orchestration.

Figure 2: Overview of the FAMA framework, depicting its staged failure analysis and selective agentic orchestration process.

Empirical Evaluation

Experimental results on $\tau$-bench (Airline/Retail), $\tau$-trait (Telehealth/Telecom), and ACEBench demonstrate statistically significant improvements for FAMA-augmented open-source LLM agents (Qwen3-4B, Qwen3-14B, Qwen3-32B, Qwen2.5-72B) over IRMA, ReAct, and FC agentic baselines. For example, on $\tau$-bench domains, FAMA yields up to 25% improvement in pass@1 over baselines; in ACEBench, end-to-end accuracy increases by up to 27% in certain model/domain settings.

The ablation studies confirm that naive context injection—incorporating all helper agents—frequently degrades performance due to context window overflows and unnecessary augmentation noise, while FAMA’s mitigation strategy avoids superfluous agent activation and overhead. Detailed agent selection statistics show that memory and domain constraint extractors are the most critical components, aligning with the observation that domain policy adherence and context carry-over are the leading sources of failure for small LLMs.

Figure 3: $\tau$-bench pass@1 performance of multiple open-source LLMs as user simulators.

Figure 4: Distribution of error categories in $\tau$-retail tasks as identified by the orchestrator agent.

Error Analysis and Agent Selection

Failure analysis across benchmarks reveals that open-source agents are most susceptible to context drift, domain constraint violations, and misattribution of user intent, especially as trajectory length increases. The orchestrator consistently identifies contextual misinterpretation and domain rule violations as responsible for the majority of failed trajectories.

The mitigation agent’s recommendations, when visualized, reveal stabilized patterns: memory modules and domain constraint extractors are universally activated, while other specialized agents are only included in response to less frequent errors. These targeted interventions sustain improvements without introducing excessive token overhead or latency.

Figure 5: Recommendations distribution from the mitigation agent across open-source agents for $\tau$-bench.

Efficiency and Resource Considerations

Efficiency analyses highlight a central advantage of FAMA: it achieves substantial accuracy gains despite incurring lower token overhead and latency than static multi-agent baselines (e.g., IRMA). Unlike reasoning-augmented (“thinking”) model variants, which severely exceed context limits and degrade task completion in multi-turn settings, FAMA’s context-aware agent selection maintains robust performance within finite token budgets.

Figure 6: Token overhead and latency comparison across orchestration frameworks with Qwen3-32B.

Generalizability and Implications

Results across all benchmarks and domains (including ACEBench’s diverse tool API coverage and $\tau$-trait’s persona-robustness tests) confirm that FAMA generalizes to diverse agentic error profiles and domain constraints, provided the agent pool covers plausible failure types. This modular approach enables extensible integration of new error-specific modules as agentic error taxonomies evolve, facilitating adaptation to future environments that incorporate open-ended tool APIs, multimodal grounding, or personal assistants.

From a practical perspective, FAMA enables cost-effective scaling of LLM-powered agentic systems using lighter-weight open-source models, essential for enterprise and privacy-sensitive deployments. Theoretically, its meta-agentic orchestration points toward a framework for curriculum learning and lifelong adaptation in agent collectives, prompting future research in agent generation, automated error category discovery, and synergistic integration with RL-fine-tuned backbones.

Conclusion

FAMA (Failure-Aware Meta-Agentic Framework) operationalizes dynamic failure-driven agentic orchestration for open-source LLMs, yielding consistent improvements in multi-turn tool-use tasks via minimal, context-efficient augmentation. Empirical evidence demonstrates that explicit trajectory-level failure attribution and model-adaptive agent selection not only enhance reliability and task success but also optimize for resource constraints endemic to open-source model deployment. Future directions include automated agent/module synthesis, broader error taxonomy coverage, and adaptation for multimodal and embodied agent settings.