Efficient Agents: Building Effective Agents While Reducing Cost (2508.02694v1)

Abstract: The remarkable capabilities of LLM-driven agents have enabled sophisticated systems to tackle complex, multi-step tasks, but their escalating costs threaten scalability and accessibility. This work presents the first systematic study of the efficiency-effectiveness trade-off in modern agent systems, addressing the critical need for cost-effective designs without sacrificing performance. We investigate three key questions: (1) How much complexity do agentic tasks inherently require? (2) When do additional modules yield diminishing returns? (3) How much efficiency can be gained through the design of efficient agent frameworks? Through an empirical analysis on the GAIA benchmark, we evaluate the impact of LLM backbone selection, agent framework designs, and test-time scaling strategies. Using the cost-of-pass metric, we quantify the efficiency-performance trade-off across these dimensions. Our findings inform the development of Efficient Agents , a novel agent framework that has an optimal complexity to task requirements. Efficient Agents retains 96.7% of the performance of OWL, one leading open-source agent framework, while reducing operational costs from $0.398 to $0.228, resulting in a 28.4% improvement in cost-of-pass. Our work provides actionable insights for designing efficient, high-performing agent systems, advancing the accessibility and sustainability of AI-driven solutions.

• The paper introduces the Efficient Agents framework that balances high performance with reduced cost using the cost-of-pass metric.
• It evaluates various backbone LLMs and shows that sparse MoE architectures offer superior efficiency for simpler tasks.
• The study explores test-time scaling, adaptive planning, tool usage, and memory design to enhance agent scalability and sustainability.

Efficient Agents: Building Effective Agents While Reducing Cost

Introduction

The digital evolution driven by LLMs has enabled impressive agent systems capable of addressing complex, multi-step tasks. However, the prohibitive costs associated with these models stifle scalability and widespread accessibility. The paper "Efficient Agents: Building Effective Agents While Reducing Cost" explores this efficiency-effectiveness trade-off, presenting empirical insights and a novel agent framework called Efficient Agents. This framework strives to maintain high performance while significantly reducing operational expenditures, thus contributing to more sustainable and accessible AI-driven solutions.

Efficiency-Effectiveness Trade-off

LLMs have primarily focused on enhancing capacity and complexity to solve intricate problems, often at significant economic overhead. The paper systematically investigates the inherent complexity required by agentic tasks, the diminishing returns from additional modules, and the efficiency gains achievable through optimized agent frameworks. The GAIA benchmark serves as the evaluation ground, examining LLM backbone selection, agent framework designs, and test-time scaling strategies using the cost-of-pass metric—a comprehensive measure of model efficiency related to its success rate and computational cost.

Figure 1: Evaluation of effectiveness and efficiency in agent system components. We adopt cost-of-pass as the metric to evaluate. We develop Efficient Agents that optimizes cost while maintaining accuracy.

Empirical Insights from Backbone Selection

Several backbones, including proprietary models like GPT-4.1 and Claude-3.7, were tested, revealing substantial efficiency differences despite comparable performance. Sparse models employing MoE architectures, such as Qwen3-30B-A3B, demonstrated superior efficiency due to selective parameter activation. This approach suits simpler tasks where computational efficiency outweighs raw performance needs. It's crucial to note that as task difficulty scales from level 1 to level 3, efficiency markedly deteriorates within reasoning models, presenting challenges in scalability and complexity management.

Figure 2: Performance of various backbone LLMs on the GAIA benchmark: Accuracy vs Cost.

Optimizing Test-Time Scaling Strategies

Test-time scaling introduces multiple inference runs to boost performance. The paper evaluates the Best-of-$N$ (BoN) strategy, observing that increased runs result in higher token consumption with only marginal performance gains. Such inefficiencies necessitate exploring more effective test-time scaling methods suitable for agent systems.

Planning Modules' Impact

Planning modules are vital for long-horizon task management. This paper indicates that moderate planning complexity enhances efficiency by regulating reasoning length, avoiding overthinking, and controlling computational cost inflation in unsolvable problems. It underscores the need for adaptive planning strategies within agent frameworks.

Tool Usage Efficiency

Incorporating tools like web browsers enhances agent capabilities but incurs a token overhead. The paper evaluates the effectiveness of various tool configurations—simple browser operations outperform advanced interactions, optimizing the search scope enhances efficiency, and broadening query expansions results in retrieving more relevant information, thus demonstrating the value of strategic tool use.

Memory Components

The agent system examined different memory configurations affirming Simple Memory's superiority over Summarized or Extra Memory strategies. Retaining observations and actions alone in the context window provides the most effective balance of cost-efficiency and performance, emphasizing simplicity in memory design.

Development of Efficient Agents Framework

The Efficient Agents framework emerged from selecting components highly effective for cost-performance trade-offs. It provides an improved efficiency metric on the GAIA benchmark, achieving 96.7% of OWL framework performance while reducing operational costs from \$0.398 to \$0.228, marking a 28.4% benefit in cost-of-pass.

Conclusion

This paper systematically addresses the efficiency-effectiveness dilemma in LLM-driven agent systems, providing actionable insights and developing an optimized framework, Efficient Agents. The success in balancing performance and economic efficiency paves the way for accessible, sustainable AI solutions. Future research into task-adaptive and resource-aware agent architectures is expected to build upon these findings, broadening AI's applicability in real-world deployments.