Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

Abstract: Agentic systems are becoming more capable: agents define strategies, take actions, and interact with different environments. This autonomy poses serious challenges for overseeing and assessing agent behavior. Most current tools are limited, focusing on observability with basic evaluation capabilities or imposing static, hand-crafted error taxonomies that cannot adapt to new domains. To address this gap, we present Agentic CLEAR, an automatic, dynamic, and easy-to-use evaluation framework. It produces textual insights into the agent behavior on three levels of granularity: system, trace, and node. Agentic CLEAR operates above the observability layer, enabling seamless integration and featuring an intuitive UI that makes agent evaluation highly accessible. In our experiments on four benchmarks, seven agentic settings, and tens of thousands of LLM calls, we show that Agentic CLEAR produces high-quality, data-driven, insightful feedback. Our analysis shows strong alignment with human-annotated errors and the ability to predict task success rate.

• The paper introduces Agentic CLEAR, an automated evaluation framework that assesses LLM agents via multi-level (node, trace, and system) analysis.
• Methodology details include a three-tier evaluation using LLM Judges like GPT-5 and OSS-120B to generate structured scores and natural language rationales.
• Experimental validation across multiple benchmarks demonstrates robust error detection and high predictive performance for trajectory success.

Agentic CLEAR: A Hierarchical, Automated Evaluation Framework for LLM Agents

Motivation and Background

The proliferation of agentic systems leveraging LLMs has enabled considerable progress in autonomous execution of multi-step complex tasks across diverse domains, including software engineering, scientific research, and web environments. However, these agentic systems are highly brittle, exhibiting recurrent and subtle failure modes such as looped behaviors, misaligned sub-agent execution, and complex error propagation. Existing observability tools are predominantly limited to basic metric aggregation or single-prompt, trace-level evaluations, imposing static taxonomies that do not adapt to new domains or agent configurations. Addressing this gap, "Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents" (2605.22608) introduces Agentic CLEAR, an automated, multi-granular framework for agent evaluation, operating above the observability layer and producing actionable, domain-adaptive textual feedback at system, trace, and node levels.

Methodology: Multi-Level Automated Evaluation

Agentic CLEAR implements a hierarchical, trace-driven evaluation pipeline, ingesting execution traces and extracting structured diagnostics across three levels of granularity: system-wide, node-specific, and trace-level. The approach is formalized by evaluating trace data from an agentic system, invoking an LLM Judge on each trace in three modes:

• Step-wise Evaluation (Node-Level): Individual input-output pairs are critiqued for correctness, completeness, and clarity, generating scores and natural language rationales for each agent node.
• Trace-wise Evaluation (System/Trace-Level): The full sequence of agent actions in a trace is holistically evaluated.
• Rubric Generation and Evaluation: Task-specific rubrics are dynamically constructed for each task description, and the judge evaluates rubric fulfillment within the trace.

Subsequently, the CLEAR aggregator clusters and summarizes critique outputs to identify recurring patterns and systemic issues at both node and system levels. The process is illustrated in Figure 1.

Figure 1: Agentic CLEAR processes execution traces, applies multi-level LLM Judge evaluation, aggregates insights via CLEAR, and surfaces system/node patterns for in-depth diagnosis.

Agentic CLEAR is accessible as a Python package (pip-installable), featuring OpenTelemetry-LangFuse compatibility and an interactive UI facilitating cross-level navigation and filtering—supporting seamless integration into existing agent development workflows.

Experimental Analysis and Empirical Validation

The authors empirically validate Agentic CLEAR across four benchmarks (e.g., SWE-Bench Verified Mini, GAIA, AppWorld, TAU-Bench), seven agentic settings, and tens of thousands of LLM calls, employing leading judge models (GPT-5 and OSS-120B). The framework consistently surfaces universal and domain-specific failure modes:

• Universal Patterns: Persistent errors include redundant tool usage, insufficient error recovery, incomplete workflows, and schema/output noncompliance.
• Domain and Node Differentiation: For example, GAIA benchmarks are dominated by insufficient sourcing and verification failures, while AppWorld exhibits workflow-specific breakdowns; node-level analysis exposes distinct planning versus execution deficiencies.

Agentic CLEAR demonstrates strong alignment with human-annotated error taxonomies (TRAIL), covering up to 12 of 12 relevant error categories, and providing more fine-grained, actionable diagnostics. Predictive evaluation analysis reveals robust AUC scores for trajectory success prediction: AppWorld results exceed 0.75 (and up to 0.823 for node-level scores with GPT-5), with trace-level scores outperforming rubric and step-wise metrics. Notably, judge model selection impacts the granularity and quality of surfaced issues; GPT-5 outputs longer, domain-specific issues, while OSS-120B produces more generic operational categories.

Implications and Future Directions

Agentic CLEAR's taxonomy-free, domain-adaptive approach to agent evaluation addresses the rigidity of prior frameworks by dynamically surfacing actionable issues without manual engineering, supporting rapid iteration and systematic diagnostic feedback in agentic workflows. Its multi-granular evaluation fosters both broad systemic analysis and localized component debugging, enabling targeted refinement of agent architectures and LLM behaviors. The alignment with human-annotated error taxonomies and predictive capability for trajectory success positions Agentic CLEAR as an effective meta-evaluation layer above standardized benchmarking infrastructures.

Practically, Agentic CLEAR will accelerate agent reliability and developer productivity through comprehensive insight generation. Theoretically, its rubric-adaptive protocol suggests new directions for interpretable LLM judge evaluation and automatic taxonomy discovery. Future research should extend CLEAR to simultaneously analyze execution-level system behaviors and LLM reasoning/planning, further refine module-specific judge reliability, and enable systematic cross-agentic comparison studies.

Conclusion

Agentic CLEAR provides a scalable, automated evaluation framework for agentic LLM systems, unifying multi-level critique aggregation and adaptive rubric generation without static error taxonomies or human engineering. Its empirical validation demonstrates strong alignment with expert error taxonomies and robust predictive performance, with implications for both practical agent development and theoretical studies of meta-evaluation protocols. The framework sets the stage for more adaptive, interpretable, and systematized evaluation of agentic LLM workflows, supporting the development of more reliable AI agents and next-generation evaluation infrastructures.