BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning

Abstract: Enterprise AI systems that translate natural language into SQL queries and orchestrate multi-step agentic reasoning pipelines require evaluation approaches fundamentally different from academic benchmarks. Spider and BIRD established execution-accuracy protocols; G-Eval and RAGAS advanced LLM-based assessment; and recent work such as Spider 2.0, BEAVER, and BIRD-Interact has begun to address enterprise and agentic dimensions. No single framework unifies text-to-SQL assessment with agentic behavior evaluation into a production-grade pipeline calibrated against human expert judgment. We present BADGER, developed at Merkle, a unified evaluation framework integrating text-to-SQL assessment with agentic behavior evaluation. BADGER offers three contributions. First, LLM-assisted SQL component extraction extending Spider methodology to handle CTE-heavy, dialect-specific SQL. Second, a hybrid execution accuracy metric (Hybrid-EX) resolving column-aliasing and numeric-tolerance brittleness by using an LLM to infer structural alignments before deterministic cell-level scoring. Validated on 150 human-annotated industry queries, Hybrid-EX achieves Cohen's kappa=0.717 95% CI: 0.600-0.822 and 87.3% balanced accuracy, outperforming all six competing frameworks (Delta-kappa: 0.322-0.502, all p<=0.001). Third, an enterprise agentic evaluation suite assembling RAGAS, G-Eval, and agent benchmark metrics into a unified pipeline; Excess Tool Usage is the sole novel element. BADGER runs entirely within the client's governed data environment, supports configurable LLM judge backends, and enables rapid prototyping of client-specific judges and metrics, serving as a continuous evaluation backbone rather than a one-time quality gate.

• The paper introduces BADGER, a unified evaluation framework for enterprise AI that integrates LLM-assisted SQL extraction with agentic behavior assessment.
• The paper presents a novel Hybrid-EX metric combining LLM inference with deterministic scoring, achieving 87.3% balanced accuracy across multiple query complexities.
• The paper demonstrates robust performance, addressing production-specific challenges like dialect variance and data governance while enabling continuous quality calibration.

BADGER: Unified Evaluation Framework for Enterprise Agentic AI Conversational Systems

Motivation and Context

BADGER addresses fundamental gaps in evaluation methodologies for enterprise AI systems that orchestrate text-to-SQL generation and agentic reasoning pipelines within governed production data environments. While academic benchmarks (e.g., Spider, BIRD) have set standards for execution accuracy and component-level exact matching, they fail to capture real-world reliability and the nuanced failure modes observed in enterprise deployment, including dialect-specific SQL, column aliasing, numeric tolerance, and multi-step agentic workflows. Prior frameworks (Spider 2.0, BEAVER, BIRD-Interact) begin to address some enterprise dimensions but lack a unified approach calibrated to human expert judgment and robust against production-specific errors.

Framework Design

BADGER is designed as a production-grade, unified evaluation pipeline that integrates text-to-SQL assessment with agentic behavior evaluation, supporting continuous quality monitoring. The framework operates on golden test sets of natural-language queries, gold SQL, execution results, agent tool-call sequences, and expected clarifications. Evaluation is split into three modes—TEXT2SQL_ONLY, AGENTIC_ONLY, and BOTH—with a comprehensive metrics inventory spanning syntactic validity, SQL component recall, hybrid execution accuracy, qualitative LLM judgment, tool-call fidelity, response faithfulness, G-Eval summary quality, and intent resolution.

LLM-Assisted SQL Component Extraction

Traditional deterministic SQL parsers are brittle when faced with the SQL generated by modern LLMs, which frequently employ deeply nested CTEs and use constructs (QUALIFY, LATERAL FLATTEN, PIVOT) absent from standard benchmarks. BADGER replaces the deterministic parser with an LLM-based structured extraction prompt, producing validated JSON outputs for seven component types (SELECT, WHERE, GROUP BY, ORDER BY, HAVING, TABLE, KEYWORDS). This improves dialect robustness and enables precise component-level metrics. Component recall ($R_k$), precision ($P_k$), and F1 are computed per type, enhancing diagnostic granularity.

SQL Complexity Classification

BADGER adopts Spider’s four-tier classifier but redesigns the signal set for enterprise SQL by adding CTE detection and structural flags (hasJoin, hasNested, hasSetOp). This enables complexity stratification reflecting genuine production workloads, with tiers ranging from easy single-table queries to extra hard multi-CTE, cross-aggregation compositions.

Hybrid Execution Accuracy: Two-Stage Metric

Standard execution accuracy metrics (EX) are ill-suited for enterprise settings, penalizing correct outputs due to column aliasing, numeric rounding, and result ordering differences. BADGER’s hybrid execution accuracy (Hybrid-EX) is its primary methodological innovation: a two-stage process where Stage 1 employs an LLM to infer structural alignment (column mappings, numeric tolerance, index columns), followed by deterministic cell-level scoring. This deterministic approach eradicates run-to-run variance, aligns closely with human judgment, and resolves the brittleness of conventional EX metrics.

Hybrid-EX achieves Cohen’s $\kappa = 0.717$ [95% CI: 0.600–0.822] against independent expert annotations across 150 industry queries, with a balanced accuracy of 87.3%, outperforming all six competing frameworks (Defog sql-eval, IBM EvalAssist, BIRD, RAGAS, Dr. Spider, Spider 2.0) on human alignment (pairwise $\Delta\kappa$ range: 0.322–0.502, $p \leq 0.001$).

Figure 1: Statistical comparison panel for BADGER vs. all evaluated frameworks (n=150) shows Hybrid-EX achieving $\kappa=0.717$ and substantial agreement.

Agentic Evaluation Suite

BADGER integrates established metrics from RAGAS (faithfulness), G-Eval (coherence, consistency, fluency, relevance), and prior tool-based agent benchmarks (Gorilla, AgentBench), assembling a robust agentic evaluation suite. The Excess Tool Usage metric is the sole novel agentic element, addressing redundant multi-tool invocation observed in production agents.

Tool-call evaluation quantifies recall and order; response faithfulness verifies NLG claims against SQL output; G-Eval metrics assess summary quality; intent resolution scores decomposition and clarification of multi-intent queries.

Empirical Validation and Results

BADGER’s empirical study spans stratified complexity tiers across 150 human-annotated queries, validated by Fleiss’ $\kappa = 0.746$ and Krippendorff’s $\alpha = 0.749$, attesting to strong inter-rater reliability. Edge cases (out-of-scope and malicious queries) are correctly refused in all instances, ensuring operational safety.

Bootstrap analysis confirms BADGER’s superiority across complexity tiers (Easy: 82.9%, Medium: 93.1%, Hard: 84.9%, Extra Hard: 91.4% balanced accuracy), and raw EX scores alone are shown to be misleading—Hybrid-EX reliably balances sensitivity and specificity, avoiding both over-permissive and over-strict biases found in baselines.

Figure 2: Bootstrap EX score confidence intervals by complexity tier validate BADGER’s consistent alignment across query hardness levels.

Production Deployment and Customization

BADGER is delivered as an internal evaluation tool running exclusively within client data environments—ensuring data residency and governance compliance. The framework includes a configuration layer supporting diverse LLM judge backends and rapid prototyping of domain-specific evaluation criteria. Custom judges (e.g., regulatory checks, factual guardrails, response format compliance) can be integrated without modifying the core pipeline, meeting bespoke enterprise requirements. Parallel execution enables scalable throughput, and each run is tracked via stable metadata for longitudinal quality analysis.

Theoretical and Practical Implications

BADGER demonstrates that integrating LLM-based semantic inference with deterministic metric calculation bridges the gap between academic and enterprise needs. Calibrated against human expert judgment, BADGER provides evidence that execution-accuracy evaluation must address structural alignment, numeric tolerance, and alias normalization to reflect real delivery quality. The agentic suite ensures that tool orchestration, response coherence, and factual verification are central to pipeline evaluation. The rapid prototyping interface offers a scalable model for continuous improvement as enterprise needs evolve.

Future extensions include multi-turn conversational evaluation and multimodal RAG assessment for chart and visualization fidelity, as well as DAG-based tool-order specification for complex workflows.

Conclusion

BADGER represents a rigorously validated, production-calibrated evaluation backbone for enterprise AI systems, combining LLM-assisted SQL structural analysis with deterministic execution accuracy metrics and agentic behavioral evaluation. Its hybrid metric substantially outperforms established frameworks in human alignment while offering practical guarantees of reproducibility and longitudinal comparability. By unifying text-to-SQL and agentic evaluation under governed deployment, BADGER establishes a robust standard for continuous enterprise AI quality monitoring and improvement.