ComplexEval: Evaluating Complexity

Updated 10 July 2026

ComplexEval is a multifaceted framework that defines evaluation under complexity, assessing mini agents, function calling, and LLM judge biases across varied contexts.
It employs techniques such as surrogate modeling with causal guarantees for mini agents, format checking and Hungarian matching for function calls, and bias detection in LLM judges.
Its applications streamline complex evaluation processes by enhancing efficiency, accuracy, and robustness in dynamic, context-rich testing environments.

ComplexEval is a name that has been used for multiple distinct evaluation systems in recent arXiv literature. In the provided record, it denotes: a framework for learned surrogate evaluation of mini agents grounded in a “computational theory of evaluation”; an automatic framework for quantitatively evaluating complex function calling in the ComplexFuncBench benchmark; and a challenge benchmark for diagnosing auxiliary-information-induced biases in LLM-as-a-judge settings (Yan, 27 Mar 2025, Zhong et al., 17 Jan 2025, Li et al., 3 Sep 2025). The shared motif is evaluation under complexity, but the objects being evaluated differ substantially: candidate agents, tool-using LLMs, and LLM judges.

1. Disambiguation and scope

The term “ComplexEval” does not denote a single standardized framework across the literature represented here. Instead, it appears in at least three technically unrelated senses.

Usage	Primary object of evaluation	Core purpose
ComplexEval in mini agent evaluation	mini agents / candidate policies / models	build evaluation model to accelerate the evaluation procedures
ComplexEval in ComplexFuncBench	complex function calling behavior	quantitatively assess both function-calling correctness across multiple steps and final natural-language response quality
ComplexEval as a judge benchmark	LLM-as-a-judge systems	expose and quantify Auxiliary Information Induced Biases

In the mini-agent setting, ComplexEval is a learned evaluator that predicts evaluation outcomes from agent representations and evaluation conditions. In the function-calling setting, ComplexEval is an automatic framework combining format checking, Hungarian matching, semantic equivalence judging, stepwise golden-path updating, and LLM-based response scoring. In the LLM-judge setting, ComplexEval is a benchmark with basic and advanced scenarios designed to measure how references, rubrics, and other auxiliary information bias judgments (Yan, 27 Mar 2025, Zhong et al., 17 Jan 2025, Li et al., 3 Sep 2025).

This multiplicity matters because statements about “ComplexEval” are otherwise ambiguous. A claim about causal guarantees, for example, belongs to the mini-agent framework, whereas statements about format bias or criteria entanglement belong to the LLM-judge benchmark.

2. ComplexEval as learned surrogate evaluation for mini agents

In “A Computational Theory for Efficient Mini Agent Evaluation with Causal Guarantees,” ComplexEval is introduced as a framework for reducing the cost of experimental evaluation for agents by building an evaluation model that predicts outcomes instead of repeatedly executing the true evaluation process (Yan, 27 Mar 2025). The paper frames the problem through “mini agents”: smaller, specialized agents or candidate policies/models whose quality must be assessed under costly evaluation conditions.

The evaluation model, denoted $EM$ , maps the subject and the evaluation condition to an estimated evaluation indicator: $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ Here, the “subject” is the mini agent or policy being evaluated, the evaluation condition $EC$ is the context of assessment, and the evaluation indicator $EI$ is the performance metric of interest. The practical point is that, once trained, the evaluation model replaces expensive direct execution with a cheap forward-pass prediction (Yan, 27 Mar 2025).

The paper instantiates this abstraction in several domains. In the medical AI evaluation, subjects are linear AI models whose input is patients and output is the alert decision; $EI$ is whether the patient still lives after 14 days of the AI model’s intervention; and $EC$ includes patient demographics, ICD documentation, hospital information, and lab measurements. In the sales conversion test, subjects are vectorized classification models deciding ad exposure; $EI$ is the total number of people who bought the product after seeing the advertisement decision; and $EC$ includes ad ID, campaign IDs, age, gender, and interest. In the trade-strategy setting, subjects are linear classification models producing buy/sell/hold decisions; $EI$ is total return in the current time slot; and $EC$ includes market information such as open, high, low, close, volume, and related fields (Yan, 27 Mar 2025).

The same paper states that the framework reduced 24.1% to 99.0% evaluation errors across 12 scenes and reduced evaluation time 3 to 7 order of magnitude per subject comparing with experiments or simulations (Yan, 27 Mar 2025). This suggests a surrogate-evaluation view of benchmarking in which evaluation itself becomes a prediction problem.

3. Formal guarantees, learning setup, and experimental structure in the mini-agent framework

The theoretical backbone of the mini-agent ComplexEval is a bound on generalized error and generalized causal effect error. The paper states: $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ 0 where $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ 1 is the number of IID loss measurements and $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ 2 is the confidence level (Yan, 27 Mar 2025). The proof uses the bounded-loss assumption on $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ 3 together with Hoeffding inequality, and the text explicitly concludes that the statement is satisfied for both GCEE and GEE.

The practical learning pipeline is described as: collect subject- $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ 4 pairs and observed evaluation outcomes, train $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ 5 to predict $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ 6, and then use the trained evaluator to score new candidate agents cheaply. The paper also mentions a meta-learner for heterogeneous agent spaces, motivated by cases in which a single fixed predictor may not generalize across very different agent families. Concrete learners used in experiments include logistic regression, linear models, XGBoost, and CatBoost, with the sales-conversion setting using a composition model $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ 7, where $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ 8 is logistic regression and $EM(\text{subject}, EC) \rightarrow \widehat{EI}.$ 9 is linear (Yan, 27 Mar 2025).

Several experimental details are explicit. In the medical setting, the paper states: ES number is 50, dataset splits is 20, learning loss is MSE loss, and evaluation loss is MSE loss. In the AKI alert treatment task, sampling methods for AI models include Sigmoid, Arctan, and Gaussian; sigmoid and arctan sampling map parameters from $EC$ 0 to $EC$ 1, while Gaussian sampling uses $EC$ 2. A synthetic evaluation indicator is generated by

$EC$ 3

where $EC$ 4 are sampled from $EC$ 5 (Yan, 27 Mar 2025).

In the trading setup, the dataset includes 3300 stocks from AKShare, and after preprocessing, stocks with fewer than 89 days were removed, leaving 3,214 stocks. The paper uses 70 days as train data and 70–80 days as test data. It also describes a semi-synthetic setup with 85% synthetic $EC$ 6 and 15% real $EC$ 7 without any subject. In the sales conversion experiment, the study was performed 5 times, with 20 train/test splits, and 5 models were tested (Yan, 27 Mar 2025).

The guarantees, however, are explicitly conditional: they require IID loss measurements, loss bounded in $EC$ 8, and sufficiently small empirical error. The paper also notes that some experiments mix synthetic and real $EC$ 9, so the reported performance depends on the quality of the synthetic construction and the realism of the data-generation process (Yan, 27 Mar 2025).

4. ComplexEval as an automatic framework for complex function calling

In “ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario,” ComplexEval denotes an automatic evaluation framework for complex function calling (Zhong et al., 17 Jan 2025). It is designed to assess both function-calling correctness across multiple steps and final natural-language response quality, and it is built specifically for ComplexFuncBench, which contains 1,000 real-world complex function-calling samples drawn from five scenarios: Hotels, Flights, Car Rental, Attraction, and Cross-domain tasks (Zhong et al., 17 Jan 2025).

The benchmark emphasizes five challenge types: multi-step function calling, constrained function calling, parameter value reasoning, long parameter filing, and 128k long-context inputs. Dataset composition is given as 600 single-domain samples and 400 cross-domain samples, with average function-calling steps of 3.26 and average function calls per sample of 5.07 (Zhong et al., 17 Jan 2025).

ComplexEval operates iteratively. At each step $EI$ 0, the model predicts a list of function calls $EI$ 1, while the gold annotation contains $EI$ 2. The framework first performs format checking: it verifies that the function name is in the available function list, that all required parameters are present, and that parameter types match the function specification. If a predicted call fails format checking, the system returns a specific error message as the API response rather than terminating evaluation, so that trial-and-error behavior and self-correction can still be assessed (Zhong et al., 17 Jan 2025).

For calls passing format checks, ComplexEval aligns predicted calls to gold calls using embeddings from bge-large-en-v1.5 and the Hungarian algorithm: $EI$ 3 After alignment, equivalence is judged by one of three criteria: exact rule-based matching, response-based matching, or GPT-4o-based LLM matching. Response-based matching handles cases such as omitted default parameters, while LLM-based matching handles semantically identical but surface-different parameters such as “NY” versus “New York” (Zhong et al., 17 Jan 2025).

The framework also updates the gold call list dynamically through the “Golden Function Call List Updating Strategy.” With annotated shortest path

$EI$ 4

successfully matched calls are removed from the current gold step and the next-step gold calls are appended. This yields a stepwise evaluation procedure rather than a single end-state comparison (Zhong et al., 17 Jan 2025).

Two function-calling metrics are reported. Success Rate measures task completion. Call Accuracy is defined as

$EI$ 5

where $EI$ 6 is the number of correct function calls in sample $EI$ 7 and $EI$ 8 is the total number of function calls in sample $EI$ 9. Final responses are then scored by GPT-4o on Completeness and Correctness, each using the discrete score set $EI$ 0 (Zhong et al., 17 Jan 2025).

The paper’s empirical findings state that closed-source models outperform open-source models, with Claude-3.5-Sonnet at 61.0% success, GPT-4o at 60.5% success, GLM-4-Long at 57.1% success, and Qwen2.5-72B as the best open-source model at 40.1% success. It also reports that value_error is the dominant error type across models, which is presented as a direct justification for semantic and response-based matching rather than exact string comparison alone (Zhong et al., 17 Jan 2025).

5. ComplexEval as a benchmark for bias in LLM-as-a-judge systems

In “Curse of Knowledge: When Complex Evaluation Context Benefits yet Biases LLM Judges,” ComplexEval is a challenge benchmark for studying how auxiliary information used in LLM-as-a-judge evaluation can improve judgment quality while also introducing systematic bias (Li et al., 3 Sep 2025). The paper positions this as a benchmark for complex evaluation settings in which judges are given reference answers, rubrics or scoring principles, and background knowledge.

The benchmark has a two-tier design. ComplexEval-Basic extends RMB-bench and contains 12 diverse domains or scenarios and 1,056 samples in preference-pair form $EI$ 1. Because the original data lacked auxiliary information, the authors synthesize reference answers with DeepSeek-V3 and generate rubrics with few-shot prompting. ComplexEval-Advanced contains 400 samples across three highly complex scenarios: Mathematical Reasoning, Creative Writing, and Role-Playing, using richer human-written or higher-complexity auxiliary information such as Olympiad solutions, multi-dimensional writing rubrics, canonical dialogues, and character backgrounds (Li et al., 3 Sep 2025).

The paper validates six bias mechanisms: Format Bias, Solution Fixation Bias, Stereotype Amplification Bias, Criteria Loophole Bias, Criteria Entanglement Bias, and the Attention Limit Phenomenon. The first three are reference-induced, the next two rubric-induced, and the last is described as a broader failure mode in multi-dimensional evaluation (Li et al., 3 Sep 2025).

For ComplexEval-Basic, point-wise scoring is written as

$EI$ 2

where $EI$ 3 is the judge model and $EI$ 4 is auxiliary information. Preference decisions use a tolerance threshold $EI$ 5: $EI$ 6 and the two main metrics are

$EI$ 7

and

$EI$ 8

For the advanced tier, the main bias metric is Attack Success Rate: $EI$ 9 Task-specific success conditions differ between mathematical reasoning and the writing or role-playing settings (Li et al., 3 Sep 2025).

The study evaluates paired general and reasoning-enhanced models, including Qwen2.5-32B versus QwQ-32B, DeepSeek-V3 versus DeepSeek-R1, GPT-4o-mini, and o4-mini. Its principal result is a paradox: auxiliary information helps on clean data but increases vulnerability under attack, and bias severity scales with task complexity. The paper also states that reasoning models often outperform general models on clean evaluation but show greater bias susceptibility in complex evaluation settings, a result it describes as “paradoxical vulnerability” (Li et al., 3 Sep 2025).

6. Comparative interpretation and recurrent design themes

Across these three usages, ComplexEval consistently denotes evaluation under structural, contextual, or combinatorial complexity, but the evaluation object and validity criterion change from one paper to another.

The mini-agent framework treats evaluation as a supervised prediction problem over subject-condition pairs and emphasizes statistical guarantees, efficiency, and consistency to estimated causal effect from deployed agents to evaluation metric by prediction. The function-calling framework treats evaluation as a process-level alignment problem over multi-step tool trajectories, with explicit mechanisms for semantic parameter equivalence and dynamic golden-path updating. The LLM-judge benchmark treats evaluation as robustness analysis under auxiliary information, formalizing bias not as ordinary inaccuracy alone but as susceptibility to manipulated references and rubrics (Yan, 27 Mar 2025, Zhong et al., 17 Jan 2025, Li et al., 3 Sep 2025).

A common pattern is the rejection of naive end-point scoring. In the mini-agent setting, repeated full deployment is replaced by a learned surrogate evaluator. In the function-calling setting, final-state-only scoring is explicitly criticized because it ignores intermediate tool-call correctness. In the LLM-judge setting, agreement on clean data is treated as incomplete because judges may still be systematically manipulable under rich context (Yan, 27 Mar 2025, Zhong et al., 17 Jan 2025, Li et al., 3 Sep 2025).

Another common pattern is that each framework enlarges the role of context. For mini agents, the evaluation condition $EC$ 0 is part of the predictor input. For function calling, long API histories, constraints, and parameter dependencies are central. For LLM judges, references, rubrics, and background knowledge are the very source of both accuracy gains and bias (Yan, 27 Mar 2025, Zhong et al., 17 Jan 2025, Li et al., 3 Sep 2025). This suggests that “ComplexEval,” across its uses, names a family resemblance rather than a single method: evaluation becomes difficult when the object being assessed interacts with heterogeneous context, and the core technical problem is how to preserve fidelity, efficiency, or robustness under that interaction.