SR-Eval: Structured Evaluation for LLMs

• SR-Eval is a framework that decomposes complex tasks into multiple iterative steps to rigorously evaluate LLMs and symbolic algorithms.
• It employs multi-turn benchmarks with precise metrics like per-turn accuracy and complete task rate to diagnose error propagation and semantic drift.
• The approach integrates context management and prompt engineering strategies to balance efficiency and accuracy in evolving task scenarios.

SR-Eval designates a class of evaluation frameworks and metrics for the rigorous assessment of LLMs and symbolic algorithms across diverse tasks involving complex, multi-step requirements. While the term appears across multiple domains—including equation recovery, dialogue evaluation, code generation, and systematic review screening—the unifying concept is the structured measurement of system performance under realistic task decompositions or evolving requirement workflows.

1. Scope and Objectives

SR-Eval frameworks are constructed to address the limitations of static, single-turn, or reference-only benchmarks by modeling the true interactive, iterative, or compositional nature of real-world scientific and engineering tasks. Central to SR-Eval methodologies is the decomposition of tasks into multiple steps, turns, or requirements, capturing the incremental and dependent nature of target problems. This design enables the disentanglement of challenges such as error propagation, context management, and semantic drift, which are not revealed by conventional one-shot evaluation paradigms (Zhan et al., 23 Sep 2025).

2. Multi-turn and Iterative Benchmark Design

The prototypical SR-Eval for code generation (Zhan et al., 23 Sep 2025) operationalizes "stepwise requirement refinement," mirroring agile development where requirements evolve over a sequence of turns. Construction of such a benchmark involves:

• Multi-agent requirement generation: Decomposing a "final" problem specification into a core requirement plus several (typically 1–4) incremental refinements. Each turn introduces a new feature, constraint, or behavioral tweak.
• Evaluator and analyzer agents: Critical review of each refinement for testability, scenario authenticity, and distinctiveness, often structured as a directed acyclic graph to enforce execution order and dependency constraints.
• Semantic-aware discriminative test case generation: For every decomposition/turn, explicit reference implementations and highly discriminative testing suites are created. These not only validate correctness but require that each new refinement is both implemented and uniquely verifiable (e.g., a new test must fail on the previous code and pass on the current).

This pipeline is essential for producing diagnostic multi-turn benchmarks for LLM-based code generation, where each task consists of several evolving requirements and associated turn-level evaluations.

3. Evaluation Metrics and Methodologies

SR-Eval frameworks employ a variety of metrics tailored to the underlying task modality. Chief metrics for multi-turn code generation (Zhan et al., 23 Sep 2025) include:

• Per-turn Accuracy: For task set $\mathcal{T}$, let $N_t$ be the number of turns for task $t$, and $S_{t,k}\in\{0,1\}$ the binary score indicating success for code generated at turn $k$. Then

$$\mathrm{Acc}^{(k)} = \frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}} S_{t,k}$$

• Average Accuracy across Turns:

$$\mathrm{AvgAcc} = \frac{1}{\sum_t N_t}\sum_{t\in\mathcal{T}}\sum_{k=1}^{N_t} S_{t,k}$$

• Complete Task Rate (CR): The fraction of tasks for which all turns are correct on the first attempt,

$$\mathrm{CR} = \frac{|\{\,t\in\mathcal{T}: \forall k, S_{t,k}=1\,\}|}{|\mathcal{T}|}$$

• Average Token Cost (ATC): Mean total tokens consumed per task (input + output), capturing efficiency under various prompting strategies.

These metrics provide high-resolution, turn-by-turn error signatures, exposing the compounding effect of early mistakes on downstream code quality and final task success.

4. Prompt Engineering and Context Management

SR-Eval mandates the assessment of multiple context-handling strategies to mirror diverse LLM usage scenarios:

• Full History: Includes all previous instructions and generated outputs in the prompt.
• Code Edit: Includes only the immediate previous turn's code and the new instruction.
• Cumulative Instruction: Includes all past instructions, omitting intermediate model outputs.

Experiments demonstrate that performance and token efficiency are significantly affected by context window composition: "Code Edit" achieves the highest practical efficiency, while "Full History" inflates cost with limited accuracy benefit. Mismanagement of prompt context also exacerbates error propagation, as reflected in the gap between "Basic" (model-generated code passed as history) and "Golden" (reference implementation history) evaluations (Zhan et al., 23 Sep 2025).

5. Experimental Results and Key Insights

SR-Eval results reveal both the promise and the current bottlenecks of LLM architectures in iterative code-generation tasks.

Setting	Python Func CR	Java Func CR	Python Repo CR	Java Repo CR
Best Model (Basic Setting)	19.72%	22.67%	6.74%	20.00%
Best Model (Golden Setting)	27.46%	23.84%	7.87%	22.50%

• Low complete task rates: Even the best-performing models successfully complete only ≈ 20% of multi-turn tasks, underscoring the challenge of correctly tracking evolving requirements, handling feedback, and executing nontrivial behavioral shifts at each step.
• Error propagation: The "Golden" setting—with reference code history—improves CR only modestly, confirming that early-turn failures degrade long-term success and that context drift and state mismanagement are central failure modes.
• Reasoning mode trade-offs: Larger models outperform smaller ones, but explicit reasoning modules may not improve—and can even harm—multi-step performance due to "overthinking" or losing alignment with evolving task context.
• Prompting impact: Code Edit and Cumulative Instruction strategies outperform Full History in token economy and often in accuracy, emphasizing the importance of context pruning and relevance filtering.

6. Significance, Challenges, and Future Directions

The SR-Eval methodology exposes crucial test scenarios for model robustness in the presence of incremental requirement changes—a regime poorly captured by existing single-turn benchmarks. Its discriminative test-case pipeline, grounded in semantic alignment and behavioral distinctiveness, creates a more stringent evaluation scaffold, mandating explicit satisfaction of each refinement stage and making "specification drift" or context misalignment readily diagnosable (Zhan et al., 23 Sep 2025).

Key open challenges include:

• Self-correction mechanisms: Integrating automated error-detection or review loops to reduce compounding turnwise failures.
• Adaptive reasoning invocation: Selectively engaging advanced reasoning to balance efficiency and accuracy.
• Advanced context pruning and retrieval: Ensuring history provided to the model minimizes distraction and error propagation.
• Benchmark extensions: Adapting SR-Eval principles to other languages, GUI/API programming, or multiparty scenarios to stress-test system generalization.

SR-Eval, as formalized in prominent code generation and symbolic regression benchmarks, provides the methodological foundation for next-generation evaluation of LLMs and symbolic systems in multi-step, contextually entangled domains (Zhan et al., 23 Sep 2025, Kahlmeyer et al., 8 Jun 2026).