ISO-Bench: Advanced ML Evaluation Suite

• ISO-Bench is a comprehensive benchmark suite that evaluates advanced machine learning models and coding agents across inference optimization, multimodal reasoning, and Text-to-GQL tasks.
• It employs a dual-metric evaluation framework combining hard execution metrics and soft semantic assessments to ensure robust performance improvements.
• Empirical results highlight key challenges in agent scaffolding and tool integration, emphasizing the need for reliable, deployable solutions in real-world applications.

ISO-Bench denotes several specialized benchmarks targeting critical gaps in the evaluation of advanced machine learning models and coding agents. Across recent research, the term references: (i) an inference-server optimization suite for coding agents operating in large-scale LLM serving stacks; (ii) a testbed for multimodal vision-language causal reasoning; (iii) a probe for representation invariance in multimodal foundational models; and (iv) an evaluation slice for Text-to-Graph Query Language (Text-to-GQL) systems. In each context, ISO-Bench enforces rigorous task design, authentic data curation, and nuanced, multi-metric performance criteria, leading to high diagnostic value for both system and model researchers.

1. ISO-Bench for Coding Agents in Inference Optimization

ISO-Bench, as introduced in "ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?" (Nangia et al., 23 Feb 2026), targets the ability of LLM-based coding agents to improve real-world GPU inference workloads. This suite distinguishes itself from previous code generation and optimization benchmarks by focusing on production, end-to-end optimization tasks from mature serving frameworks—specifically vLLM (PagedAttention kernel, advanced memory management) and SGLang (model serving DSL compiled to kernels).

Each task is defined by a codebase snapshot (pre-optimization commit) and a natural-language bottleneck description. Agents are granted up to 120 minutes for solution development, submitting a diff patch, which is evaluated versus expert human solutions for both correctness and measurable performance gains. Accepted patches must reproduce a ≥5% performance improvement on H100 GPUs, verified by rigorous, real deployment workloads, and not alter APIs or dependencies.

2. Dataset Construction and Task Structure

ISO-Bench's task set (54 total: 39 vLLM, 15 SGLang) is curated through a multi-stage pipeline:

1. Keyword Filtering: Mining 15,234 vLLM and 8,421 SGLang commits for optimization or performance keywords, yielding 892 and 547 candidates respectively.
2. Scope Filtering: Limiting to localized PRs (≤10 files changed).
3. LLM-Based Classification: Employing GPT-5-mini to separate true GPU inference optimizations from mere refactoring or bug fixes.
4. Manual Verification: Ensuring reproducibility, extracting standard benchmarks, and authoring bottleneck descriptions that mask the underlying solution.

Tasks span a range of realistic optimizations—kernel call restructuring, regex elimination, cache logic improvement, and kernel launch overhead reduction. Each is coupled to quantitative benchmarks measuring either streaming latency (Time To First Token, TTFT) or batch throughput.

3. Evaluation Framework and Quadrant Taxonomy

ISO-Bench employs a dual-metric evaluation protocol to identify authentic optimization:

• Hard Metrics (Execution-Based): Quantitative improvements are computed as

$$\Delta\mathrm{TTFT} = \frac{\mathrm{TTFT}_h - \mathrm{TTFT}_a}{\mathrm{TTFT}_h} \times 100$$

$$\Delta\mathrm{Throughput} = \frac{\mathrm{Thr}_a - \mathrm{Thr}_h}{\mathrm{Thr}_h} \times 100$$

Results are categorized into Beats (>5%), Similar (±5%), Worse (<−5%), and Failed (errors).

• Soft Metrics (Semantic/LLM Evaluation): Using Gemini-3-Flash, agent patches are compared to human patches along:
  • Bottleneck Targeting: Same, related, or different code region as human fix.
  • Implementation Approach: Mirrored, valid alternative, partial, or ineffective.

This two-dimensional grid yields four outcome quadrants:

	Performance Good	Performance Bad
Correct Target	Q1: True Success	Q2: Good Intent, Bad Execution
Wrong Target	Q3: Lucky Win	Q4: Complete Failure

Functional correctness is always revalidated in Hard Success cases. "True Success" (Q1) requires both a hard-metric win and semantic alignment. "Lucky wins" (Q3—hard-metric win without correct bottleneck) are not counted toward genuine optimization rates as they often undermine accuracy.

4. Benchmark Results and Failure Analysis

Empirically, there is no single dominating agent:

• On vLLM, Claude Code (Sonnet 4.5) achieves 46.2% True Success, outperforming alternatives but rarely mirroring the human approach.
• For SGLang, open-source agents (TRAE, Codex CLI) and TRAE(GPT-5/Sonnet) attain up to 86.7% True Success by closely matching human patches.

A consistent observation is that reliance on hard metrics alone overstates agent ability (overestimation by 10–20% on SGLang using Claude Code) due to "lucky wins." The main bottleneck is not in bottleneck identification—agents correctly localize performance issues in 75–90% of vLLM tasks (Q1+Q2)—but in reliably synthesizing correct, deployable solutions, leading to frequent Q2 outcomes (Good Intent, Bad Execution). Notably, agent "scaffolding" (tool access, orchestration, stopping criteria) accounts for large differences even when backbone models are identical.

Experiments using open-source models with the TRAE framework showed no working optimizations: frequent issues were planning without execution, improper tool usage, and workflow deadlocks.

5. Key Formulas, Definitions, and Metric Taxonomy

ISO-Bench formalizes its evaluation using the following formulas and categories:

• $\Delta\mathrm{TTFT}$ and $\Delta\mathrm{Throughput}$ are defined as above.
• Hard-metric outcome categories:
  • Beats: >5%
  • Similar: ±5%
  • Worse: <−5%
  • Failed: benchmarking errors
• Soft-metric categories: As defined for bottleneck targeting and implementation.
• Quadrants:
  • Q1 (True Success): Hard and semantic metrics both positive.
  • Q2 (Good Intent, Bad Execution): Correct location, but performance not delivered.
  • Q3 (Lucky Win): Incorrect location, but accidental performance gain.
  • Q4 (Complete Failure): No correct diagnosis and no performance gain.

6. Broader Implications and Comparative Position

ISO-Bench, as a production-oriented code optimization benchmark, exposes failure cases (such as execution, rather than comprehension, being the dominant agent gap) not visible in prior static code or microbenchmark kernels. Its dual-metric structure prevents agents from gaming runtime numbers and yields a more realistic signal of deployable optimization success. The pronounced impact of scaffolding over backbone architectural improvements points to the importance of agent orchestration and tool integration—even with identical LLMs[^1].

[^1]: This suggests future progress for LLM-based coding agents may depend as much on robust agent workflows as on fundamental model advances.

7. Future Directions

Recognized directions include refining agent scaffolding strategies, extending tasks and tooling to further industrial codebases, and integrating robustness checks for functional correctness. The public release of tasks, prompts, container images, and evaluation code positions ISO-Bench as a platform for systematic research on generalizable, production-grade LLM coding agents (Nangia et al., 23 Feb 2026).

Further, given the existence of other benchmarks under the "ISO-Bench" designation in allied domains—notably for multimodal causal reasoning (Sadana et al., 30 Jul 2025), isomorphic representation robustness (Fu et al., 2024), and Text-to-GQL graph agent evaluation (Lyu et al., 12 Feb 2026)—the term "ISO-Bench" is increasingly synonymous with rigorous, diagnostic evaluation suites targeting unsolved or underappreciated challenges in both code and model generalization.