SWE-InfraBench: Evaluating AWS CDK Edits

• SWE-InfraBench is a benchmark for evaluating LLM-driven incremental edits in AWS CDK repositories, simulating enterprise cloud engineering workflows.
• It tests repository-grounded modifications using JSON diffs verified by synthesized CloudFormation templates to assess both syntax and cloud-specific semantics.
• The benchmark utilizes a semi-automated generator–critic–human pipeline across 100 tasks from 34 repositories, highlighting current LLM challenges in complex cloud environments.

SWE-InfraBench is a benchmark for evaluating LLMs on realistic infrastructure-as-code development, specifically incremental edits to AWS CDK repositories in Python. It is designed to reflect enterprise-style cloud engineering workflows in which a model must modify an existing codebase in response to a natural-language change request and succeed by passing tests over synthesized CloudFormation templates, rather than by generating an entire configuration from scratch. The benchmark contains 100 tasks derived from 34 IaC codebases, and its initial evaluation reports that even strong systems remain far from robust single-shot performance: in one-turn mode, the best model, Claude 3.7 Sonnet, reaches 34% correctness, while specialized reasoning models such as DeepSeek R1 reach 24% (Tarasova et al., 3 Jun 2026).

1. Definition and domain scope

SWE-InfraBench targets infrastructure-as-code as a software engineering domain in which correctness depends on cloud-provider semantics, resource orchestration, and multi-resource consistency rather than only on local program logic. In this setting, IaC is critical for reliability, scalability, security and compliance, and DevOps integration because infrastructure changes are versioned, reviewed, and tested like software artifacts (Tarasova et al., 3 Jun 2026).

A central distinction in the benchmark is between declarative and imperative IaC. Declarative systems such as Terraform specify a desired end state, whereas imperative systems such as AWS CDK encode infrastructure in a general-purpose programming language with loops, conditionals, abstraction, and tight integration with tests and application code. SWE-InfraBench focuses on the latter: AWS CDK in Python, using incremental repository edits rather than greenfield generation (Tarasova et al., 3 Jun 2026).

This focus makes the benchmark structurally different from conventional code benchmarks. Correctness requires reasoning about AWS-specific semantics such as IAM actions and ARNs, VPC and subnet relationships, API Gateway integration, Lambda wiring, and resource lifecycle configuration. Subtle mis-wiring can break the synthesized stack even when the generated code is syntactically plausible. The benchmark therefore frames IaC editing as a repository-grounded software engineering problem whose semantics are encoded in a synthesized resource graph and verified by tests over that graph (Tarasova et al., 3 Jun 2026).

2. Dataset construction and task representation

The dataset contains 100 tasks derived from 34 IaC codebases, including both open-source AWS CDK repositories and custom-developed repositories that were substantially modified and instrumented with tests for the benchmark. On average, each task includes about 10 context files and about 30,000 characters of context. The covered CDK versions range from 2.0.0 to approximately 2.189.1, with a heavy concentration around 2.178.2 (Tarasova et al., 3 Jun 2026).

Task construction follows a three-stage semi-automated pipeline. In Stage I, human experts identify promising repository regions, and an LLM such as Sonnet 3.5 or 3.7 proposes a code block to mask, a natural-language prompt, and a corresponding test file. In Stage II, candidates undergo validation and critique. Two independent critic LLMs assess prompt–requirement alignment, implementation generality, test quality, and test completeness, each returning structured JSON feedback; a suggestion passes only if both critics accept on all four dimensions. A generator then rewrites the prompt and tests using the critics’ feedback. In Stage III, cloud engineers review each candidate for fairness, coverage, and challenge level before final inclusion (Tarasova et al., 3 Jun 2026).

Each task is stored as JSON with fields such as task_id, entry_point, prompt, cdk_version, context, canonical_solution, and tests. The model is asked to produce a JSON object of the same structure for canonical_solution, using unified diffs keyed by file path. A defining constraint is that only additions are allowed: the benchmark forbids deletions, so an edit is implemented as additive patch-style code insertion into the existing repository (Tarasova et al., 3 Jun 2026).

This representation is closely tied to the benchmark’s target workflow. The model is not asked to output full files or an entire repository; it must instead integrate a localized, structured modification into an existing CDK application. Typical edits create resources, modify configuration, wire dependencies, add IAM policies, or adjust lifecycle properties (Tarasova et al., 3 Jun 2026).

3. Cloud semantics and verification protocol

The benchmark’s semantics are centered on AWS CDK constructs and the CloudFormation templates they synthesize. Tasks frequently involve Lambda functions, API Gateway REST APIs, EventBridge, IAM roles and policies, S3, ECS/Fargate, RDS, ElastiCache, and Bedrock Agents. The underlying reasoning problem is therefore one of dependency-preserving graph construction: a correct patch must create or modify the appropriate subgraph of resources, references, permissions, and configuration attributes (Tarasova et al., 3 Jun 2026).

Evaluation is execution-based but deployment-free. The harness applies the model’s unified diffs to the repository, synthesizes the CDK application, and runs pytest tests built around aws_cdk.assertions.Template. These tests inspect the synthesized CloudFormation, asserting resource existence, property values, tags, and cross-resource relationships. A solution counts as correct only if the patch applies successfully, the project synthesizes, and all tests pass (Tarasova et al., 3 Jun 2026).

The paper defines four principal metrics. Correctness is the average number of tasks completed, equivalent to pass@1 in the single-trial setting. Generation Success measures whether the model produced valid, applicable JSON diffs. Passed Tests Share measures the fraction of individual test cases that pass across all tasks. For repeated sampling, the benchmark reports pass@k, defined as the probability that at least one of $k$ attempts for a task is fully correct. The appendix also introduces Task Success Consistency, derived from the gap between the best and worst attempt-level correctness for the same task (Tarasova et al., 3 Jun 2026).

Prompting is standardized around repository context plus a natural-language change request. Anthropic-style models receive XML-like tags around prompt and context blocks, while other models use a default template. In two-turn settings, the second turn includes the previous attempt, error messages, test results, and optionally retrieved AWS documentation snippets. The model must then return JSON containing an error_analysis field and a regenerated_solution field (Tarasova et al., 3 Jun 2026).

4. Models, results, and error structure

SWE-InfraBench evaluates 20 models spanning proprietary and open-source systems. The proprietary set includes Claude 3.7 Sonnet, Claude 3.5 Sonnet V2, Claude 3.5 Sonnet, Claude 3 Haiku, Gemini 2.5 Pro previews, Gemini 2.0 Flash Lite, Gemini 2.0 Flash, o3, o4-mini, GPT-4.1, and GPT-4o Mini. The open-source set includes DeepSeek R1, Mistral Large, Codestral, LLaMA 3.1 405B, LLaMA 3.1 70B, LLaMA 4 Maverick 17B, LLaMA 4 Scout 17B, and Qwen2.5 72B Instruct Turbo (Tarasova et al., 3 Jun 2026).

In one-turn evaluation, the headline result is that the benchmark is difficult even for current frontier systems. Claude 3.7 Sonnet achieves 34% Correctness, 100% Generation Success, and 53.1% Passed Tests Share. Claude 3.5 Sonnet V2 reaches 32% / 100% / 47.0%. Gemini 2.5 Pro (03-25 Preview) reaches 29% / 97% / 42.5%. o3 reaches 23% / 100% / 30.8%, and DeepSeek R1 reaches 24% / 100% / 34.9%. Performance then drops sharply for many other models: Mistral Large obtains 14% / 89% / 20.3%, LLaMA 3.1 70B obtains 3% / 97% / 7.7%, and Qwen2.5 72B Instruct Turbo obtains 0% / 94% / 2.7% (Tarasova et al., 3 Jun 2026).

A notable empirical pattern is that Generation Success is often high while Correctness remains low. This indicates that formatting the JSON diff is usually not the dominant problem; the bottleneck is functional correctness under cloud-specific semantics and repository integration. The benchmark therefore distinguishes syntactic compliance from semantic success more sharply than many code-generation tasks (Tarasova et al., 3 Jun 2026).

Repeated sampling improves upper-bound solvability. For a five-trial subset, Claude 3.7 Sonnet reaches pass@5 of 41%, Gemini 2.5 Pro (03-25) reaches 47%, DeepSeek R1 reaches 46%, o3 reaches 45%, and LLaMA 3.1 405B reaches 13% (Tarasova et al., 3 Jun 2026). This suggests that the problem is only partly one of missing capability; stochastic search over alternative patches also matters.

The reported failure modes are dominated by syntax and logic errors. Syntax errors account for 40–85% of failures and include incorrect CDK property names, outdated APIs, CDK v1/v2 mismatches, non-parsable diffs, and Python syntax errors. Logical errors account for 5–30% of failures and include incorrect IAM actions or ARNs, wrong API Gateway proxy configuration, missing required resources or tags, or incorrect wiring to existing constructs. Response-format errors are comparatively rare, especially for Claude and OpenAI models (Tarasova et al., 3 Jun 2026).

The benchmark also evaluates two-turn repair loops. A second turn with error feedback consistently improves performance. Claude 3.7 Sonnet rises from 34% in one turn to 64% with two-turn high-verbosity feedback. Claude 3.5 Sonnet V2 reaches 56% with two-turn high verbosity and 65% with two-turn plus RAG at high verbosity. GPT-4.1 rises from 18% to 48% with two-turn high verbosity, while DeepSeek R1 reaches 45% with two-turn plus RAG at low verbosity. The paper also notes that high-verbosity tracebacks may be over-informative, because they can reveal expected properties or values that act as implicit hints (Tarasova et al., 3 Jun 2026).

5. Position within software engineering benchmark research

SWE-InfraBench is positioned against both IaC-specific and general software engineering benchmarks. Relative to declarative IaC benchmarks such as IaC-Eval and CloudEval-YAML, it shifts the task from whole-configuration synthesis to imperative AWS CDK editing inside an existing repository. Relative to HumanEval, MBPP, and APPS, it replaces function-level algorithmic tasks with repository-level cloud resource reasoning. Relative to CrossCodeEval, it adds provider-specific cloud semantics. Relative to SWE-Bench and SWE-PolyBench, it narrows domain scope but increases fidelity to enterprise-style infrastructure editing (Tarasova et al., 3 Jun 2026).

Its novelty lies in four design choices: imperative AWS CDK rather than Terraform- or YAML-centered IaC; incremental edits rather than greenfield generation; execution-based verification over synthesized CloudFormation rather than heuristics; and a generator–critic–human pipeline for producing benchmark tasks (Tarasova et al., 3 Jun 2026). The benchmark thus occupies a distinct niche inside repository-grounded SWE evaluation: cloud infrastructure code as a specialized domain with strong dependency, permission, and configuration semantics.

Related work broadens this context. TeleSWEBench shows a parallel move into telecom infrastructure, evaluating commit-driven edits in the srsRAN 5G stack and separating localization from functional correctness in a mathematically rigorous C++ domain (Gajjar et al., 3 Jun 2026). SWE-Interact adds an orthogonal capability axis by reframing SWE tasks as user-driven, multi-turn coding sessions with progressive requirement reveal (Raghavendra et al., 29 Jun 2026). SWE-Gym provides executable repository environments for training real-world SWE agents, while SWE-smith emphasizes scalable per-repository environment construction and synthetic execution-backed task generation (Pan et al., 2024, Yang et al., 30 Apr 2025). An executable benchmarking suite for tool-using agents pushes further on infra methodology by separating workload, driver, and admitted evidence under a shared auditing contract (Zhong et al., 10 May 2026).

Taken together, these systems indicate that “software engineering benchmark” is no longer a single category. A plausible implication is that repository-grounded evaluation is fragmenting into domain-specific and infrastructure-aware families: cloud IaC, telecom stacks, stateful interactive workflows, training environments, and evidence-centric execution suites.

6. Design significance, limitations, and future directions

Several design choices define the benchmark’s methodological identity. AWS CDK in Python was chosen because it reflects imperative IaC integrated with tests and application code. Incremental edits were chosen because they mirror how cloud engineers typically modify existing stacks. Test-based evaluation over synthesized templates was chosen because it is objective and lightweight relative to real deployment. The add-only unified-diff interface reduces destructive edits and simplifies patch application. The semi-automated generator plus dual-critic plus human-review pipeline aims to preserve fairness while scaling task creation (Tarasova et al., 3 Jun 2026).

The benchmark also has explicit limitations. Its coverage is restricted to AWS CDK in Python; it does not include Terraform, Pulumi, Azure, GCP, or multi-cloud settings. Donor-repository selection may introduce bias toward tasks that are both interesting and testable. Test-based evaluation does not capture every relevant property of infrastructure, including cost, security, or performance characteristics not encoded in tests. The two-turn high-verbosity setting may be more informative than many real-world developer workflows. Finally, the 100-task scale reflects a balance between realism and evaluation cost rather than exhaustive coverage of cloud engineering practice (Tarasova et al., 3 Jun 2026).

The paper proposes several extensions: other IaC frameworks and providers, richer agent tooling such as cdk synth, partial tests, static analysis, and external documentation access, and stronger evaluation criteria including static analyzers, cost estimation, and LLM-as-a-Judge for maintainability or readability (Tarasova et al., 3 Jun 2026). This suggests a broader research program in which SWE-InfraBench is less an endpoint than a template for infrastructure-aware SWE evaluation.

In that sense, SWE-InfraBench marks a shift in emphasis from generic code generation to domain-constrained repository modification. It operationalizes cloud infrastructure editing as a benchmarkable SWE problem whose core difficulties are dependency reasoning, service-specific semantics, and correct integration into an existing codebase. Its reported results indicate that present models can often produce syntactically valid patches, and can improve substantially when given iterative feedback, but still struggle to deliver robust single-shot correctness on imperative IaC tasks (Tarasova et al., 3 Jun 2026).