---
title: 'SWE-InfraBench: Evaluating AWS CDK Edits'
url: https://www.emergentmind.com/topics/swe-infrabench
type: topic
---

# SWE-InfraBench: Evaluating AWS CDK Edits

SWE-InfraBench is a benchmark for evaluating large language models on realistic infrastructure-as-code development, specifically incremental edits to AWS CDK repositories in Python. It is designed to reflect enterprise-style cloud engineering workflows in which a model must modify an existing codebase in response to a natural-language change request and succeed by passing tests over synthesized CloudFormation templates, rather than by generating an entire configuration from scratch. The benchmark contains 100 tasks derived from 34 IaC codebases, and its initial evaluation reports that even strong systems remain far from robust single-shot performance: in one-turn mode, the best model, Claude 3.7 Sonnet, reaches 34% correctness, while specialized reasoning models such as DeepSeek R1 reach 24% [2606.05249].

## 1. Definition and domain scope

SWE-InfraBench targets infrastructure-as-code as a software engineering domain in which correctness depends on cloud-provider semantics, resource orchestration, and multi-resource consistency rather than only on local program logic. In this setting, IaC is critical for reliability, scalability, security and compliance, and DevOps integration because infrastructure changes are versioned, reviewed, and tested like software artifacts [2606.05249].

A central distinction in the benchmark is between declarative and imperative IaC. Declarative systems such as Terraform specify a desired end state, whereas imperative systems such as AWS CDK encode infrastructure in a general-purpose programming language with loops, conditionals, abstraction, and tight integration with tests and application code. SWE-InfraBench focuses on the latter: AWS CDK in Python, using incremental repository edits rather than greenfield generation [2606.05249].

This focus makes the benchmark structurally different from conventional code benchmarks. Correctness requires reasoning about AWS-specific semantics such as IAM actions and ARNs, VPC and subnet relationships, API Gateway integration, Lambda wiring, and resource lifecycle configuration. Subtle mis-wiring can break the synthesized stack even when the generated code is syntactically plausible. The benchmark therefore frames IaC editing as a repository-grounded software engineering problem whose semantics are encoded in a synthesized resource graph and verified by tests over that graph [2606.05249].

## 2. Dataset construction and task representation

The dataset contains 100 tasks derived from 34 IaC codebases, including both open-source AWS CDK repositories and custom-developed repositories that were substantially modified and instrumented with tests for the benchmark. On average, each task includes about 10 context files and about 30,000 characters of context. The covered CDK versions range from 2.0.0 to approximately 2.189.1, with a heavy concentration around 2.178.2 [2606.05249].

Task construction follows a three-stage semi-automated pipeline. In Stage I, human experts identify promising repository regions, and an LLM such as Sonnet 3.5 or 3.7 proposes a code block to mask, a natural-language prompt, and a corresponding test file. In Stage II, candidates undergo validation and critique. Two independent critic LLMs assess prompt–requirement alignment, implementation generality, test quality, and test completeness, each returning structured JSON feedback; a suggestion passes only if both critics accept on all four dimensions. A generator then rewrites the prompt and tests using the critics’ feedback. In Stage III, cloud engineers review each candidate for fairness, coverage, and challenge level before final inclusion [2606.05249].

Each task is stored as JSON with fields such as `task_id`, `entry_point`, `prompt`, `cdk_version`, `context`, `canonical_solution`, and `tests`. The model is asked to produce a JSON object of the same structure for `canonical_solution`, using unified diffs keyed by file path. A defining constraint is that only additions are allowed: the benchmark forbids deletions, so an edit is implemented as additive patch-style code insertion into the existing repository [2606.05249].

This representation is closely tied to the benchmark’s target workflow. The model is not asked to output full files or an entire repository; it must instead integrate a localized, structured modification into an existing CDK application. Typical edits create resources, modify configuration, wire dependencies, add IAM policies, or adjust lifecycle properties [2606.05249].

## 3. Cloud semantics and verification protocol

The benchmark’s semantics are centered on AWS CDK constructs and the CloudFormation templates they synthesize. Tasks frequently involve Lambda functions, API Gateway REST APIs, EventBridge, IAM roles and policies, S3, ECS/Fargate, RDS, ElastiCache, and Bedrock Agents. The underlying reasoning problem is therefore one of dependency-preserving graph construction: a correct patch must create or modify the appropriate subgraph of resources, references, permissions, and configuration attributes [2606.05249].

Evaluation is execution-based but deployment-free. The harness applies the model’s unified diffs to the repository, synthesizes the CDK application, and runs `pytest` tests built around `aws_cdk.assertions.Template`. These tests inspect the synthesized CloudFormation, asserting resource existence, property values, tags, and cross-resource relationships. A solution counts as correct only if the patch applies successfully, the project synthesizes, and all tests pass [2606.05249].

The paper defines four principal metrics. `Correctness` is the average number of tasks completed, equivalent to pass@1 in the single-trial setting. `Generation Success` measures whether the model produced valid, applicable JSON diffs. `Passed Tests Share` measures the fraction of individual test cases that pass across all tasks. For repeated sampling, the benchmark reports `pass@k`, defined as the probability that at least one of \(k\) attempts for a task is fully correct. The appendix also introduces Task Success Consistency, derived from the gap between the best and worst attempt-level correctness for the same task [2606.05249].

Prompting is standardized around repository context plus a natural-language change request. Anthropic-style models receive XML-like tags around prompt and context blocks, while other models use a default template. In two-turn settings, the second turn includes the previous attempt, error messages, test results, and optionally retrieved AWS documentation snippets. The model must then return JSON containing an `error_analysis` field and a `regenerated_solution` field [2606.05249].

## 4. Models, results, and error structure

SWE-InfraBench evaluates 20 models spanning proprietary and open-source systems. The proprietary set includes Claude 3.7 Sonnet, Claude 3.5 Sonnet V2, Claude 3.5 Sonnet, Claude 3 Haiku, Gemini 2.5 Pro previews, Gemini 2.0 Flash Lite, Gemini 2.0 Flash, o3, o4-mini, GPT-4.1, and GPT-4o Mini. The open-source set includes DeepSeek R1, Mistral Large, Codestral, LLaMA 3.1 405B, LLaMA 3.1 70B, LLaMA 4 Maverick 17B, LLaMA 4 Scout 17B, and Qwen2.5 72B Instruct Turbo [2606.05249].

In one-turn evaluation, the headline result is that the benchmark is difficult even for current frontier systems. Claude 3.7 Sonnet achieves 34% Correctness, 100% Generation Success, and 53.1% Passed Tests Share. Claude 3.5 Sonnet V2 reaches 32% / 100% / 47.0%. Gemini 2.5 Pro (03-25 Preview) reaches 29% / 97% / 42.5%. o3 reaches 23% / 100% / 30.8%, and DeepSeek R1 reaches 24% / 100% / 34.9%. Performance then drops sharply for many other models: Mistral Large obtains 14% / 89% / 20.3%, LLaMA 3.1 70B obtains 3% / 97% / 7.7%, and Qwen2.5 72B Instruct Turbo obtains 0% / 94% / 2.7% [2606.05249].

A notable empirical pattern is that `Generation Success` is often high while `Correctness` remains low. This indicates that formatting the JSON diff is usually not the dominant problem; the bottleneck is functional correctness under cloud-specific semantics and repository integration. The benchmark therefore distinguishes syntactic compliance from semantic success more sharply than many code-generation tasks [2606.05249].

Repeated sampling improves upper-bound solvability. For a five-trial subset, Claude 3.7 Sonnet reaches pass@5 of 41%, Gemini 2.5 Pro (03-25) reaches 47%, DeepSeek R1 reaches 46%, o3 reaches 45%, and LLaMA 3.1 405B reaches 13% [2606.05249]. This suggests that the problem is only partly one of missing capability; stochastic search over alternative patches also matters.

The reported failure modes are dominated by syntax and logic errors. Syntax errors account for 40–85% of failures and include incorrect CDK property names, outdated APIs, CDK v1/v2 mismatches, non-parsable diffs, and Python syntax errors. Logical errors account for 5–30% of failures and include incorrect IAM actions or ARNs, wrong API Gateway proxy configuration, missing required resources or tags, or incorrect wiring to existing constructs. Response-format errors are comparatively rare, especially for Claude and OpenAI models [2606.05249].

The benchmark also evaluates two-turn repair loops. A second turn with error feedback consistently improves performance. Claude 3.7 Sonnet rises from 34% in one turn to 64% with two-turn high-verbosity feedback. Claude 3.5 Sonnet V2 reaches 56% with two-turn high verbosity and 65% with two-turn plus RAG at high verbosity. GPT-4.1 rises from 18% to 48% with two-turn high verbosity, while DeepSeek R1 reaches 45% with two-turn plus RAG at low verbosity. The paper also notes that high-verbosity tracebacks may be over-informative, because they can reveal expected properties or values that act as implicit hints [2606.05249].

## 5. Position within software engineering benchmark research

SWE-InfraBench is positioned against both IaC-specific and general software engineering benchmarks. Relative to declarative IaC benchmarks such as IaC-Eval and CloudEval-YAML, it shifts the task from whole-configuration synthesis to imperative AWS CDK editing inside an existing repository. Relative to HumanEval, MBPP, and APPS, it replaces function-level algorithmic tasks with repository-level cloud resource reasoning. Relative to CrossCodeEval, it adds provider-specific cloud semantics. Relative to SWE-Bench and SWE-PolyBench, it narrows domain scope but increases fidelity to enterprise-style infrastructure editing [2606.05249].

Its novelty lies in four design choices: imperative AWS CDK rather than Terraform- or YAML-centered IaC; incremental edits rather than greenfield generation; execution-based verification over synthesized CloudFormation rather than heuristics; and a generator–critic–human pipeline for producing benchmark tasks [2606.05249]. The benchmark thus occupies a distinct niche inside repository-grounded SWE evaluation: cloud infrastructure code as a specialized domain with strong dependency, permission, and configuration semantics.

Related work broadens this context. TeleSWEBench shows a parallel move into telecom infrastructure, evaluating commit-driven edits in the srsRAN 5G stack and separating localization from functional correctness in a mathematically rigorous C++ domain [2606.05001]. SWE-Interact adds an orthogonal capability axis by reframing SWE tasks as user-driven, multi-turn coding sessions with progressive requirement reveal [2606.30573]. SWE-Gym provides executable repository environments for training real-world SWE agents, while SWE-smith emphasizes scalable per-repository environment construction and synthetic execution-backed task generation [2412.21139; 2504.21798]. An executable benchmarking suite for tool-using agents pushes further on infra methodology by separating workload, driver, and admitted evidence under a shared auditing contract [2605.11030].

Taken together, these systems indicate that “software engineering benchmark” is no longer a single category. A plausible implication is that repository-grounded evaluation is fragmenting into domain-specific and infrastructure-aware families: cloud IaC, telecom stacks, stateful interactive workflows, training environments, and evidence-centric execution suites.

## 6. Design significance, limitations, and future directions

Several design choices define the benchmark’s methodological identity. AWS CDK in Python was chosen because it reflects imperative IaC integrated with tests and application code. Incremental edits were chosen because they mirror how cloud engineers typically modify existing stacks. Test-based evaluation over synthesized templates was chosen because it is objective and lightweight relative to real deployment. The add-only unified-diff interface reduces destructive edits and simplifies patch application. The semi-automated generator plus dual-critic plus human-review pipeline aims to preserve fairness while scaling task creation [2606.05249].

The benchmark also has explicit limitations. Its coverage is restricted to AWS CDK in Python; it does not include Terraform, Pulumi, Azure, GCP, or multi-cloud settings. Donor-repository selection may introduce bias toward tasks that are both interesting and testable. Test-based evaluation does not capture every relevant property of infrastructure, including cost, security, or performance characteristics not encoded in tests. The two-turn high-verbosity setting may be more informative than many real-world developer workflows. Finally, the 100-task scale reflects a balance between realism and evaluation cost rather than exhaustive coverage of cloud engineering practice [2606.05249].

The paper proposes several extensions: other IaC frameworks and providers, richer agent tooling such as `cdk synth`, partial tests, static analysis, and external documentation access, and stronger evaluation criteria including static analyzers, cost estimation, and LLM-as-a-Judge for maintainability or readability [2606.05249]. This suggests a broader research program in which SWE-InfraBench is less an endpoint than a template for infrastructure-aware SWE evaluation.

In that sense, SWE-InfraBench marks a shift in emphasis from generic code generation to domain-constrained repository modification. It operationalizes cloud infrastructure editing as a benchmarkable SWE problem whose core difficulties are dependency reasoning, service-specific semantics, and correct integration into an existing codebase. Its reported results indicate that present models can often produce syntactically valid patches, and can improve substantially when given iterative feedback, but still struggle to deliver robust single-shot correctness on imperative IaC tasks [2606.05249].

Source: https://www.emergentmind.com/topics/swe-infrabench