Multi-IaC-Bench: LLM IaC Evaluation
- Multi-IaC-Bench is a benchmark dataset that organizes IaC evaluation into triplets of initial templates, modification requests, and updated templates across CloudFormation, Terraform, and CDK.
- It employs a synthetic data pipeline with static linting (CFN-Lint, TFLint, Checkov) and LLM-based semantic judgment to ensure both syntactic and semantic validity.
- Empirical findings show that iterative retry loops boost static pass rates from 60–80% to above 95%, with models like Sonnet 3.5 V2 achieving superior performance.
Multi-IaC-Bench is a benchmark dataset for evaluating LLM-based Infrastructure as Code (IaC) generation and mutation across multiple cloud configuration formats, introduced in “Multi-IaC-Eval: Benchmarking Cloud Infrastructure as Code Across Multiple Formats” (Davidson et al., 21 Aug 2025). It targets a central problem in cloud engineering: different cloud service providers and toolchains rely on diverse IaC representations, while the absence of a standardized format requires cloud architects to be proficient in multiple IaC languages. Multi-IaC-Bench addresses this by organizing evaluation around triplets of an initial IaC template, a natural-language modification request, and an updated template that exactly implements the requested change. The benchmark spans AWS CloudFormation, Terraform, and Cloud Development Kit (CDK), and was constructed through a synthetic generation pipeline with static and LLM-based validation (Davidson et al., 21 Aug 2025).
1. Dataset scope and internal structure
Multi-IaC-Bench consists of 337 CloudFormation triplets, 171 Terraform triplets, and 191 CDK triplets, with the CDK portion divided into 96 Python examples and 95 TypeScript examples (Davidson et al., 21 Aug 2025). Each data point is a 3-element tuple:
initial— the original IaC template, which may be empty and is represented as YAML/JSON for CloudFormation, HCL for Terraform, or a code repository for CDKutterance— a synthetic natural-language “customer request” to add or modify one or more resourcesexpected— the updated IaC template that exactly implements the change described in the utterance
This triplet design supports both generation from scratch and mutation of existing stacks. Input templates range from an empty skeleton to complex stacks with multiple resources and security settings. The benchmark tracks the number of resources per template, measured as the number of top-level Resource blocks or HCL blocks. Across CloudFormation the mean is approximately 4.7 resources per template; for Terraform it is approximately 5.1; for CDK it is approximately 4.5 (Davidson et al., 21 Aug 2025).
| Format | Triplets |
|---|---|
| CloudFormation | 337 |
| Terraform | 171 |
| CDK | 191 |
The benchmark also reports frequent resource types across the three formats. For CloudFormation, the most common include AWS::[IAM](https://www.emergentmind.com/topics/inter-agent-messaging-iam)::Role (85), AWS::EC2::Subnet (71), AWS::EC2::SecurityGroup (71), and AWS::Route (32). For Terraform, common types include aws_s3_bucket (73), aws_iam_role (44), aws_iam_role_policy (37), and kubernetes_config_map (15). For CDK, common types include AWS::EC2::Subnet (95), AWS::S3::Bucket (57), AWS::SSM::Parameter (50), and AWS::Budgets::Budget (17) (Davidson et al., 21 Aug 2025). This distribution suggests that the benchmark is oriented toward practical cloud-stack mutations rather than narrow single-resource edits.
2. Synthetic generation and validation pipeline
The dataset is generated through a multi-stage synthetic data pipeline. First, source CloudFormation and Terraform templates are collected from public GitHub repositories, specifically iac-model-evaluation, iac-eval, and aws-cloudformation-templates (Davidson et al., 21 Aug 2025). These templates are then filtered by static analysis: CloudFormation templates are checked with CFN-Lint, Terraform templates with TFLint, and both are additionally checked with Checkov. Any template that fails the linter is discarded.
For each remaining template, the process repeats up to a limit in order to obtain multiple triplets per input. An LLM, identified as Amazon Nova Pro via AWS Bedrock with temperature $0.9$, is prompted to propose a “realistic” change not seen in prior requests, produce the corresponding natural-language utterance, and generate the updated IaC file (Davidson et al., 21 Aug 2025). The updated file is again subjected to static analysis; failures are discarded and retried. A second validation stage uses an LLM-based judge to verify both that the changes align with the utterance and that no unwanted edits were made. If the judge flags misalignment, the result is discarded and retried.
A stratified sample of 96 CloudFormation triplets is then converted into CDK using the AWS CDK Migrate tool, yielding 191 CDK triplets in Python and TypeScript (Davidson et al., 21 Aug 2025). This creates a benchmark in which one portion is directly sourced from CloudFormation and Terraform corpora, while the CDK portion is derived through conversion. A plausible implication is that CDK evaluation in this benchmark is tied not only to code-editing difficulty but also to properties induced by the CloudFormation-to-CDK migration process.
The pseudocode reported for the pipeline is:
1 2 3 4 5 6 7 8 9 10 11 |
for each source_template in CFN_and_TF_corpus:
if not passes_lint(source_template): continue
prev_requests = ∅
for trial in 1…MaxAttempts:
(request, updated) = LLM_mutate(source_template, prev_requests)
prev_requests += {request}
if not passes_lint(updated): continue
if not LLM_judge_aligns(source_template, request, updated): continue
emit (source_template, request, updated)
break
convert_some_CFNs_to_CDK() |
3. Validation regime and benchmark metrics
Multi-IaC-Bench distinguishes between syntactic correctness and semantic correctness. Syntactic checks are performed using CFN-Lint for CloudFormation, TFLint for Terraform, and Checkov to ensure no formatting, schema, or security errors (Davidson et al., 21 Aug 2025). Semantic checks are performed by an LLM judge, which was validated against human evaluation on a 60-sample subset and achieved 91.6% agreement with Pearson (Davidson et al., 21 Aug 2025).
The benchmark defines three core counts: total_attempts, syntactically_valid, and semantically_correct. On that basis, ValidityRate is defined as syntactically_valid / total_attempts, and SemanticAccuracy is defined as semantically_correct / total_attempts (Davidson et al., 21 Aug 2025). Two additional diagnostics are included: EditDistance, the average Levenshtein distance between generated and reference templates, and AvgLLMCalls, the average number of LLM invocations per test case, which accounts for retries. The benchmark also reports LLMJudgeScore, described as the average “alignment rating” on a Likert scale returned by the judge LLM (Davidson et al., 21 Aug 2025).
A common misconception in IaC generation is that passing linters is equivalent to solving the task. The benchmark’s metric design explicitly rejects that equivalence. High linter pass rates capture static validity and partial compliance with schema and security tooling, but do not establish that the requested edit was performed and only that edit. The inclusion of judge-based semantic evaluation is intended to separate these failure modes.
4. Evaluation protocol for LLMs
The reported evaluation covers Llama 3.2 11B Instruct, DeepSeek R1, and Anthropic Sonnet 3.5 V2, with all inference calls served via AWS Bedrock and experiments run at temperature $0.5$ (Davidson et al., 21 Aug 2025). The prompting protocol contains four variants.
The Basic Prompt provides minimal context, consisting of the initial template and user request. The Full Prompt adds IAM best-practice guidelines and instructs the model to keep changes minimal. Chain-of-Thought (CoT) encourages the model to “think step by step” through the edit. The Retry Mechanism re-prompts on any linter or semantic failure with error messages included (Davidson et al., 21 Aug 2025).
The retry loop is treated as part of the evaluation methodology rather than only as a post-processing heuristic. After a failure, the model receives feedback from linter or Checkov errors and is asked to correct them. The reported effect is substantial: this iterative loop boosts static pass rates from 60–80% up to greater than 95% (Davidson et al., 21 Aug 2025). Within the benchmark’s framing, the number of LLM calls is therefore not incidental; it is a measured component of system behavior.
This setup makes the benchmark useful for comparing not only model quality but also prompt-policy design. A plausible implication is that Multi-IaC-Bench evaluates composite inference procedures—prompt template plus retry policy plus model—rather than a pure one-shot generative capability.
5. Empirical findings across CloudFormation, Terraform, and CDK
The benchmark reports selected results for each model and format in terms of Lint Pass %, Checkov Pass %, AvgLLMCalls, and Judge Score (Davidson et al., 21 Aug 2025).
| Format / Model | Lint Pass % / Checkov Pass % | AvgLLMCalls / Judge Score |
|---|---|---|
| CFN / Llama 3.2 | 72.1% / 89.9% | 2.71 / 1.89 |
| CFN / DeepSeek R1 | 92.0% / 92.6% | 1.79 / 2.06 |
| CFN / Sonnet 3.5 V2 | 98.5% / 98.8% | 1.82 / 2.23 |
| TF / Llama 3.2 | 84.8% / 100% | 2.73 / 2.01 |
| TF / DeepSeek R1 | 98.8% / 98.8% | 1.81 / 2.12 |
| TF / Sonnet 3.5 V2 | 100% / 100% | 2.10 / 2.39 |
| CDK / Llama 3.2 | 36.7% / 98.4% | 3.72 / 1.31 |
| CDK / DeepSeek R1 | 85.3% / 85.9% | 1.60 / 1.65 |
| CDK / Sonnet 3.5 V2 | 95.8% / 96.3% | 1.59 / 1.75 |
The comparative analysis identifies Sonnet 3.5 V2 as strongest across formats, especially CloudFormation and Terraform (Davidson et al., 21 Aug 2025). Llama 3.2 struggles most on CDK conversion, where lint pass is low even though Checkov remains high. The benchmark characterizes CDK as the hardest format because of its more complex code structure, even with the CloudFormation-to-CDK conversion trick (Davidson et al., 21 Aug 2025). It also concludes that retry loops yield the largest gains, whereas CoT adds minor improvements but increases the number of calls.
The abstract summarizes the broader pattern as follows: modern LLMs can achieve high success rates, exceeding 95%, in generating syntactically valid IaC across formats, but significant challenges remain in semantic alignment and handling complex infrastructure patterns (Davidson et al., 21 Aug 2025). That contrast is central to the benchmark’s empirical message.
6. Ablation results and representative triplets
The ablation study is reported for Sonnet 3.5 V2 on CloudFormation. With the Basic Prompt, results are Lint 61.9%, Checkov 56.3%, Judge 2.45, and 1.00 call. With the Full Prompt, results are Lint 69.3%, Checkov 51.1%, Judge 2.40, and 1.00 call. Adding the Retry Loop yields Lint 98.5%, Checkov 98.8%, Judge 2.23, and 1.82 calls. Adding CoT yields Lint 96.0%, Checkov 97.7%, Judge 2.24, and 1.94 calls (Davidson et al., 21 Aug 2025).
The stated findings are that adding best-practice text alone yields modest gains, the retry loop is critical because the static pass rate jumps by more than 30 points, and Chain-of-Thought helps slightly for CDK but is marginal for CloudFormation and Terraform (Davidson et al., 21 Aug 2025). These results place error-aware iteration at the center of the benchmarked workflow.
The benchmark’s example triplets are intentionally simple and illustrate the triplet formalism across formats. In CloudFormation, the original template contains an AWS::Budgets::Budget resource with Amount: 100, the utterance requests increasing the monthly cost budget from $100` to `$1000, and the expected template updates Amount to 1000. In Terraform, the original template contains an aws_budgets_budget resource with limit_amount = "100", the utterance requests raising the limit_amount to 1000 [USD](https://www.emergentmind.com/topics/uncertainty-aware-semantic-decoding-usd), and the expected template changes that field to "1000". In CDK Python, the original code constructs budgets.CfnBudget with amount=100, the utterance requests modifying the monthly cost budget so that the amount is 1000 USD instead of 100, and the expected code changes the value to amount=1000 (Davidson et al., 21 Aug 2025).
These examples clarify an important property of the benchmark: the task is not generic code completion, but controlled infrastructure mutation under a natural-language edit specification. This suggests a close relationship to edit-based code generation benchmarks, but with domain-specific static analyzers and security checks integrated directly into the evaluation loop.
7. Position within AI-assisted infrastructure management
Multi-IaC-Bench is presented as a benchmark intended to facilitate further research in AI-assisted infrastructure management and to establish standardized evaluation metrics for this domain (Davidson et al., 21 Aug 2025). Its contribution lies in combining three dimensions that are often separated: cross-format coverage, mutation-oriented triplets, and a validation stack that includes both conventional static tooling and semantic judgment by an LLM.
Within that framing, the benchmark operationalizes IaC generation as a sequence of constrained transformations over machine-readable infrastructure definitions. The use of CloudFormation, Terraform, and CDK places declarative and code-centric IaC styles in a single evaluation space. The inclusion of IAM best-practice guidance, Checkov-based security validation, and minimal-change prompting reflects the fact that IaC correctness in practice is not reducible to parsing or compilation alone.
The benchmark also highlights a persistent tension in LLM evaluation for software artifacts. Strong static validity rates, including results above 95%, do not eliminate semantic misalignment or difficulty with complex infrastructure patterns (Davidson et al., 21 Aug 2025). A plausible implication is that future work on IaC assistants will need to improve not only format-specific syntax handling but also edit localization, preservation of untouched resources, and reliable satisfaction of natural-language change constraints across heterogeneous IaC ecosystems.
