---
title: 'Multi-IaC-Bench: LLM IaC Evaluation'
url: https://www.emergentmind.com/topics/multi-iac-bench
type: topic
---

# Multi-IaC-Bench: LLM IaC Evaluation

Multi-IaC-Bench is a benchmark dataset for evaluating LLM-based Infrastructure as Code (IaC) generation and mutation across multiple cloud configuration formats, introduced in “Multi-IaC-Eval: Benchmarking Cloud Infrastructure as Code Across Multiple Formats” [2509.05303]. It targets a central problem in cloud engineering: different cloud service providers and toolchains rely on diverse IaC representations, while the absence of a standardized format requires cloud architects to be proficient in multiple IaC languages. Multi-IaC-Bench addresses this by organizing evaluation around triplets of an initial IaC template, a natural-language modification request, and an updated template that exactly implements the requested change. The benchmark spans AWS CloudFormation, Terraform, and Cloud Development Kit (CDK), and was constructed through a synthetic generation pipeline with static and LLM-based validation [2509.05303].

## 1. Dataset scope and internal structure

Multi-IaC-Bench consists of 337 CloudFormation triplets, 171 Terraform triplets, and 191 CDK triplets, with the CDK portion divided into 96 Python examples and 95 TypeScript examples [2509.05303]. Each data point is a 3-element tuple:

1. `initial` — the original IaC template, which may be empty and is represented as YAML/JSON for CloudFormation, HCL for Terraform, or a code repository for CDK  
2. `utterance` — a synthetic natural-language “customer request” to add or modify one or more resources  
3. `expected` — the updated IaC template that exactly implements the change described in the utterance

This triplet design supports both generation from scratch and mutation of existing stacks. Input templates range from an empty skeleton to complex stacks with multiple resources and security settings. The benchmark tracks the number of resources per template, measured as the number of top-level `Resource` blocks or HCL blocks. Across CloudFormation the mean is approximately 4.7 resources per template; for Terraform it is approximately 5.1; for CDK it is approximately 4.5 [2509.05303].

| Format | Triplets |
|---|---:|
| CloudFormation | 337 |
| Terraform | 171 |
| CDK | 191 |

The benchmark also reports frequent resource types across the three formats. For CloudFormation, the most common include `AWS::IAM::Role` (85), `AWS::EC2::Subnet` (71), `AWS::EC2::SecurityGroup` (71), and `AWS::Route` (32). For Terraform, common types include `aws_s3_bucket` (73), `aws_iam_role` (44), `aws_iam_role_policy` (37), and `kubernetes_config_map` (15). For CDK, common types include `AWS::EC2::Subnet` (95), `AWS::S3::Bucket` (57), `AWS::SSM::Parameter` (50), and `AWS::Budgets::Budget` (17) [2509.05303]. This distribution suggests that the benchmark is oriented toward practical cloud-stack mutations rather than narrow single-resource edits.

## 2. Synthetic generation and validation pipeline

The dataset is generated through a multi-stage synthetic data pipeline. First, source CloudFormation and Terraform templates are collected from public GitHub repositories, specifically `iac-model-evaluation`, `iac-eval`, and `aws-cloudformation-templates` [2509.05303]. These templates are then filtered by static analysis: CloudFormation templates are checked with CFN-Lint, Terraform templates with TFLint, and both are additionally checked with Checkov. Any template that fails the linter is discarded.

For each remaining template, the process repeats up to a limit \(K\) in order to obtain multiple triplets per input. An LLM, identified as Amazon Nova Pro via AWS Bedrock with temperature \(0.9\), is prompted to propose a “realistic” change not seen in prior requests, produce the corresponding natural-language utterance, and generate the updated IaC file [2509.05303]. The updated file is again subjected to static analysis; failures are discarded and retried. A second validation stage uses an LLM-based judge to verify both that the changes align with the utterance and that no unwanted edits were made. If the judge flags misalignment, the result is discarded and retried.

A stratified sample of 96 CloudFormation triplets is then converted into CDK using the AWS CDK Migrate tool, yielding 191 CDK triplets in Python and TypeScript [2509.05303]. This creates a benchmark in which one portion is directly sourced from CloudFormation and Terraform corpora, while the CDK portion is derived through conversion. A plausible implication is that CDK evaluation in this benchmark is tied not only to code-editing difficulty but also to properties induced by the CloudFormation-to-CDK migration process.

The pseudocode reported for the pipeline is:

```text
for each source_template in CFN_and_TF_corpus:
  if not passes_lint(source_template): continue
  prev_requests = ∅
  for trial in 1…MaxAttempts:
    (request, updated) = LLM_mutate(source_template, prev_requests)
    prev_requests += {request}
    if not passes_lint(updated): continue
    if not LLM_judge_aligns(source_template, request, updated): continue
    emit (source_template, request, updated)
    break
convert_some_CFNs_to_CDK()
```

## 3. Validation regime and benchmark metrics

Multi-IaC-Bench distinguishes between syntactic correctness and semantic correctness. Syntactic checks are performed using CFN-Lint for CloudFormation, TFLint for Terraform, and Checkov to ensure no formatting, schema, or security errors [2509.05303]. Semantic checks are performed by an LLM judge, which was validated against human evaluation on a 60-sample subset and achieved 91.6% agreement with Pearson \(r = 0.64\) [2509.05303].

The benchmark defines three core counts: `total_attempts`, `syntactically_valid`, and `semantically_correct`. On that basis, `ValidityRate` is defined as `syntactically_valid / total_attempts`, and `SemanticAccuracy` is defined as `semantically_correct / total_attempts` [2509.05303]. Two additional diagnostics are included: `EditDistance`, the average Levenshtein distance between generated and reference templates, and `AvgLLMCalls`, the average number of LLM invocations per test case, which accounts for retries. The benchmark also reports `LLMJudgeScore`, described as the average “alignment rating” on a Likert scale returned by the judge LLM [2509.05303].

A common misconception in IaC generation is that passing linters is equivalent to solving the task. The benchmark’s metric design explicitly rejects that equivalence. High linter pass rates capture static validity and partial compliance with schema and security tooling, but do not establish that the requested edit was performed and only that edit. The inclusion of judge-based semantic evaluation is intended to separate these failure modes.

## 4. Evaluation protocol for LLMs

The reported evaluation covers Llama 3.2 11B Instruct, DeepSeek R1, and Anthropic Sonnet 3.5 V2, with all inference calls served via AWS Bedrock and experiments run at temperature \(0.5\) [2509.05303]. The prompting protocol contains four variants.

The **Basic Prompt** provides minimal context, consisting of the initial template and user request. The **Full Prompt** adds IAM best-practice guidelines and instructs the model to keep changes minimal. **Chain-of-Thought (CoT)** encourages the model to “think step by step” through the edit. The **Retry Mechanism** re-prompts on any linter or semantic failure with error messages included [2509.05303].

The retry loop is treated as part of the evaluation methodology rather than only as a post-processing heuristic. After a failure, the model receives feedback from linter or Checkov errors and is asked to correct them. The reported effect is substantial: this iterative loop boosts static pass rates from 60–80% up to greater than 95% [2509.05303]. Within the benchmark’s framing, the number of LLM calls is therefore not incidental; it is a measured component of system behavior.

This setup makes the benchmark useful for comparing not only model quality but also prompt-policy design. A plausible implication is that Multi-IaC-Bench evaluates composite inference procedures—prompt template plus retry policy plus model—rather than a pure one-shot generative capability.

## 5. Empirical findings across CloudFormation, Terraform, and CDK

The benchmark reports selected results for each model and format in terms of Lint Pass %, Checkov Pass %, AvgLLMCalls, and Judge Score [2509.05303].

| Format / Model | Lint Pass % / Checkov Pass % | AvgLLMCalls / Judge Score |
|---|---|---|
| CFN / Llama 3.2 | 72.1% / 89.9% | 2.71 / 1.89 |
| CFN / DeepSeek R1 | 92.0% / 92.6% | 1.79 / 2.06 |
| CFN / Sonnet 3.5 V2 | 98.5% / 98.8% | 1.82 / 2.23 |
| TF / Llama 3.2 | 84.8% / 100% | 2.73 / 2.01 |
| TF / DeepSeek R1 | 98.8% / 98.8% | 1.81 / 2.12 |
| TF / Sonnet 3.5 V2 | 100% / 100% | 2.10 / 2.39 |
| CDK / Llama 3.2 | 36.7% / 98.4% | 3.72 / 1.31 |
| CDK / DeepSeek R1 | 85.3% / 85.9% | 1.60 / 1.65 |
| CDK / Sonnet 3.5 V2 | 95.8% / 96.3% | 1.59 / 1.75 |

The comparative analysis identifies Sonnet 3.5 V2 as strongest across formats, especially CloudFormation and Terraform [2509.05303]. Llama 3.2 struggles most on CDK conversion, where lint pass is low even though Checkov remains high. The benchmark characterizes CDK as the hardest format because of its more complex code structure, even with the CloudFormation-to-CDK conversion trick [2509.05303]. It also concludes that retry loops yield the largest gains, whereas CoT adds minor improvements but increases the number of calls.

The abstract summarizes the broader pattern as follows: modern LLMs can achieve high success rates, exceeding 95%, in generating syntactically valid IaC across formats, but significant challenges remain in semantic alignment and handling complex infrastructure patterns [2509.05303]. That contrast is central to the benchmark’s empirical message.

## 6. Ablation results and representative triplets

The ablation study is reported for Sonnet 3.5 V2 on CloudFormation. With the **Basic Prompt**, results are Lint 61.9%, Checkov 56.3%, Judge 2.45, and 1.00 call. With the **Full Prompt**, results are Lint 69.3%, Checkov 51.1%, Judge 2.40, and 1.00 call. Adding the **Retry Loop** yields Lint 98.5%, Checkov 98.8%, Judge 2.23, and 1.82 calls. Adding **CoT** yields Lint 96.0%, Checkov 97.7%, Judge 2.24, and 1.94 calls [2509.05303].

The stated findings are that adding best-practice text alone yields modest gains, the retry loop is critical because the static pass rate jumps by more than 30 points, and Chain-of-Thought helps slightly for CDK but is marginal for CloudFormation and Terraform [2509.05303]. These results place error-aware iteration at the center of the benchmarked workflow.

The benchmark’s example triplets are intentionally simple and illustrate the triplet formalism across formats. In CloudFormation, the original template contains an `AWS::Budgets::Budget` resource with `Amount: 100`, the utterance requests increasing the monthly cost budget from `$100` to `$1000`, and the expected template updates `Amount` to `1000`. In Terraform, the original template contains an `aws_budgets_budget` resource with `limit_amount = "100"`, the utterance requests raising the `limit_amount` to `1000 USD`, and the expected template changes that field to `"1000"`. In CDK Python, the original code constructs `budgets.CfnBudget` with `amount=100`, the utterance requests modifying the monthly cost budget so that the amount is `1000 USD instead of 100`, and the expected code changes the value to `amount=1000` [2509.05303].

These examples clarify an important property of the benchmark: the task is not generic code completion, but controlled infrastructure mutation under a natural-language edit specification. This suggests a close relationship to edit-based code generation benchmarks, but with domain-specific static analyzers and security checks integrated directly into the evaluation loop.

## 7. Position within AI-assisted infrastructure management

Multi-IaC-Bench is presented as a benchmark intended to facilitate further research in AI-assisted infrastructure management and to establish standardized evaluation metrics for this domain [2509.05303]. Its contribution lies in combining three dimensions that are often separated: cross-format coverage, mutation-oriented triplets, and a validation stack that includes both conventional static tooling and semantic judgment by an LLM.

Within that framing, the benchmark operationalizes IaC generation as a sequence of constrained transformations over machine-readable infrastructure definitions. The use of CloudFormation, Terraform, and CDK places declarative and code-centric IaC styles in a single evaluation space. The inclusion of IAM best-practice guidance, Checkov-based security validation, and minimal-change prompting reflects the fact that IaC correctness in practice is not reducible to parsing or compilation alone.

The benchmark also highlights a persistent tension in LLM evaluation for software artifacts. Strong static validity rates, including results above 95%, do not eliminate semantic misalignment or difficulty with complex infrastructure patterns [2509.05303]. A plausible implication is that future work on IaC assistants will need to improve not only format-specific syntax handling but also edit localization, preservation of untouched resources, and reliable satisfaction of natural-language change constraints across heterogeneous IaC ecosystems.

Source: https://www.emergentmind.com/topics/multi-iac-bench