Instructions-as-Code: An Executable Paradigm

• Instructions-as-Code is the practice of converting guidelines and directives into structured, executable artifacts that can be versioned, validated, and enforced.
• It operationalizes DevOps and infrastructure automation by replacing ambiguous textual runbooks with precise, replicable code and policy files.
• Recent studies demonstrate its impact on security, clarity, and error reduction by measuring compliance, robustness, and performance in various code generation benchmarks.

Searching arXiv for papers on “Instructions-as-Code” and closely related work to ground the article in current literature. Instructions-as-Code is the treatment of operational procedures, software-development requirements, agent directives, and policy rules as structured or executable artifacts rather than as passive prose. In this formulation, the practical meaning of an instruction emerges when it is encoded in forms such as Terraform HCL, Jenkins Groovy pipelines, shell and Python scripts, pseudo-code prompts, repository instruction files, or Cedar policies, so that it can be versioned, executed, checked, or enforced (Saxena et al., 20 Mar 2025, Mondl et al., 25 Jun 2026, Arabat et al., 11 Jun 2026). Taken together, these works suggest a common shift: instructions cease to be merely advisory text and become artifacts with execution semantics, validation paths, and measurable failure modes.

1. Conceptual foundations and definitional scope

A recurrent starting point is that many operational and software-engineering practices are conceptually clear yet operationally ambiguous. In the DevOps setting, DevOps is described as a culture and philosophy using cross-functional teams to build, test, and release software faster and more reliably via automation, but the definition often “does not clarify how to implement DevOps.” The proposed remedy is to make DevOps concrete through automated pipelines, Infrastructure as Code, and Pipeline as Code, so that desired behaviors such as build, test, deploy, and monitor are expressed as code artifacts (Saxena et al., 20 Mar 2025). In the repository-instruction-file literature, the same logic is extended to files such as .github/copilot-instructions.md, AGENTS.md, CLAUDE.md, and .cursorrules, which are treated as persistent, versioned artifacts that guide AI agents and whose development should itself be considered a software engineering activity (Arabat et al., 11 Jun 2026).

A second conceptual line treats natural-language specifications as latent programs that must be compiled into executable behavior. BigCodeBench is organized around function-level tasks where the model receives imports, a signature, and either a detailed docstring or a shorter instruction, and must produce Python code that satisfies a high-coverage test suite; the docstring or instruction therefore acts as an executable specification in natural language (Zhuo et al., 2024). ClarEval makes the complementary point that real instructions are often malformed specifications: it formalizes missing goal, missing premises, and ambiguous terminology as distinct ambiguity types, and evaluates whether an agent can refine an ambiguous instruction through dialogue before generating code (Li et al., 27 Feb 2026).

A third line generalizes the idea from software construction to governance and safety. Autoformalization into Cedar policies treats system prompts, MCP tool definitions, and natural-language policy documents as source material for formally enforced policy-as-code; ContextCov treats agent instruction files as a high-level policy language from which executable checks are synthesized across static AST analysis, runtime shell interception, and architectural validation (Mondl et al., 25 Jun 2026, Sharma, 28 Feb 2026). This suggests that Instructions-as-Code is not a single language or framework, but a family resemblance across artifacts whose defining property is that human intent is rewritten into forms that admit execution, validation, or enforcement.

2. Operationalization in DevOps and infrastructure automation

In DevOps automation, the concept is instantiated directly through IaC and Pipeline as Code. The pipeline described in “DevOps Automation Pipeline Deployment with IaC (Infrastructure as Code)” provisions an AWS EC2 instance, installs OpenJDK and Jenkins, starts Jenkins, and configures a web server using Terraform and HCL rather than manual shell access. The paper notes that the Jenkins server IP and initial password can appear as outputs of terraform apply, turning a multi-step administrative runbook into a single command governed by configuration files (Saxena et al., 20 Mar 2025). The same pipeline is then expressed in Jenkins Groovy, with jobs for code pull, build/modify, automated testing on a development server, necessary modification or wait, deployment to production, and monitoring, triggered by GIT SCM polling and augmented by optional interactive approval steps.

The significance of this formulation lies in repeatability and disambiguation. The paper states that “the whole infrastructure can be replicated anywhere, anytime because all its configuration can be managed using code,” and that pipeline-as-code “makes it easy to replicate the pipeline really fast if needed in the future” while enabling “better collaboration between different teams” (Saxena et al., 20 Mar 2025). A concrete deployment log shows a Jenkins job invoking a shell script to copy any.html, index.html, and Jenkinsfile into a production volume, replacing an informal operational instruction such as “copy the application files into the production volume” with an exact command sequence.

Security-oriented IaC work pushes the same logic one level higher by treating security guidance as instructions over instructions. GenSIaC studies whether LLMs can recognize and prevent major IaC weaknesses such as hard-coded secrets, empty passwords, unrestricted IP addresses, HTTP without TLS, weak cryptography, and no integrity check, and then introduces an instruction-fine-tuning dataset that teaches models to annotate or avoid such weaknesses in Ansible, Chef, and Puppet (Li et al., 15 Nov 2025). In the reported evaluation, CodeLlama-13B improves from an F1-score of 0.303 to 0.858 on code inspection after GenSIaC fine-tuning, while syntax validity reaches 1.000 in both low- and high-detail settings. Here the instruction is not only “generate infrastructure” but also “apply a security policy while generating infrastructure.”

3. Natural-language specifications as code-generation contracts

Benchmarking work has made the executable-specification interpretation explicit. BigCodeBench contains 1,140 function-level programming tasks, 139 libraries, 723 distinct function calls across 7 domains, an average of 5.6 unit tests per task, and approximately 99% branch coverage on the ground-truth solution. Its evaluation shows that even strong models remain limited: GPT-4o reaches 61.1% Pass@1 on the docstring-based version and 51.1% on the instruction-oriented version, whereas human performance on a sample is approximately 97% (Zhuo et al., 2024). The shortfall is not only raw coding difficulty; it is the inability to treat the instruction as a strict contract governing tool use, edge cases, and multi-library composition.

CodeIF formalizes this contract view further by decomposing tasks into atomic, machine-checkable constraints. It contains 1,200 tasks across Java, Python, Go, and C++, with 50 atomic constraint types grouped into 8 categories, and evaluates models using Completely Satisfaction Rate, Soft Satisfaction Rate, Rigorous Satisfaction Rate, and Consistent Continuity Satisfaction Rate (Yan et al., 26 Feb 2025). In this formulation, an instruction list acts like a structured specification whose clauses can have dependencies. The reported results show that even top models satisfy only a minority of hard tasks perfectly: Claude-3.5-Sonnet-20241022 achieves a CSR of 0.362 on Hard and 0.444 on Full, while DeepSeek-V3 leads SSR and RSR rather than perfect compliance (Yan et al., 26 Feb 2025). The benchmark therefore measures not just whether code works, but whether the instruction set has been obeyed as a bundle of constraints.

Several works study how code-shaped instruction formats alter this contract-following behavior. “Prompting with Pseudo-Code Instructions” manually constructs pseudo-code prompts for 132 tasks from Super-NaturalInstructions and reports average absolute gains of 7–16 points in F1 for classification tasks and relative improvements of 12–38% in aggregate ROUGE-L across all tasks when pseudo-code instructions replace natural-language prompts (Mishra et al., 2023). IFIM extends fill-in-the-middle training by adding an explicit instruction segment to the input; on HumanEval-infilling, Pass@1 rises from 84.6% to 93.6%, and the improvement is reported without compromising the models’ original FIM performance when no instructions are provided (Sun et al., 29 Sep 2025). COCO, by contrast, stresses robustness: it concretizes natural-language instructions with code features and finds 466.66% and 104.02% more robustness inconsistencies than two text-to-text baselines, while fine-tuning on concretized instructions reduces inconsistencies by 18.35% to 53.91% (Yan et al., 2023). “Can It Edit?” adds the transformation perspective, modeling instructional code editing as $M(c, I) \rightarrow c'$ and showing that fine-tuning open Code LLMs on permissively licensed edit datasets narrows the gap to closed models on a 210-task benchmark (Cassano et al., 2023).

Taken together, these results suggest that Instructions-as-Code in code generation has two distinct meanings. Factual benchmark design treats instructions as evaluable contracts; representational work rewrites them into pseudo-code, concretized templates, or instruction-aware FIM formats so that models encounter them in a more code-like regime.

4. Ambiguity, clarification, and robustness

ClarEval addresses the case where the instruction itself is ill-formed. It constructs 2,250 ambiguous instances from 750 base tasks by injecting three ambiguity types—Missing Goal, Missing Premises, and Ambiguous Terminology—and then evaluates 11 agents using Key Question Coverage, Premise Identification Rate, Missing Premises Recall, Average Turns to Clarify, and Efficiency-Adjusted Recall (Li et al., 27 Feb 2026). The empirical gap is large: in a GPT-4o ablation, clarified tasks achieve Pass@1 of approximately 89.02%, whereas the ambiguous baseline is approximately 8.94% (Li et al., 27 Feb 2026). This is a direct demonstration that instruction execution presupposes instruction well-formedness; without a clarification phase, even strong models fail catastrophically.

The benchmark also shows that clarification quality is not reducible to raw coding ability. GPT5-Coder attains the best single-turn KQC of 0.867 and PIR of 0.661, and in multi-turn settings the best MPR of 0.951 with the lowest ATC of 1.493, while Claude-Opus 4.1 leads multi-turn KQC at 0.754 (Li et al., 27 Feb 2026). Ambiguous terminology is the hardest class, and the case study around “organize the list of items based on the 'id'” shows the difference between targeted clarification, fragmented questioning, and silent assumption. In an Instructions-as-Code frame, this corresponds to type-checking the specification before execution.

RoCoIns studies robustness under textual adversarial perturbations and reaches a related conclusion from a different direction. It rewrites natural-language classification instructions into pseudo-code or code-like classes and combines this with an adversarial-context few-shot method. On eight robustness datasets, the method consistently outperforms natural-language instructions; with gpt-3.5-turbo, adversarial-set accuracy improves by 5.68% and Attack Success Rate is reduced by 5.66 points (Zhang et al., 2024). The reported mechanism is not formal verification, but reduced ambiguity and stronger structural cues: class definitions, typed parameters, and explicit print("positive") or entailment labels constrain interpretation more tightly than prose.

A plausible implication is that ambiguity handling and robustness are complementary aspects of the same problem. ClarEval exposes the cost of under-specified instructions; RoCoIns shows that even when tasks are specified, code-like formulations can reduce brittleness under perturbation.

5. Repository instruction files, executable guardrails, and policy enforcement

Repository-local instruction files are a prominent practical form of Instructions-as-Code. The large-scale study of agentic pull requests analyzes 15,549 agentic PRs from 148 projects in the AIDev dataset and compares project outcomes before and after the creation of instruction files such as .github/copilot-instructions.md, AGENTS.md, CLAUDE.md, and .cursorrules (Arabat et al., 11 Jun 2026). The results are explicitly mixed: with instruction files, 27.7% of projects increased their merge rate by at least 20%, while 26.35% decreased it by at least 20%; projects with at least a 20% merge-rate increase had median instruction-file length of 976 words, compared with 569 words for those with at least a 20% decrease, and they also exhibited more H1, H2, and H3 headers (Arabat et al., 11 Jun 2026). The study therefore rejects the simple claim that “more instructions” are automatically beneficial and instead treats instruction-file authoring as an engineering activity requiring standards, structure, and quality metrics.

ContextCov moves from passive instruction files to active enforcement. On 723 open-source repositories with agent instruction files, it parses Markdown into a header-aware AST, refines leaf constraints with an LLM, routes them into process, source, architectural deterministic, and architectural semantic domains, and synthesizes 46,316 checks; among 34,374 executable checks, 34,373 parse successfully, yielding 99.997% syntax validity (Sharma, 28 Feb 2026). When run against repository states, the framework reports more than 500,000 violations and finds that 81% of repositories have at least one violation. The architecture is explicitly multi-domain: Tree-sitter-based AST analysis for source constraints, PATH-injected shell shims for process constraints, graph validators for architectural constraints, and LLM-as-judge warnings for semantic architectural constraints.

Autoformalization into Cedar takes the enforcement idea into agent safety. The “Verification Sandwich” converts system instructions, MCP tool definitions, and natural-language policy documents into Cedar policies via a grounding layer, an LLM policy generator, and hard and soft critics, then deploys the verified policies in an external, fail-closed engine (Mondl et al., 25 Jun 2026). On MedAgentBench, the system starts from the same 88-rule natural-language policy used by Hong et al., while Hong et al.’s prior symbolic guardrails cover 23 of those rules. In replay evaluation, Cedar Block Rate exceeds Hong et al.’s Unsafe rate in all overall conditions, and for trajectories with POST write attempts the reported block rate is 94.0% to 100.0%, reaching 100.0% in all adversarial POST conditions (Mondl et al., 25 Jun 2026). Here Instructions-as-Code is no longer only guidance for generation; it becomes deterministically enforced authorization logic.

6. Quality, defects, security, and enduring limitations

Empirical studies of IaC defects and anti-patterns show that turning instructions into code changes where failures occur. Rahman and colleagues’ study of Puppet scripts across Mirantis, Mozilla, OpenStack, and Wikimedia Commons reports that 49.3%, 36.5%, 57.6%, and 62.7% of IaC defects are syntax and configuration-related, and that such defects are more prevalent in IaC than in previously studied non-IaC software (Rahman et al., 2018). The later mixed-methods study on 2,138 open-source IaC scripts and a survey with 51 practitioners identifies five development anti-patterns—“boss is not around”, “many cooks spoil”, “minors are spoiler”, “silos”, and “unfocused contribution”—as recurring development activities related to defective IaC scripts (Rahman et al., 2020). These findings place Instructions-as-Code firmly within software engineering process research: defects arise not only from syntax and schema errors in the instruction artifact, but also from ownership, collaboration structure, and contribution patterns around it.

Security-aware generation reinforces the point that executable instructions require governance beyond mere functional synthesis. GenSIaC shows that base LLMs are weak at recognizing major IaC security weaknesses during generation and inspection, and that instruction tuning on security-annotated IaC materially improves both recognition and code quality. The reported jump from 0.303 to 0.858 in F1-score on inspection for CodeLlama-13B, alongside 1.000 syntax validity in generation after tuning, indicates that security policy can itself be encoded as an instruction layer over infrastructure code (Li et al., 15 Nov 2025). A plausible implication is that quality control for Instructions-as-Code needs at least three strata: correctness of the target artifact, correctness of the instruction specification, and correctness of the socio-technical process that evolves both.

Across the literature, the open problems are strikingly consistent. BigCodeBench shows that shorter, more natural instructions still produce a substantial performance drop relative to docstring-like specifications (Zhuo et al., 2024). ClarEval shows that real collaboration requires requirement elicitation, not silent assumption (Li et al., 27 Feb 2026). The instruction-file study argues for standards, quality metrics, and tooling around instruction authoring (Arabat et al., 11 Jun 2026). ContextCov and Cedar-based autoformalization show that natural-language instructions can be compiled into active checks and formally analyzable policies, but also expose over-restriction, under-specification, statelessness, and schema dependence as practical limitations (Sharma, 28 Feb 2026, Mondl et al., 25 Jun 2026). Taken together, these works suggest that Instructions-as-Code is best understood not as a single technique, but as an engineering regime in which instructions are authored, versioned, transformed, verified, and enforced with the same seriousness traditionally reserved for source code.