Enhancing Security Control Production With Generative AI

Abstract: Security controls are mechanisms or policies designed for cloud based services to reduce risk, protect information, and ensure compliance with security regulations. The development of security controls is traditionally a labor-intensive and time-consuming process. This paper explores the use of Generative AI to accelerate the generation of security controls. We specifically focus on generating Gherkin codes which are the domain-specific language used to define the behavior of security controls in a structured and understandable format. By leveraging LLMs and in-context learning, we propose a structured framework that reduces the time required for developing security controls from 2-3 days to less than one minute. Our approach integrates detailed task descriptions, step-by-step instructions, and retrieval-augmented generation to enhance the accuracy and efficiency of the generated Gherkin code. Initial evaluations on AWS cloud services demonstrate promising results, indicating that GenAI can effectively streamline the security control development process, thus providing a robust and dynamic safeguard for cloud-based infrastructures.

• The paper introduces a GenAI framework that automates Gherkin code generation for security controls, aiming to reduce development time from days to under a minute.
• The proposed methodology leverages LLMs with in-context learning, retrieval-augmented generation, and detailed prompting to interpret documentation and improve the quality of generated security control code.
• Evaluation using human experts and a structured rubric showed that generated Gherkin scripts for key control types like Encryption and Logging achieved average scores (3.02, 3.05) above the acceptance threshold (2.5).

This paper introduces a GenAI-based framework designed to accelerate the production of security controls, specifically focusing on the generation of Gherkin code. The authors address the challenges associated with the traditional, labor-intensive processes involved in developing security controls for cloud-based services.

The authors highlight the time-consuming nature of traditional security control development, which involves research, threat modeling, and custom code development. To address these inefficiencies, the paper proposes a structured framework leveraging LLMs and in-context learning to automate Gherkin code generation. The goal is to reduce the development time from 2-3 days to under a minute. The approach uses detailed task descriptions, step-by-step instructions, and retrieval-augmented generation to improve the accuracy and efficiency of the generated code.

The development of security controls is described as a multi-stage process:

• Identifying the service, resource, and control type.
• Writing Gherkin scripts.
• Reviewing scripts.
• Developing executable code.
• Deployment and testing.

The paper identifies two key challenges in automating Gherkin code generation:

• Accurate interpretation of complex service documentation.
• Ensuring the quality and reliability of generated Gherkin files.

To address these challenges, the proposed framework incorporates the following:

• A comprehensive task description for the LLM, broken down into detailed steps.
• Examples of existing security controls and associated Gherkin scripts.
• A final query to guide the LLM in generating the desired security control.

The authors leverage in-context learning with retrieval-augmented generation. The LLM is provided with a detailed task description and chain-of-thought reasoning, combined with examples of existing security controls. Public APIs are used to gather information about AWS security best practices, which is then integrated into the prompt.

The paper focuses on nine critical security control types:

1. Encryption of Data at Rest
2. Encryption of Data in Transit
3. Tagging
4. Resources Running on a Supported Version
5. Backup Enabled
6. Multi-AZ Deployment
7. Inbound IP Connection Control
8. Resource Accessibility
9. Audit Logging Enabled

An agent-based system is introduced to automatically determine applicable control types for a given service and resource, using detailed descriptions of each security control type.

The evaluation methodology employs a human-in-the-loop approach, using expert human judgment to assess the quality of generated Gherkin scripts against a structured rubric. The rubric includes criteria for scenario evaluation (S1-S5) and rule evaluation (R1-R2).

The overall score for a Gherkin script is calculated using the formula:

$score = (S1+S2+S3+S4+S5) \times (R1+R2) / 2$

Where:

• $S1$ = The number of scenarios recorded is correct.
• $S2$ = The field specified in the scenario exists.
• $S3$ = The resulting compliance status is possible.
• $S4$ = The configuration of the resource specified by the scenario is possible.
• $S5$ = The conclusion of the scenario is correct.
• $R1$ = The rule name correctly describes the control specified by the collection of scenarios.
• $R2$ = The description correctly describes the control specified by the collection of scenarios.

The acceptance threshold is $\ge 2.5$ for the generated Gherkins.

Experiments were conducted in collaboration with AWS domain experts, using data from available AWS services and resources, and the Claude-3-Sonnet LLM hosted on AWS Bedrock.

Histograms of evaluated Gherkins show that the majority of generated Gherkins for Encryption and Logging types fall within the acceptable range (score $\ge 2.5$). The distribution patterns are similar, with most scores clustering around 3.

The average scores for Encryption of Data-at-rest and Logging were also measured. For each category, security engineers reviewed ten randomly selected generated Gherkin scripts. The scenario scores were 4.19 and 4.07, respectively, while the rule scores were 0.72 and 0.75, respectively. The final scores were 3.02 and 3.05, respectively.