---
title: 'Siemens AG Longitudinal Study: DevSecOps Compliance'
url: https://www.emergentmind.com/topics/longitudinal-study-at-siemens-ag
type: topic
---

# Siemens AG Longitudinal Study: DevSecOps Compliance

Siemens AG conducted a five-year longitudinal study (2019–2023) to address the integration of security compliance within rapid, agile DevOps practices, focusing on industrial automation products subject to certifiable standards such as IEC 62443-4-1. The study resulted in the conception, validation, and application of RefA—an artefact-centric, prescriptive DevSecOps framework—coupled with corresponding assessment (RefA-AP) and reporting (RefA-AR) protocols. The primary aim was to enable continuous, team-driven security compliance assessments aligned to both agile development and regulatory requirements, facilitating knowledge transfer and actionable process improvements for both security and non-security professionals [2512.14453].

## 1. Study Motivation, Scope, and Structure

The principal motivation was to resolve the operational tension between Siemens AG’s adoption of agile/DevOps methodologies and the necessitated adherence to IEC 62443-4-1 security controls in the engineering of software-intensive products, particularly for critical infrastructure domains. Practitioners faced barriers in embedding and persistently assessing security-compliance artefacts within non-linear, high-frequency DevOps workflows.

The program comprised ten empirical studies, engaging 102 practitioners across twenty-six products (twelve at Siemens). The sequence consisted of:

- **Inception (Problem Analysis):** Workshops and roundtables identified and prioritized 15 major security-in-DevOps challenges (e.g., transitioning from gate-based assurance to risk-based, continuous security and enabling distributed security ownership).
- **Validation and Construction (Studies 3–7):** Assessment and refinement of artefact mapping, activity models, questionnaires, and reporting templates, culminating in RefA and its assessment protocol.
- **Initial Adoption and Evaluation (Studies 8–10):** Pilot training, industry deployment, and expert appraisal established feasibility and benefit of the approach.

Empirical data collection featured semi-structured interviews, focus groups, artefact/code audits, structured questionnaires, and quantitative/qualitative feedback cycles.

## 2. RefA Framework: Structure and Methodology

RefA is an artefact- and phase-based model that prescribes a DevSecOps lifecycle mapped to the canonical DevOps phases (Plan, Code, Build, Test, Release, Deploy, Operate, Monitor). Its dual-perspective approach is defined as follows:

- **Artefact View:** Employs a UML-derived notation to specify explicit security-related input/output artefacts for each phase, such as Secure Coding Standards, Software Bill of Materials (SBOM), Static Analysis Reports, and Incident Response Plans.
- **Practice Areas View:** Aggregates artefacts into secure-engineering domains and overlays these on top of DevOps stages, distinguishing between software engineering and security-centric activities.

Each IEC 62443-4-1 control is cross-mapped to tangible DevOps artefacts, with additional augmentation from industry norms (e.g., infrastructure-as-code hardening, continuous feedback loops). The artefact-centric mapping aims to simplify the identification and demonstration of compliance evidence, facilitating knowledge transfer to engineering personnel.

### Assessment Protocols

- **Assessment Questionnaire (RefA-AQ):** Comprises 58 closed questions aligned to DevOps phases and grouped by relevant practice areas, each associating answers to mandatory artefact evidence and including domain-specific assessor guidance.
- **Assessment Workflow (RefA-AW):** Designed to be compatible with sprint cycles; six sessions (four 2-hour for paired DevOps phases, two 1.5-hour summary sessions) collecting evidence and prioritizing improvement targets.
- **Assessment Reporting (RefA-AR):**
  - **Improvement Roadmap (RefA-IR):** Populates a quadrant matrix (Impact vs. Effort) with non-compliance findings, classifying them as Quick Wins, Major Projects, Fill-ins, or Thankless Tasks.
  - **Maturity Report (RefA-MR):** Utilizes radar (Kiviat) charts to visualize current vs. target maturity per practice area (0–5 scale, spanning Sit→Fly stages).

#### Compliance Metric

Compliance is quantified as:

$$
C = \frac{N_{\mathrm{compliant}}}{N_{\mathrm{total}}}
$$

where $N_{\mathrm{compliant}}$ is the number of “compliant” RefA-AQ responses and $N_{\mathrm{total}}=58$.

## 3. Empirical Outcomes

### Quantitative Results

- **Assessment Efficiency:** Complete assessments required an average of 5 sessions (±1), aggregating to 6.25 hours—operationally compatible with a two-week sprint cadence.
- **Self-assessment Accuracy:** The mean absolute difference between security specialist and non-security maturity scoring was low ($\overline{\Delta}\approx 0.29$–$0.35$), evidencing convergence and indicating non-security engineers’ ability to self-evaluate with high fidelity.
- **Practitioner Perception:** Median ratings from RefA-AP users (n=7, Study 8) on structuredness, goal orientation, completeness, and overall suitability were 7/8; simplicity (median 6), perceived knowledge background required (median 4).
- **Adoption Intent:** 8/11 practitioners expressed intent to use RefA post-training; pilot teams incorporated roadmap actions into standard backlog workflows without external prompting.

### Qualitative and Contextual Findings

**Benefits**
- Artefact-phase alignment exposes security gaps rapidly and accommodates phase-specific improvement targeting.
- Promotes left-shift of security activities and supports modular, continuous assessment.
- The artefact-centric schema aids knowledge transfer to engineers without deep security expertise.

**Barriers**
- Initial complexity and density of technical terminology increased the entry threshold.
- Artefact catalogue required additional extensibility and detailed documentation.
- Non-security team members occasionally hesitated to define target maturity scores, highlighting a need for expert facilitation.

**Facilitators**
- Modular questionnaire design enabled phase or domain scoping.
- Visual and structured reporting facilitated stakeholder engagement and concrete action planning.

## 4. Lessons for Practice

### Knowledge Transfer and Process Integration

Mapping artefacts directly to DevOps pipelines enhances transparency: engineers can associate compliance directly with tangible, inspectable assets rather than abstract activities. Cross-functional teams demonstrated the ability to self-administer compliance assessments with minimal reliance on security specialists, supporting the goal of distributed ownership.

### Operationalization and Best Practices

- **Process Integration:** Compliance and maturity assessments are treated as deliberate sprint backlog tasks; improvement is approached iteratively, reinforcing agile principles.
- **Automation and Risk Orientation:** Security checks (SAST, SCA, container hardening, SBOM generation) are integrated and automated within pipelines; a “risk-based” paradigm supplants rigid, gate-dependent approaches.
- **Outcome-focused Adoption:** Quick Wins and Major Projects are prioritized using the Improvement Roadmap, enabling early returns and targeted investment.
- **Lean Security Culture:** Iterative learning and acceptance of non-compliance as process feedback parallel agile retrospectives.

### Sector-specific Considerations

RefA extends beyond the explicit remit of IEC 62443-4-1 by integrating operational best practices from NIST SSDF and CIS Benchmarks, addressing domain-specific concerns such as cloud-native configurations and third-party trust relationships. In industrial automation and control system (IACS) environments, confidentiality restrictions necessitated anonymization and aggregation to manage sensitive data.

## 5. Comparative Framework Analysis and External Appraisal

RefA’s prescriptive structuring, artefact focus, and assessment protocol were compared by senior IEC 62443-4-1 assessors to incumbent frameworks. Its distinguishing characteristics included:

- Artefact-level mapping facilitating transparency and direct traceability.
- Sprint-compatible assessment protocol reducing compliance friction and supporting continuous improvement.
- Visual reporting and phased modularity empowering both technical and non-technical stakeholders.

Empirical validation across academic, industrial, and expert evaluation contexts confirmed feasibility and utility, with particular strength in facilitating non-expert involvement and integrating compliance as a lived component of the DevOps process.

## 6. Future Development Trajectories

The anticipated future directions are:

- **Assessment Automation:** Develop pipelines to update RefA artefact catalogs dynamically and aggregate assessment outcomes across multiple teams for broader-scale analytics.
- **Cross-standard Expansion:** Systematic mapping of RefA artefacts to alternative compliance norms, such as NIST SP 800-218 and CSA Cloud Controls, to generalize RefA for hybrid cloud and OT environments.
- **Risk Management Integration:** Incorporate explicit risk management artefacts and workflows within each DevOps phase (e.g., automatic threat backlog generation).
- **Collaboration and Tooling:** Analyze and support cross-team collaboration and toolchain integration (including GitOps pipelines) to further minimize compliance overhead and improve efficiency [2512.14453].

This approach consolidates artefact-driven, modular security compliance assessment and process improvement within the realities of modern software engineering, while maintaining certifiable regulatory alignment and supporting heterogeneous practitioner backgrounds.

Source: https://www.emergentmind.com/topics/longitudinal-study-at-siemens-ag