Siemens AG Longitudinal Study: DevSecOps Compliance

• The paper presents RefA, an artefact-centric framework that maps IEC 62443-4-1 controls to DevOps phases to streamline security compliance.
• The study employed empirical methods, including interviews, surveys, and artefact audits across 10 studies and 102 practitioners for comprehensive validation.
• Outcomes indicate enhanced collaboration, precise compliance measurement, and iterative process improvements integrating security seamlessly into agile workflows.

Siemens AG conducted a five-year longitudinal study (2019–2023) to address the integration of security compliance within rapid, agile DevOps practices, focusing on industrial automation products subject to certifiable standards such as IEC 62443-4-1. The study resulted in the conception, validation, and application of RefA—an artefact-centric, prescriptive DevSecOps framework—coupled with corresponding assessment (RefA-AP) and reporting (RefA-AR) protocols. The primary aim was to enable continuous, team-driven security compliance assessments aligned to both agile development and regulatory requirements, facilitating knowledge transfer and actionable process improvements for both security and non-security professionals (Moyón et al., 16 Dec 2025).

1. Study Motivation, Scope, and Structure

The principal motivation was to resolve the operational tension between Siemens AG’s adoption of agile/DevOps methodologies and the necessitated adherence to IEC 62443-4-1 security controls in the engineering of software-intensive products, particularly for critical infrastructure domains. Practitioners faced barriers in embedding and persistently assessing security-compliance artefacts within non-linear, high-frequency DevOps workflows.

The program comprised ten empirical studies, engaging 102 practitioners across twenty-six products (twelve at Siemens). The sequence consisted of:

• Inception (Problem Analysis): Workshops and roundtables identified and prioritized 15 major security-in-DevOps challenges (e.g., transitioning from gate-based assurance to risk-based, continuous security and enabling distributed security ownership).
• Validation and Construction (Studies 3–7): Assessment and refinement of artefact mapping, activity models, questionnaires, and reporting templates, culminating in RefA and its assessment protocol.
• Initial Adoption and Evaluation (Studies 8–10): Pilot training, industry deployment, and expert appraisal established feasibility and benefit of the approach.

Empirical data collection featured semi-structured interviews, focus groups, artefact/code audits, structured questionnaires, and quantitative/qualitative feedback cycles.

2. RefA Framework: Structure and Methodology

RefA is an artefact- and phase-based model that prescribes a DevSecOps lifecycle mapped to the canonical DevOps phases (Plan, Code, Build, Test, Release, Deploy, Operate, Monitor). Its dual-perspective approach is defined as follows:

• Artefact View: Employs a UML-derived notation to specify explicit security-related input/output artefacts for each phase, such as Secure Coding Standards, Software Bill of Materials (SBOM), Static Analysis Reports, and Incident Response Plans.
• Practice Areas View: Aggregates artefacts into secure-engineering domains and overlays these on top of DevOps stages, distinguishing between software engineering and security-centric activities.

Each IEC 62443-4-1 control is cross-mapped to tangible DevOps artefacts, with additional augmentation from industry norms (e.g., infrastructure-as-code hardening, continuous feedback loops). The artefact-centric mapping aims to simplify the identification and demonstration of compliance evidence, facilitating knowledge transfer to engineering personnel.

Assessment Protocols

• Assessment Questionnaire (RefA-AQ): Comprises 58 closed questions aligned to DevOps phases and grouped by relevant practice areas, each associating answers to mandatory artefact evidence and including domain-specific assessor guidance.
• Assessment Workflow (RefA-AW): Designed to be compatible with sprint cycles; six sessions (four 2-hour for paired DevOps phases, two 1.5-hour summary sessions) collecting evidence and prioritizing improvement targets.
• Assessment Reporting (RefA-AR):
  • Improvement Roadmap (RefA-IR): Populates a quadrant matrix (Impact vs. Effort) with non-compliance findings, classifying them as Quick Wins, Major Projects, Fill-ins, or Thankless Tasks.
  • Maturity Report (RefA-MR): Utilizes radar (Kiviat) charts to visualize current vs. target maturity per practice area (0–5 scale, spanning Sit→Fly stages).

Compliance Metric

Compliance is quantified as:

$$C = \frac{N_{\mathrm{compliant}}}{N_{\mathrm{total}}}$$

where $N_{\mathrm{compliant}}$ is the number of “compliant” RefA-AQ responses and $N_{\mathrm{total}}=58$.

3. Empirical Outcomes

Quantitative Results

• Assessment Efficiency: Complete assessments required an average of 5 sessions (±1), aggregating to 6.25 hours—operationally compatible with a two-week sprint cadence.
• Self-assessment Accuracy: The mean absolute difference between security specialist and non-security maturity scoring was low ($\overline{\Delta}\approx 0.29$–$0.35$), evidencing convergence and indicating non-security engineers’ ability to self-evaluate with high fidelity.
• Practitioner Perception: Median ratings from RefA-AP users (n=7, Study 8) on structuredness, goal orientation, completeness, and overall suitability were 7/8; simplicity (median 6), perceived knowledge background required (median 4).
• Adoption Intent: 8/11 practitioners expressed intent to use RefA post-training; pilot teams incorporated roadmap actions into standard backlog workflows without external prompting.

Qualitative and Contextual Findings

Benefits

• Artefact-phase alignment exposes security gaps rapidly and accommodates phase-specific improvement targeting.
• Promotes left-shift of security activities and supports modular, continuous assessment.
• The artefact-centric schema aids knowledge transfer to engineers without deep security expertise.

Barriers

• Initial complexity and density of technical terminology increased the entry threshold.
• Artefact catalogue required additional extensibility and detailed documentation.
• Non-security team members occasionally hesitated to define target maturity scores, highlighting a need for expert facilitation.

Facilitators

• Modular questionnaire design enabled phase or domain scoping.
• Visual and structured reporting facilitated stakeholder engagement and concrete action planning.

4. Lessons for Practice

Knowledge Transfer and Process Integration

Mapping artefacts directly to DevOps pipelines enhances transparency: engineers can associate compliance directly with tangible, inspectable assets rather than abstract activities. Cross-functional teams demonstrated the ability to self-administer compliance assessments with minimal reliance on security specialists, supporting the goal of distributed ownership.

Operationalization and Best Practices

• Process Integration: Compliance and maturity assessments are treated as deliberate sprint backlog tasks; improvement is approached iteratively, reinforcing agile principles.
• Automation and Risk Orientation: Security checks (SAST, SCA, container hardening, SBOM generation) are integrated and automated within pipelines; a “risk-based” paradigm supplants rigid, gate-dependent approaches.
• Outcome-focused Adoption: Quick Wins and Major Projects are prioritized using the Improvement Roadmap, enabling early returns and targeted investment.
• Lean Security Culture: Iterative learning and acceptance of non-compliance as process feedback parallel agile retrospectives.

Sector-specific Considerations

RefA extends beyond the explicit remit of IEC 62443-4-1 by integrating operational best practices from NIST SSDF and CIS Benchmarks, addressing domain-specific concerns such as cloud-native configurations and third-party trust relationships. In industrial automation and control system (IACS) environments, confidentiality restrictions necessitated anonymization and aggregation to manage sensitive data.

5. Comparative Framework Analysis and External Appraisal

RefA’s prescriptive structuring, artefact focus, and assessment protocol were compared by senior IEC 62443-4-1 assessors to incumbent frameworks. Its distinguishing characteristics included:

• Artefact-level mapping facilitating transparency and direct traceability.
• Sprint-compatible assessment protocol reducing compliance friction and supporting continuous improvement.
• Visual reporting and phased modularity empowering both technical and non-technical stakeholders.

Empirical validation across academic, industrial, and expert evaluation contexts confirmed feasibility and utility, with particular strength in facilitating non-expert involvement and integrating compliance as a lived component of the DevOps process.

6. Future Development Trajectories

The anticipated future directions are:

• Assessment Automation: Develop pipelines to update RefA artefact catalogs dynamically and aggregate assessment outcomes across multiple teams for broader-scale analytics.
• Cross-standard Expansion: Systematic mapping of RefA artefacts to alternative compliance norms, such as NIST SP 800-218 and CSA Cloud Controls, to generalize RefA for hybrid cloud and OT environments.
• Risk Management Integration: Incorporate explicit risk management artefacts and workflows within each DevOps phase (e.g., automatic threat backlog generation).
• Collaboration and Tooling: Analyze and support cross-team collaboration and toolchain integration (including GitOps pipelines) to further minimize compliance overhead and improve efficiency (Moyón et al., 16 Dec 2025).

This approach consolidates artefact-driven, modular security compliance assessment and process improvement within the realities of modern software engineering, while maintaining certifiable regulatory alignment and supporting heterogeneous practitioner backgrounds.