Safety & Security in Critical Systems

• Safety & Security are dual, interdependent attributes that minimize accidental harms and thwart malicious threats, ensuring system dependability in IoT, AI, and industrial applications.
• Unified frameworks combine stochastic safety models with game-theoretic security metrics to quantitatively assess risks and guide effective design decisions.
• Integrated methodologies use formal models, automated reasoning, and co-engineering practices to balance trade-offs and resolve conflicts between safety controls and security measures.

Safety & Security

Safety and security are foundational, orthogonal properties of critical systems, cyber-physical infrastructure, and intelligent technologies. Safety denotes the systemic minimization of risks of unintentional accidental harm—failures, malfunctions, or hazardous outcomes without malicious intent—while security addresses the prevention of intentional, adversarial threats targeting confidentiality, integrity, or availability. In modern environments such as industrial IoT, autonomous robotics, or AI-driven systems, these domains are tightly coupled: safety incidents can be triggered by security breaches, and security controls may themselves induce safety hazards if coordinated inadequately. As complexities, attack surfaces, and operational scales escalate, rigorous models, integrated methodologies, and formal frameworks are essential to achieve robust, accountable, and comprehensively governed safety & security postures.

1. Formal Definitions and Conceptual Boundaries

System dependability incorporates reliability, availability, maintainability, and safety (RAMS), with safety defined as the ability to prevent catastrophic consequences arising from either accidental or malicious behaviors. The distinction is formalized as follows:

• Safety ("sécurité innocuité"): Concerned with unintentional faults—hardware failures, environmental perturbations, software bugs, and human errors without malice.
• Security ("sécurité confidentialité"): Concerned with intentional, malicious behaviors—intrusions, unauthorized data access or modification, sabotage.

This duality has historically led to ambiguities, particularly in multilingual or industrial contexts where a single term ("sécurité") may conflate the two. Modern practice demands explicit partitioning of system-level requirements, risk assessments, and assurance arguments along both axes. Standards such as EN 50126/50128/50129 (rail), IEC 61508 (industrial automation), and RAMSS frameworks codify this separation, recommending that risk matrices, specification documents, and validation plans distinguish requirements for accidental (safety) and malicious (security) threats down to the system architecture and lifecycle processes (Schon, 2010).

2. Threat Models, Integrated Risk Frameworks, and Formal Quantification

Unified risk assessment must address both stochastic accidental events and adversarial, game-theoretic threats. Key formalizations include:

• Classical Safety Risk (accidental): $$R_S = f(\text{Severity}, \text{Failure Probability})$$, typically aggregated as the expectation over system evolution paths.
• Security Risk (adversarial): $R_C = f(\text{Impact}, \text{Attack Likelihood})$, with likelihood often determined as a function of attacker resources, know-how, and system susceptibility.
• Unified Frameworks: Formally, given a stochastic discrete-event system $M$ (in DEVS/STDEVS form), safety risk is the expected criticality across all evolution traces:

$$R_{\text{safety}}(h) = \sum_{\tau \in L(h)} p(\tau)\,c(\tau)$$

Security risk, under adversarial strategy, employs minimax aggregation:

$$R(q) = \begin{cases} \max_{q'\in\text{Succ}(q)} R(q'), & \text{attacker's turn} \ \min_{q'\in\text{Succ}(q)} R(q'), & \text{defender's turn} \end{cases}$$

Quantitative metrics such as mission-level risk, mean time to failure (MTTF), and attack success probability are computed over hybrid models integrating both accidental and malicious event spaces (Draeger et al., 2017, Chockalingam et al., 2017).

• Risk Aggregation in CPS/Robotics: $R_{\text{total}} = f(S, C)$, e.g., $R_{\text{total}} = S + C - S\cdot C$ (assuming independence) or weighted combinations $R_{\text{total}} = \alpha S + \beta C$ as stakeholder priorities dictate (Gleirscher et al., 2020).
• Agentic/AI Systems: Emergent risk is a functional of state/action and context, $$R_t = R(S_{1:t}, A_{1:t}, C_t) = \mathbb{E}[\text{Harm} \mid S_{1:t}, A_{1:t}, C_t] \in [0,1]$$ (Ghosh et al., 27 Nov 2025).
• Differential Privacy (for privacy-security integration):

$$\forall D, D', S: \Pr[M(D)\in S] \leq e^\varepsilon\,\Pr[M(D')\in S] + \delta$$

Common taxonomies encompass:

• Confidentiality (C), Integrity (I), and Availability (A) for security.
• Accidental hazard prevention, environment protection, and fail-safe operation for safety (Fu et al., 2020, Sidhpurwala et al., 2024, Gleirscher et al., 2020).

3. Interdependency and Conflict: Safety, Security, Privacy, and Usability

Modern interconnected systems (IoT, CPS, AI) exhibit tightly coupled dependencies wherein safety, security, privacy, and usability form an inseparable socio-technical system:

• Interdependency Model: Poor usability can cascade to misconfiguration (weakened security), compromised privacy, and direct safety violations. Conversely, overly restrictive security can degrade utility, prompting unsafe workarounds or decreased compliance.
• Conflict Analysis: Formal methods identify and resolve contradictions at the requirement or architectural level. For example, in industrial control systems, the STPA-SafeSec-CDCL methodology encodes safety and security constraints as SAT clauses, detecting conflicts and learning resolution clauses to maintain robust invariants (Agbo et al., 2023).
• Trade-off Metrics: Robust security controls (e.g., encryption, isolation) may introduce safety hazards due to latency or loss of control signal integrity. Conversely, safety mechanisms may reduce security by exposing diagnostic or fail-safe interfaces to adversaries (Nigam et al., 2018).
• Case Studies: Scenarios such as smart home or hospital IoT ecosystems exemplify the necessity of balancing these axes simultaneously, underlining the call for holistic, multi-faceted design and evaluation frameworks (Fu et al., 2020, Hadi et al., 2023).

4. Methodologies: Co-Engineering, Analysis, and Automation

A spectrum of integrated frameworks and formal methods has been developed, moving beyond siloed, sequential treatments:

• Combined Analysis Techniques:
  • Fault-Tree/Attack-Tree and Extended CFT: Graph-based integration enables identification of failure and attack cut sets, with conditional dependencies and antagonistic relations modeled via gates or logical operators (Nicoletti et al., 2021, Chockalingam et al., 2017).
  • STPA/STPA-Sec: Control structure modeling with unified analysis of unsafe control actions and security constraint violations.
  • Model-Based Engineering: Safety (GSN) models transformed into attack–defense trees, permitting direct propagation of security findings into safety argument confidence levels (Nigam et al., 2018).
• Automated Reasoning and Pattern Recommendation:
  • Semantically-Rich Architecture Patterns (e.g., SafSecPat): Declarative DSLs and knowledge representation methods (disjunctive ASP) automate the placement, co-design, and consequence analysis of safety and security patterns (redundancy, firewalls), surfacing unintended risk propagation and supporting flexible co-engineering (Dantas et al., 2022).
• Dynamic/Emergent Risk Discovery:
  • Agentic AI frameworks employ auxiliary evaluator agents (red-team and defender roles) within sandboxed environments, operationalizing stateful risk scoring functions and supporting continuous, context-sensitive risk management (Ghosh et al., 27 Nov 2025).
• Evaluation and Auditing:
  • Automated pipelines such as Soteria perform static code analysis, extract global event-driven models, and apply LTL model checking to enforce safety and access control invariants across both individual and composed IoT apps (Celik et al., 2018).
  • Black-box evaluation platforms (e.g., aiXamine) aggregate multi-dimensional benchmarks to expose trade-offs and vulnerabilities in LLMs, quantifying adversarial robustness, code security, fairness, privacy, and alignment in unified reports (Deniz et al., 21 Apr 2025).

5. Domain-Specific Standards, Controls, and Best Practices

Effective realization of joint safety and security demands domain-specific yet harmonized operational and regulatory controls:

• Device-Level Standards (IoT, CPS): Secure boot, authenticated firmware updates, enforced lifecycle policies (mandatory device removal or remote-disable at end of support), and incident disclosure requirements; cryptographic assurance for all external interfaces (TLS, DTLS); formal or open-source codebases for transparency and auditability (Fu et al., 2020, Fu et al., 2017).
• Lifecycle Integration: From the earliest system definition phase, extend context and item definitions in IEC/ISO safety standards (e.g., 62061, 26262, 61508) to include security-relevant assets, threat agents, and interfaces, unifying hazard logs and risk matrices under a single scheme (Hänninen et al., 2018).
• Co-assurance Governance (e.g., Industrial Robotics): Synchronization checkpoints between independently led safety and security teams; cross-domain evidence bases; joint incident and patch management; explicit architectural separation of risk control boundaries (e.g., firewalls outside safety-critical controllers) (Gleirscher et al., 2020).
• AI/ML-Specific Controls: Integrated risk management frameworks distinguishing safety (average-case, non-adversarial) and security (worst-case, adversarial) in evaluation, governance, and liability; layered runtime architectures separating safety filters and security monitors; standardized disclosures (model cards, VEX/HEX formats, coordinated vulnerability and hazard reporting) (Sidhpurwala et al., 2024, Qi et al., 2024).
• Policy and Human Factors: Mandated user-centered research on misconfiguration risks, mental models for device/app collections, adaptive regulatory frameworks that accommodate technological dynamics (Fu et al., 2020, Sidhpurwala et al., 2024).

6. Open Challenges and Future Directions

Key obstacles and research priorities persist for the full integration of safety and security:

• Expressiveness and Metrics: Most formal models combine existing safety and security constructs without dedicated "co-dependency" gates or quantitatively rigorous trade-off metrics (e.g., co-dependence, antagonistic risk) (Nicoletti et al., 2021). Articulating these explicitly (e.g., through novel model operators or dynamic Bayesian nets) is an identified gap.
• Scalability: Formal verification and automated reasoning must scale to massive, heterogeneous systems (IoT fleets, adaptive infrastructure), requiring compositional methods and efficient abstraction (Celik et al., 2018, Fu et al., 2017).
• Dynamic Adaptation: Operational risk profiles, especially in agentic and AI-enabled systems, are highly contextual and evolve with both technology and attacker capability, necessitating continuous runtime assurance, red-teaming, and adaptive governance (Ghosh et al., 27 Nov 2025).
• Conflict Resolution: Systematic detection and automated mitigation of requirement or architectural conflicts—particularly where security controls impinge on emergent safety properties—remain complex and underexplored, though SAT/CDCL and answer-set-based schemes provide promising initial workflows (Agbo et al., 2023, Nigam et al., 2018, Dantas et al., 2022).
• Ecosystem-Level Coordination: Lifecycle tracking, ownership attribution, and coordinated disclosure in open-source and consortium AI/model communities are necessary for trustworthy deployments, with community-driven standards for metadata, safety metrics, and continuous benchmarking shaping future regulatory and technical ecosystems (Sidhpurwala et al., 2024).

In summary, the convergence of safety and security is not merely additive but demands deeply integrated, mathematically rigorous, and socio-technically anchored approaches. Future progress will depend upon the adoption of unified models, scalable and explainable co-assurance methodologies, and regulatory policies that are both adaptable and enforceable across the expanding landscape of interconnected, intelligent, and safety-critical systems.