Detecting Architectural Drift in Safety-Critical Firmware through Runtime Trace Analysis

Abstract: Maintaining consistency between architectural design and runtime-observed behavior is challenging in long-lived safety-critical firmware. This paper presents a runtime-informed methodology for detecting architectural drift in ISO 26262-compliant firmware. The approach collects hardware-assisted execution traces, abstracts them into message exchanges among firmware components, and compares the resulting runtime behavior with design-time sequence diagrams through a deterministic differencing step. The computed delta identifies discrepancies as confirmed, missing, additional, or inverted, while a constrained LLM-based step generates a human-readable report only to support expert review. We evaluate the methodology in an industrial firmware context through agreement-based validation and a practitioner survey. Results over 26 test cases show strong agreement between the generated deltas and expert-curated references, while practitioners perceive the reports as useful for interpreting drift, reducing manual analysis effort, and supporting safety-oriented documentation activities. The findings suggest that combining runtime trace analysis, deterministic architectural differencing, and constrained LLM-based reporting can practically support architectural drift detection in evolving safety-critical firmware.

• The paper introduces a method that leverages hardware instruction tracing and LLM narrative generation to detect architectural drift in safety-critical firmware.
• It employs a three-phase deterministic approach to map runtime evidence onto UML sequence diagrams, enabling precise detection of confirmed, missing, additional, and inverted interactions.
• Empirical evaluations demonstrate high precision and recall in drift detection with practical industrial acceptance, though challenges remain in handling concurrent scenarios.

Detecting Architectural Drift in Safety-Critical Firmware via Runtime Trace Analysis

Introduction

This work addresses the challenge of maintaining alignment between architectural models and runtime behavior in ISO 26262-compliant, safety-critical firmware. Long-lived embedded firmware systems face recurrent architectural drift due to feature growth, bug fixes, and continuous integration pressure, resulting in divergence between outdated design artifacts and the actual system behavior. The study introduces a methodology for runtime-informed drift detection, leveraging hardware instruction tracing, architectural differencing, and LLM-guided reporting to bridge this gap and provide actionable, scenario-specific delta views for expert analysis.

Figure 1: Conceptual model of the industrial problem addressed: parallel evolution of architecture, features, bugs, and test artifacts complicates reasoning about runtime conformance.

Methodology

The proposed method is organized in three deterministic, auditable phases, complemented by a constrained LLM-based reporting layer:

• Phase 1: Acquisition and abstraction of hardware-assisted instruction traces are mapped to runtime UML sequence diagrams, using symbol information from firmware binaries.
• Phase 2: Deterministic differencing between design-time sequence diagrams and runtime traces, classifying discrepancies as confirmed, missing, additional, or inverted interactions.
• Phase 3: Mining of code repository metadata and LLM-based narrative generation for drift reports, enhancing expert review workflows without influencing the automated analysis.

This strict separation ensures that LLMs provide only interpretive support, eliminating any risk of hallucination in primary compliance evidence.

Figure 2: Workflow and artifacts of Phase 1: post-processing of instruction traces to extract an architectural runtime view.

Figure 3: Workflow of Phase 2: normalized differencing of design vs. runtime sequences to compute an explicit, inspectable delta.

Figure 4: Architecture of Phase 3: construction of human-oriented delta reports via LLM conditioning on machine-computed deltas and development story summaries.

Drift Taxonomy and Delta Views

Four classes of architectural drift are discriminated:

• Confirmed: Runtime evidence matches design specification for interaction, both in presence and order Figure 5.
• Additional: Unexpected interaction observed at runtime, violating isolation or introducing latent dependencies Figure 6.
• Missing: Design-specified interaction omitted at runtime, indicating potential inactivity, dead code, or untriggered paths Figure 7.
• Inverted: Observed order of interaction contradicts the design, surfacing subtle behavioral shifts due to code evolution Figure 8.

Figure 5: Confirmed interactions—intended inter-component communication realized as designed.

Figure 6: Additional interaction—spurious or evolution-induced runtime interaction absent from static design.

Figure 7: Missing interaction—designed interaction not exhibited at runtime.

Figure 8: Inverted interaction—ordering anomaly between design-time and runtime views.

This message-exchange-level classification operates under a strictly sequential execution assumption, consistent with high-integrity, deterministically scheduled embedded platforms.

Industrial Application and Empirical Evaluation

The approach was instantiated and validated in an ISO 26262-governed automotive firmware development context, comprising 26 feature-linked test cases with high architectural complexity (average 9 lifelines and 25 messages). Two primary evaluation tracks were undertaken:

• Correctness: Against expertly reconstructed ground truth, the methodology achieved high precision (96.3%), recall (92.9%), and $F_1$ (94.6%) for confirmed interactions. Missing and additional interaction detection was robust ($F_1 = 88.2\%$ and $86.4\%$, respectively). Inverted ordering detection remains challenging ($F_1 = 60.0\%$), reflecting the increased complexity of monotonic order divergence analysis.
• Report Faithfulness: LLM-generated root-cause reports, inspected by three senior architects, achieved Fleiss' $\kappa = 0.692$ (substantial agreement), but borderline root-cause inference remains error-prone.

For practical adoption, a TAM-inspired practitioner survey (n=23) demonstrated strong perceived usefulness, clarity, and audit support for the generated deltas and LLM reports. Explicit linkage to compliance work products and test artifact mapping remain desired enhancements.

Figure 9: Distribution of practitioner Likert-scale ratings for perceived utility, ease of use, and compliance support of methodology outputs.

Implications and Industrial Lessons

• Evidence Separation: Deterministic differencing and LLM report generation must remain strictly decoupled. This ensures drift detection remains reproducible, non-fabricatable, and subject only to expert appraisal.
• Differential Utility: Serialized delta outputs (e.g., PlantUML) are essential for automated integration and compliance processes; however, human-oriented narratives catalyze in-depth architectural review and safety argumentation.
• Scope Constraints: Current linear scenario support and disregard for UML combined fragments force separate analysis per scenario path. Handling concurrency, branching, and data-flow deviations require research attention.
• Traceability: Practitioners require mapping between observed deltas and safety work products, ASIL assignments, and verification evidence for comprehensive ISO 26262 audits—an open area for tool-chain integration.

Related Work

Prior efforts predominantly decouple static architectural recovery [amalfitano2024], [zhang2014] and runtime trace analysis [convent2018], [chamski2010]. Existing methods either reconstruct architectural views from code artifacts or analyze low-level execution traces for runtime verification and performance. This method extends the landscape by algorithmically aligning runtime behavioral evidence with architecture-level sequence diagrams, exposing architectural drift concretely and augmenting traditional static recovery pipelines. Unlike approaches that task LLMs with architectural inference [hatahet2025generating], this methodology uses LLMs purely for narrative aid, not analysis or reasoning, thus ensuring traceability and reproducibility.

Conclusion

This methodology delivers an auditable, scenario-specific framework for detecting and interpreting architectural drift in evolving ISO 26262-compliant firmware. By synthesizing hardware instruction trace-based runtime evidence, deterministic architectural differencing, and structured LLM narrative generation, it enables both process-level integration and expert-driven review. The architecture is effective for confirmed, missing, and additional discrepancies; future directions include increased coverage of non-linear scenarios, direct test-artifact traceability, and further industrial validation. This positions runtime-informed architecture conformance analysis as a foundational capability in safety-critical embedded firmware engineering.

---

References

• (2607.03135)