- The paper proposes an integration-first process that measures structural coverage during functional and system testing on the device under test, using unit tests only to resolve justified gaps.
- The paper combines passive hardware trace with hybrid runtime analysis to support verifiable decision, condition, and MC/DC coverage, reporting indicative overhead of 1.3% in code size and 15.6% in execution time versus gcov’s 62.5% and 147%.
- The paper introduces Hyper Coverage to consolidate evidence across test levels, configurations, and variants, exposing dormant or untested source lines that per-variant 100% reports can conceal.
The coverage representativeness problem
Structural coverage is conventionally collected at unit level in embedded projects, not because that is where it is most meaningful, but because instrumentation and observability are cheapest there. This paper, authored by researchers at Accemic Technologies and Razorcat Development, argues that this produces a systematic mismatch: the most representative completeness signal would come from integration and system tests executed on the device under test (DUT), yet the measurement technologies available for those tests either perturb the system under observation or lose fidelity exactly where certification demands it most.
The authors frame the problem through two properties of any coverage report. Representativeness requires that the measured binary, configuration, and execution environment correspond to what ships. Mapping integrity requires that the correspondence between observed execution and source-level constructs be repeatable, reviewable, and robust against compiler transformations. At unit level both are easy to obtain; at integration level they pull in opposite directions. Software instrumentation preserves mapping integrity but alters timing, cache behaviour, memory footprint, and multi-core contention patterns — potentially masking or creating race conditions and invalidating real-time behaviour. Trace-only measurement preserves representativeness because hardware trace acquisition (e.g., Arm ETM via CoreSight, Infineon AURIX TriCore trace units) is passive, but its object-to-source mapping degrades sharply under optimization precisely for decisions, conditions, and MC/DC — the criteria that matter under DO-178C, ISO 26262, and IEC 61508.
An indicative measurement on an SQLite workload running on an Arm Cortex-A53 at 1.5 GHz quantifies the instrumentation cost: gcc/gcov instrumentation increased code size by 62.5% and execution time by 147% relative to an uninstrumented baseline. The authors are careful to qualify these figures as workload- and layout-specific, and they explicitly concede that some workloads absorb arc counters almost entirely in the microarchitecture. Their structural argument is subtler: because overhead is workload-dependent, the timing behaviour of a production binary cannot be bounded a priori from an instrumented build — and bounding it is what integration-level evidence is supposed to accomplish.
An integration-first process
The proposed process treats integration and system tests as the baseline completeness metric rather than as a supplement to unit testing. It proceeds in three steps: unit tests exist during development for fast feedback but are not assumed audit-grade by default; functional integration and system tests run against requirements on the DUT while structural coverage is measured continuously; and a gap-analysis loop resolves every uncovered element by adding an integration test, adding a missing requirement, removing unreachable code, justifying an infeasible gap, or — only where justified — adding a complementary unit test.
The paper's economic claim is deliberately narrow: the goal is not to abolish unit testing but to stop writing audit-grade unit tests whose sole purpose is manufacturing coverage numbers for code that realistic integration tests already exercise. Notably, the authors decline to state a general figure for how much baseline coverage well-structured integration suites achieve, citing project-specificity and non-disclosure obligations — a candid omission given the conference title's "half of my test cases for free" framing.
Justified gaps receive explicit treatment: defensive paths, test-environment economics, safety constraints preventing state provocation, unused variants, and partially used third-party code all carry documented rationales traceable to affected code elements. Unreachable code demands an explicit decision — removal if genuinely dead, documented justification otherwise — supported by tooling for marking and re-reviewing such decisions across code-base changes.
Trace-based measurement and the limits of naive mapping
The measurement pipeline captures the trace stream, reconstructs executed instruction addresses and control flow, aggregates evidence across runs, produces object-level coverage as primary truth, and derives source-level coverage via explicit mapping artifacts. Live processing (as implemented in CEDARtools.Coverage) removes the constraints of finite trace buffers and post-processing latency, which matters for system tests running hours or days.
The naive approach — mapping addresses back through DWARF debug information — works acceptably for statement coverage under modest optimization but fails structurally for demanding criteria: optimizing compilers rewrite exactly the control flow the mapping depends upon. Two examples make this concrete. Under if-conversion, Arm gcc compiles a source-level branch into a conditional move (moveq), so no branch exists for trace to observe — the decision remains in source but leaves no control-flow footprint. Under -O3, complex Boolean decisions may lower into arithmetic evaluation with no per-condition branches at all, making MC/DC reconstruction from control flow alone unreliable. The consequence the authors draw is strong: trace-only mapping may become non-verifiable under optimization, hence not auditable as evidence.
Hybrid runtime analysis
Hybrid runtime analysis (hRA) bridges the trade-off by combining passive trace observation with minimal, semantics-preserving source-code supplementation confined to decision and condition boundaries. The design goals are explicit: retain a release-like optimized build, preserve source-level control-flow intent at decision points, produce stable trace anchors, externalize all counting and coverage state off-target, and render the mapping verifiable rather than heuristic. The modifications are distinguished from classical instrumentation by three properties: no per-point counters in target memory, no read-modify-write patterns, and confinement to keeping decision/condition evaluation events distinguishable in the trace stream.
For MC/DC, hRA introduces observability points for each atomic condition, locally protected against the specific optimizations that would erase the decision structure; off-target analysis then reconstructs the condition histories MC/DC requires. The concrete preservation scheme is withheld — described only as beyond the paper's scope and subject to a pending international patent application by Accemic Technologies, a limitation on reproducibility that readers should note.
The reported overhead on the same SQLite/Cortex-A53 workload is +1.3% code size and +15.6% execution time versus gcov's +62.5% and +147%. Again flagged as indicative rather than benchmark-grade, these figures have a structural explanation the authors emphasize: residual cost stems from locally preventing selected compiler transformations, not from runtime counter traffic. The paper also concedes plainly that any build modification can affect timing — hRA targets a release-like artifact, not an identical one, and the nonzero on-target footprint should be characterized per platform rather than assumed away.
Assurance rests on three reviewable artifacts: a transformation report listing modified locations and applied rules, a build fingerprint binding the mapping to one specific binary, and independently reprocessable trace-derived evidence. Together these support a third-party-reproducible argument that source intent is preserved and the observation mechanism does not depend on modifying runtime behaviour to record it.
Hyper Coverage consolidation
Two distinct consolidation problems arise in large embedded projects: heterogeneous evidence formats across test levels and harnesses, and per-variant reporting that hides dormant code. Conditional compilation can exclude code from a measured configuration entirely; a 100% per-variant report says nothing about source-tree code never active in that build — a blind spot with both safety and security implications, since dormant code may ship in another configuration carrying defects no measurement touched.
Hyper Coverage (in TESSY) consolidates evidence across levels, runs, configurations, and variants over a project-defined set V of relevant variants, with exclusions from V themselves documented. Consolidation occurs at original-source line level with three verdicts: passed (all relevant results satisfy the selected criteria), failed-partial (some coverage exists but at least one measure falls below 100% in at least one relevant variant), and failed-none (no coverage observed). The value of this formulation is that it exposes untested lines even when every per-variant, per-function view appears fully covered. Integration evidence exchanges through common formats such as LCOV, complemented by structured artifacts for decision, condition, and MC/DC evidence.
Deciding when unit tests remain necessary
Unit tests remain warranted when a requirement cannot be stimulated safely or economically on the integrated system, physical constraints prevent exploration of a state space (sensor ranges, mechanical limits), a module reused across products needs independent evidence, or gap analysis identifies a decision infeasible at integration level without unacceptable risk. In each case the unit test carries a documented justification linked to the specific gap — the linkage being what prevents unit tests from silently substituting for, and thereby masking, missing integration tests.
Limitations and practical prerequisites
Three prerequisites bound applicability. First, embedded trace must be available and accessible during integration testing — trace pins routed, debug access policies permitting use, streaming possible without perturbing behaviour — and retrofitting these late in a programme is expensive or impossible. Second, reliable object-to-source mapping requires a stable, reproducible build environment with controlled compiler versions and optimization settings. Third, tool qualification considerations apply depending on certification context, and the hRA mechanism itself is only partially disclosed due to the pending patent application, limiting independent technical evaluation of the preservation scheme.
Conclusion
The paper contributes an integration-first coverage process, a trace-based pipeline treating object-level execution as primary truth, hybrid runtime analysis enabling verifiable decision/condition/MC/DC evidence on optimized binaries, and cross-variant consolidation exposing lines untested in every relevant variant. Its strongest quantitative claim — roughly two orders of magnitude less code-size overhead than gcov instrumentation on the demonstrated workload (+1.3% vs. +62.5%) — is presented with appropriate caveats about generality. The open questions the work leaves are concrete: the generalizability of the overhead figures across compilers, architectures, and workloads; independent assessment of the undisclosed control-flow preservation scheme; and characterization of hRA's residual timing footprint across target platforms.
