- The paper introduces InSPECtor, a specification-driven framework that combines constraint solving, symbolic Pcode execution, and hardware differential testing to validate SLEIGH processor specifications.
- Testing Ghidra 10.3 specifications generated 1,208,531 cases across x86-64, AArch64, ARM/Thumb, RISC-V, and MSP430, achieving 90–99% constructor coverage and uncovering 125 confirmed root-cause bugs.
- The evaluation shows InSPECtor reached full constructor coverage in under 40 minutes, while random testing missed 38% of constructors and 47% of bugs, demonstrating the value of targeted generation for emulator and reverse-engineering reliability.
Motivation and problem statement
SLEIGH is a domain-specific language used to describe processor specifications that decode machine code into Ghidra's architecture-agnostic intermediate representation, Pcode. Over 35 processor families are expressed in SLEIGH, and these specifications sit on the critical path of Ghidra's disassembler and decompiler as well as SLEIGH-based emulators such as Icicle, MetaEmu, Alligator, and Styx. Errors in a specification—incorrect operand decoding, missing side effects, or inconsistent semantics—propagate silently through static and dynamic analysis tools, can mislead decompilation, and can be exploited by malware to detect emulated environments. Despite this central role, specification-level correctness had received little attention relative to prior work on testing CPU emulators, hypervisors, hardware designs, and binary lifters.
The paper presents InSPECtor, a framework for systematically validating open-source SLEIGH specifications by differential testing against real hardware. The key observation is that a specification itself encodes enough structure—its constructor hierarchy, pattern constraints, and Pcode semantics—to drive exhaustive, targeted test generation without redundant exploration of operand combinations.
Specification-driven test case generation
InSPECtor treats instruction generation as constraint solving over the decode tree. Starting from the root table, it traverses constructors, translating each constructor's pattern section into constraints on an instruction bit vector and a context bit vector. Three classes of constraints are handled:
- Simple constraints (equality of bit ranges against constants) become fixed bit assignments, enabling early pruning when unsatisfiable.
- Complex constraints (inequalities, non-constant expressions) are checked for potential satisfiability and deferred; after traversal completes, Z3 solves the combined constraints to produce concrete encodings.
- Implied constraints arise from overlapping constructors: since SLEIGH matches more-specific constructors first, generating an encoding for a less-specific constructor requires negating the constraints of overlapping predecessors, including forcing decoder backtracking where subtable evaluation would otherwise preempt the target constructor.
The traversal also models attachments (excluding invalid field values), self-referencing constructors with decode actions (phased decoding and prefix parsing), and imposes a traversal limit to avoid exponential blowup in combinatorial constructs such as ARM register lists (216 paths). For each decoded path, the solver produces representative encodings at constraint boundaries (minimum/maximum via Z3 optimization) plus aliasing-forced variants, targeting off-by-one errors in partial constraints.
Complementing instruction generation, state generation symbolically executes a constructor's Pcode semantics and, for each Pcode operation, asserts edge cases (overflow, zero/negative results, oversized shifts, NaN operands, branch conditions, etc.) solved independently to avoid combinatorial explosion. This yields initial register/memory states that trigger subtle semantic defects even when arbitrary states would not.
Differential testing environment
Generated test cases execute on two systems: an emulator built from the specification under test (Icicle, executing Pcode) and hardware references. For x86-64, AArch64, and ARM/Thumb, KVM-based virtual machines serve as hardware oracles, with careful configuration of processor mode, control registers, page tables, and exception handling to support single-stepping. For MSP430 (an MSP430FR5969 microcontroller using its EEM debug module) and RISC-V (a BeagleV-Ahead with Xuantie C910 cores driven via ptrace), debugger-based references substitute for unavailable virtualization support. After single-stepping an instruction from a prepared initial state, final registers, memory, and exception outcomes are compared between emulator and hardware.
Results
Using the Ghidra 10.3 specifications, InSPECtor generated 1,208,531 test cases across five ISAs with constructor coverage of 90–99%:
| Architecture |
Instructions |
States |
Constructor coverage |
| x86-64 |
71,797 |
713,231 |
97% |
| AArch64 |
25,042 |
123,407 |
97% |
| ARM/Thumb |
30,521 |
207,716 |
96% |
| RISC-V |
9,725 |
39,393 |
99% |
| MSP430 |
7,440 |
124,784 |
90% |
Differential testing produced 589,713 raw discrepancies corresponding to 38,920 unique constructor combinations. Manual triage (approximately four hours per bug) reduced these to 125 unique root-cause bugs with proposed fixes submitted to Ghidra maintainers, distributed as 32 x86-64, 23 AArch64, 17 ARM/Thumb, 18 RISC-V, and 35 MSP430 bugs. Bug classes comprise decoding issues (16), semantic ordering issues (9), incorrect semantics (87), and aliasing issues (9). Representative examples include AArch64's BCAX performing OR instead of XOR, STLRB storing word-sized values, RISC-V floating-point instructions lacking NaN boxing, MSP430X ignoring the upper 4 bits of 20-bit constants, and x86-64's PACKUSWB converting 0x00ff to zero due to an off-by-one comparison. Several fixes have been merged into Ghidra releases (11.1–12.1).
An ablation study quantifies the necessity of each technique: removing deferred-constraint solving reduces generation accuracy by up to 69.6% (MSP430); removing implied overlap constraints costs up to 54.7% accuracy (RISC-V); disabling backtracking leaves up to 75 constructors untested. Against a naive random-generation baseline on x86-64, which missed 38% of constructors and 47% of bugs after 1.1×107 test cases over eight hours, InSPECtor achieves full constructor coverage in under 40 minutes and finds all 32 annotated bugs—including cases like CVTSS2SI (complex prefix-dependent encoding) and PACKUSWB (single-value boundary condition) that random testing cannot reach.
The authors also document cross-vendor hardware discrepancies (e.g., Intel versus AMD handling of address-size overrides on control-flow instructions, VGATHER mask upper bits, RCPPS approximation algorithms) and cases where hardware deviates from published documentation (MSP430 stack pointer alignment, an undocumented SXTX.W encoding).
Recommendations for the SLEIGH language
From the unresolvable bug classes, the paper distills eight recommendations: decompiler hints to reconcile emulation fidelity with decompilation readability; templated constructors and arrays to eliminate copy-paste duplication (the source of register-list bugs); safer accessor semantics for export statements, whose deferred evaluation caused nine aliasing bugs; inline helper functions replacing black-box pcodeop placeholders; vectorized Pcode operations or structured loops for SIMD; explicit floating-point rounding modes and conversion operations; first-class annotation of unpredictable/undefined outputs; and per-operation memory alignment information. The authors argue the core methodology could extend to other specification languages such as Sail and VADL, though language-specific representation bridging remains the principal engineering challenge.
Limitations and open questions
The approach validates correctness but not completeness: instructions absent from a specification are never generated or tested, although their failure mode (invalid-instruction exceptions) is usually conspicuous in emulation, if less so in disassembly-based control-flow recovery. Single-instruction testing cannot cover cross-instruction state dependencies such as ARM/Thumb's IT blocks, which alter the execution of subsequent instructions. Discrepancy-to-bug attribution is inherently manual; the reported counts treat discrepancies as a lower bound, and categories such as exceptions and flags mix genuine defects with ISA-defined unpredictable behavior that no specification change can resolve. The RISC-V evaluation is further constrained by hardware extension coverage and privilege levels, limiting testing to unprivileged instructions. Whether automated triage can replace the roughly four-hour-per-bug manual analysis remains open.
Conclusion
This work establishes the first systematic validation methodology for SLEIGH processor specifications, combining symbolic traversal of decoding rules, edge-case-driven state generation, and hardware-oracle differential testing. Its application to five diverse specifications yielded 125 confirmed bugs with fixes contributed upstream, demonstrated the insufficiency of naive fuzzing through controlled comparison, and produced concrete language-level recommendations to reduce future defect classes. The released tooling provides a foundation for ongoing, automated assurance of specifications that underpin widely used reverse-engineering and emulation infrastructure.