Symbolic Sandbox: A Runtime for Binary Symbolic Execution

• Symbolic Sandbox is a framework for direct, path-sensitive symbolic execution of binary code using formal, mechanized ISA semantics.
• It leverages machine-readable ISA specifications and free-monad techniques to eliminate IR translation ambiguities and ensure precise state transitions.
• The framework is extensible to multiple ISAs and integrates with Z3 to provide efficient symbolic reasoning and bug detection.

A symbolic sandbox is an accurate and extensible runtime framework for symbolic execution of binary code, constructed upon formal input–output semantics of a target instruction set architecture (ISA). The approach pioneered in BinSym avoids the pitfalls of traditional IR-based symbolic execution by leveraging machine-readable, mechanized ISA specifications to provide exact semantics for each instruction. This enables direct, path-sensitive symbolic execution at the binary level, without the semantic ambiguities or maintenance overheads associated with manual instruction lifters or English-language ISA descriptions. BinSym achieves its extensibility and correctness by parameterizing all instruction semantics over a small set of stateful primitives, defined as operation constructors in a free monad, and by instantiating symbolic values as paired concrete words and SMT solver ASTs (Tempel et al., 2024).

1. Formal Representation of ISA Semantics

BinSym builds upon the LibRISCV Haskell library, which comprehensively encodes the RV32I core and M/A/C extensions as algebraic data types and generalized algebraic data types (GADTs) describing both bit-vector expressions and primitive stateful effects. Each RISC-V instruction acquires a direct mechanized definition, e.g.,

$\mathsf{Operations}\;v\;r$0

LibRISCV’s expressions constitute a freely-generated algebra $E$, with constructors such as $\mathsf{FromInt}$, $\mathsf{ZExt}$, $\mathsf{Add}$, $\mathsf{Eq}$, etc. All stateful ISA effects (register/memory access, control flow) are represented as a GADT $\mathsf{Operations}\;v\;r$, with typical constructors:

• $$\mathsf{ReadRegister} :: v \to \mathsf{Operations}\;v\;v$$
• $$\mathsf{WriteRegister} :: v \to v \to \mathsf{Operations}\;v\;()$$
• $\mathsf{LoadMem}$, $\mathsf{StoreMem}$, $\mathsf{FromInt}$0, $\mathsf{FromInt}$1

A generic interpreter for $\mathsf{FromInt}$2, using standard free-monad techniques, yields a RISC-V emulator. BinSym reuses this machinery, instantiating $\mathsf{FromInt}$3 as a concolic value (concrete bit and symbolic Z3 AST) and hooking $\mathsf{FromInt}$4 to the Z3 backend instead of native unboxed integer code.

Key Definitions

• Let $\mathsf{FromInt}$5 (RISC-V machine state): $\mathsf{FromInt}$6
• Symbolic value $\mathsf{FromInt}$7
• Path condition $\mathsf{FromInt}$8: finite set of Z3 Boolean ASTs

2. Symbolic States and Expressions

The symbolic sandbox maintains a symbolic state $\mathsf{FromInt}$9, where:

• $\mathsf{ZExt}$0: the complete RISC-V machine state, with all registers, memory, and program counter as pairs of concrete values and SMT ASTs.
• $\mathsf{ZExt}$1: the path condition, a finite conjunction of Boolean ASTs (each corresponding to a control-flow decision).

The core expression type $\mathsf{ZExt}$2 is interpreted by: $\mathsf{Operations}\;v\;r$1 Each constructor (e.g., $\mathsf{ZExt}$3) is dispatched to the corresponding Z3 bit-vector operator:

• $\mathsf{ZExt}$4
• $\mathsf{ZExt}$5

Distinctively, each concolic value pairs a concrete word for decoding and an optional AST for symbolic path reasoning, eliminating the overhead of symbolic opcode resolution and ensuring precise branching semantics.

3. Inference Rules for Core Instructions

Instruction semantics are extractible into standardized inference rules, mapping symbolic state transitions. Each semantic primitive is mapped directly to a monad action, avoiding IR translation:

• BEQ (Conditional Branch):

Let $\mathsf{ZExt}$6, $\mathsf{ZExt}$7, $\mathsf{ZExt}$8, and $\mathsf{ZExt}$9.

Branch taken:

$\mathsf{Add}$0

Not taken:

$\mathsf{Add}$1

• ADD:

$\mathsf{Add}$2

• LW (Load Word):

Let $\mathsf{Add}$3, $\mathsf{Add}$4,

$\mathsf{Add}$5

All state transitions are mediated directly by the monadic $\mathsf{Add}$6, ensuring the sandbox runs strictly according to the mechanized ISA semantics.

4. Architecture and Execution Framework

The sandbox relies upon the following layered runtime:

1. ISA parser: Direct import of LibRISCV (or similarly formalized ISA), exposing functions: $\mathsf{Operations}\;v\;r$2 This provides canonical instruction semantics as specified in the source ISA, with no hand-written lifter.
2. Semantics generator: A generic free-monad interpreter parameterized over $\mathsf{Add}$7 (concrete or concolic) and supporting a mapping from $\mathsf{Add}$8 to Z3 ASTs.
3. Symbolic executor: Maintains a worklist of symbolic states $\mathsf{Add}$9. Each step proceeds as follows:
   • Decode $\mathsf{Eq}$0 concretely, yielding a 32-bit opcode.
   • Lookup $\mathsf{Eq}$1 for the decoded instruction.
   • Run the concolic interpreter to compute successor states.
   • When encountering $\mathsf{Eq}$2, spawn two branches extending $\mathsf{Eq}$3 with either guard.
   • Query Z3 with each resulting $\mathsf{Eq}$4 to generate distinguishing inputs.

The pipeline proceeds as:

$\mathsf{Operations}\;v\;r$3

5. Extensibility and Multi-ISA Support

Extending the symbolic sandbox to new ISAs requires only providing a module (e.g., MyFancyISA.hs) that defines:

1. $\mathsf{Eq}$5 constructors for any new ops.
2. An $\mathsf{Eq}$6 GADT listing requisite stateful actions.
3. Monad definitions of instruction semantics, e.g., $\mathsf{Operations}\;v\;r$4

The symbolic interpreter is generic over any $\mathsf{Eq}$7 instance. At startup, multiple ISA specs may be loaded and merged:

$\mathsf{Operations}\;v\;r$5

This design ensures that no solver-specific or execution-engine code requires modification to support additional ISAs.

6. Case Study: Evaluation and Bug Discovery

A comparative evaluation was performed on BinSym versus three open-source RISC-V symbolic engines:

• SymEx-VP (direct RV32 exec in SystemC)
• BinSec (DBA-lifted static exec)
• angr (VEX-lifted dynamic exec in Python)

Benchmark tasks included base64-encode, bubble-sort, is-prime, insertion-sort, and uri-parser, each given a fixed array of symbolic bytes. Each tool was run five times per benchmark using Docker+Z3, exploring all feasible paths (100%).

Table: Benchmark Runtimes (seconds)

Tool	base64	sort-bubble	is-prime	insertion	uri-parser
BinSym	169	52	98	67	54
SymEx-VP	217	44	129	122	67
BinSec	229	82	136	128	95
angr	32*	256	207	393	322

*angr on base64 explored only 125 of 6250 feasible paths; all others covered 100%.

Key findings:

• BinSym solved the suite ~30% faster than SymEx-VP, ~40% faster than BinSec, and ~5× faster than angr.
• angr failed to discover 6125/6250 feasible base64 paths due to a branch encoding error.

Five previously undocumented bugs were identified in angr’s RISC-V front-end:

1. BEQ immediate miscalculation (offset error)
2. I-type immediate SEXT-ZEXT confusion (incorrect sign)
3. Omitted decoding of certain CSR funct3 values
4. Absent load-word alignment check disabled unaligned branches
5. SRA implemented as logical shift

Each was detected as a failing symbolic path, producing minimal counterexamples.

7. Performance, Trade-offs, and Limitations

BinSym’s symbolic sandbox demonstrates competitive performance with minimal decoding overhead due to direct Z3 AST manipulation and elimination of IR lifters. However:

• Using boxed bit-vectors may be suboptimal compared to potential direct-bitblast backends.
• Concolic values circumvent the opcode decoding issue but complicate verification for fully symbolic address ranges; future work is anticipated on lazy arrays or advanced heap models.
• The free-monad interpreter introduces minor per-operation overhead, but most instructions require only a direct $\mathsf{Eq}$8 call and single register update.

Proposed extensions include Coq or HOL4 proofs of $\mathsf{Eq}$9 correctness (relative to golden models such as Sail), hardware-peripheral modelling for firmware support, and ISA augmentation (e.g., RV64, CHERI, ARMv8) by plugging additional GADTs into the same interpreter.

For experimental reproducibility, code and benchmarks are released at https://github.com/agra-uni-bremen/binsym; all requirements are distributed for re-execution under Guix (Tempel et al., 2024).