Tree-of-Parsers Methods

• Tree-of-Parsers methods are a systematic approach for synthesizing specialized binary parsers via decision tree induction, dynamic instrumentation, and loop generalization.
• They leverage precise byte-level log collection and affine loop summarization to reconstruct parser logic and adapt to variable file formats with robust safety checks.
• Decision tree induction partitions file variants to generate drop-in parser replacements that guarantee semantic equivalence and prevent runtime errors.

Tree-of-Parsers (ToP) methods comprise a systematic approach for inferring and synthesizing collections of binary data parsers by inducing a decision tree over file types, where each leaf encodes a specialized parser. BIEBER (Byte-IdEntical Binary parsER) exemplifies this methodology, enabling the regeneration of drop-in binary parsers whose semantic and structural outputs (in-memory representations) are byte-for-byte identical to the originals. ToP pipelines leverage dynamic instrumentation, loop summarization, symbolic generalization, decision-tree induction, and guaranteed safety checks, forming a comprehensive solution to the automatic reverse engineering and robust reproduction of binary format parsers from program executions.

1. Dynamic Instrumentation and Log Collection

The starting point of ToP is the dynamic instrumentation of an existing parser to capture detailed data-flow logs. In BIEBER, the parser is compiled under DIODE, a byte-level data-flow tracker. The instrumented program is executed on a corpus of representative input files, producing logs that map output buffer bytes to their computation from input file bytes: $$\text{out}[i] = f(\text{in}[j_1], \ldots, \text{in}[j_n])$$ These logs represent precise byte-level dependencies but omit control-flow and internal loop structure. Logs are tailored to each input's type and size, ensuring exactness but requiring later generalization.

2. Loop Summarization and Generalization

To recover the underlying parsing logic, ToP methods use loop summarization. BIEBER analyzes flat out–in byte correspondences to synthesize nested, zero-based for-loops with constant bounds and affine index functions. This process reconstructs the basic structure:

• Loops are initially instantiated with constant bounds inferred from exemplar files.
• Loop bodies are matched to array-like accesses and header-field patterns.

Generalization is necessary to handle variant file sizes. ToP methods detect the natural stride (e.g., 1, 2, 3, ... bytes) using parsimony heuristics that optimize for minimal IR instruction count. The bounds for these loops (previously fixed at $N$) are then rewritten as symbolic arithmetic functions of header bytes: $N = f(h_{i_1}, h_{i_2}, \ldots, h_{i_k})$ Template expressions are instantiated and validated by corpus-wide voting on which candidate best generalizes the observed bounds. Safety is enforced by wrapping each generalized loop bound with run-time checks ensuring header values are within permitted ranges (not exceeding file length, non-negative unless required).

3. Decision Tree Induction and Specialized Parser Construction

In formats where multiple variants exist (e.g., differing compression or channel-counts), ToP induces a decision tree whose internal predicates inspect file header fields. BIEBER formalizes this as: $$\text{Tree} ::= \mathrm{Leaf}(\text{Parser}) \;\big|\; \mathrm{Node}(p, \;\text{Tree}_{\rm true}, \;\text{Tree}_{\rm false})$$ Each predicate $p$ is a test of the form $\text{in}[i]=c$ that segregates passing and failing exemplars as the tree is built. Leafs (terminal nodes) correspond to parsers (IR fragments) specialized for the specific file variant, constructed from the generalized loop representations above. The induction pseudocode takes the form: $$\begin{array}{l} \textbf{function}\ \mathit{BuildTree}(E, L, \mathit{oracle}) \ \qquad P = \mathrm{BuildIndivParser}(E, L) \ \qquad (\mathit{ok}, \mathit{fail}) = \mathrm{Test}(P, E, \mathit{oracle}) \ \qquad\mathbf{if}\ |\mathit{fail}|=0\ \mathbf{return}\ \mathrm{Leaf}(P) \ \qquad\mathbf{else if}\ |\mathit{ok}|=0\ \mathbf{return}\ \mathrm{Leaf}(\mathit{null})\ \qquad\text{else }p=\mathrm{PickPredicate}(\mathit{ok},\mathit{fail})\ \qquad\quad(\mathit{sat},\mathit{unsat})= \mathrm{Split}(E, p)\ \qquad\mathbf{return}\,\mathrm{Node}\bigl(p, \;\mathit{BuildTree}(\mathit{sat},L), \;\mathit{BuildTree}(\mathit{unsat},L)\bigr) \end{array}$$ where PickPredicate identifies the most discriminative header-byte test.

4. Intermediate Representation, Code Generation, and Semantic Preservation

ToP synthesizes inferred parsers into an imperative IR supporting:

• Zero-based for-loops with symbolic bounds and strides
• Let-bindings for intermediate computations
• Conditional dispatch (predicates) for the decision tree
• Calls to readHeader and writeByte constructs

Example IR fragment for 16-bit stereo WAV:

MIN_Y := read32(fp,40); FACTOR := 2; for (i=0; i<MIN_Y; i+=4) { j = i * FACTOR; out[j] = DIODE_EXPR(i + 44); }

C backends lower DIODE_EXPR formulas into fixed-width arithmetic with checked helpers for fseek/fread and dynamic-array safe writes. Perl backends emulate bit-vectors via wrappers.

Semantic equivalence to the original is achieved by matching the arithmetic expressions, guarding all I/O operations, and replicating output structure exactly. Thus, the output can serve as a safe, drop-in parser replacement.

5. Safety Properties and Formal Guarantees

ToP systems, as instantiated by BIEBER, incorporate strong safety mechanisms:

• Input reads are checked to prevent seeking or reading past EOF.
• Output writes use dynamically resizing arrays to prevent memory errors and buffer overflows.
• Parsers are constructed to avoid segfaults and enforce strict bounds on malformed files.

Guaranteed correctness properties include:

• 100% coverage of training examples by iterative log expansion.
• High generalization: held-out cross-validation yields ≥99.98% accuracy on WAV and BMP formats over 100 random splits.
• Each tree leaf is exact for its file variant, often generalized to all files of that variant.

6. Evaluation, Instrumentation Efficiency, and Format Reverse Engineering

Empirical evaluations demonstrate ToP's robust accuracy and efficiency:

• Decision trees parsed a WAV corpus (1654 files) and BMP corpus (11,008 files across four subformats) with mean accuracies of 99.98% (WAV 99.76–100%, BMP 99.93–100%).
• For MT76x0 firmware (5 variants), a single-leaf parser handled all inputs post-generalization.
• Instrumentation overhead is minimized: rather than logging all files (which would entail ~13,000 CPU-days for BMPs), BIEBER incrementally logs only the few smallest unparseable files, reducing to 74 files (~7.5 CPU-days), a substantial computational saving.

Reverse engineering applications are readily supported. BIEBER's methodology isolates variant-specific bugs by placing problematic inputs into distinct tree leaves with unique predicates. For stb_image, this surfaced two new non-memory bugs and re-discovered one known defect, attributed to unusual compression or masking fields (e.g., in[14]=3 for "compression type 3" BMP, in[28]=32 for 32 bpp cases).

7. Methodological Significance and Plausible Implications

Taken together, Tree-of-Parsers approaches, as demonstrated by BIEBER, constitute the first fully automatic system for regenerating drop-in binary parsers from instrumented executions, providing formal guarantees of coverage and safety. This methodology not only yields semantically faithful parser code with built-in runtime protection but also streamlines reverse engineering processes by exposing key semantic distinctions in format variants. A plausible implication is the broader applicability of ToP methods to automated format discovery, vulnerability detection, and legacy software migration, predicated on the provable correctness and resource efficiency documented in longitudinal evaluations.