Angora: Efficient Fuzzing by Principled Search (1803.01307v2)

Abstract: Fuzzing is a popular technique for finding software bugs. However, the performance of the state-of-the-art fuzzers leaves a lot to be desired. Fuzzers based on symbolic execution produce quality inputs but run slow, while fuzzers based on random mutation run fast but have difficulty producing quality inputs. We propose Angora, a new mutation-based fuzzer that outperforms the state-of-the-art fuzzers by a wide margin. The main goal of Angora is to increase branch coverage by solving path constraints without symbolic execution. To solve path constraints efficiently, we introduce several key techniques: scalable byte-level taint tracking, context-sensitive branch count, search based on gradient descent, and input length exploration. On the LAVA-M data set, Angora found almost all the injected bugs, found more bugs than any other fuzzer that we compared with, and found eight times as many bugs as the second-best fuzzer in the program who. Angora also found 103 bugs that the LAVA authors injected but could not trigger. We also tested Angora on eight popular, mature open source programs. Angora found 6, 52, 29, 40 and 48 new bugs in file, jhead, nm, objdump and size, respectively. We measured the coverage of Angora and evaluated how its key techniques contribute to its impressive performance.

• The paper presents Angora, a mutation-based fuzzer that integrates gradient descent and taint tracking to efficiently solve path constraints.
• Angora’s context-sensitive branch coverage and byte-level taint tracking enable optimized input mutations, outperforming traditional random mutation and symbolic execution methods.
• Evaluations on both the LAVA-M dataset and mature open-source programs show that Angora detects significantly more bugs, setting a new standard in fuzzing.

Angora: Efficient Fuzzing by Principled Search

The paper "Angora: Efficient Fuzzing by Principled Search" introduces a novel mutation-based fuzzer, Angora, designed to enhance bug detection in software by effectively solving path constraints without resorting to symbolic execution. Fuzzing, a key technique in software testing, faces challenges in balancing input quality and execution speed. While fuzzers using symbolic execution generate high-quality inputs but operate slowly, those relying on random mutation execute quickly but struggle with input quality. Angora addresses these limitations, delivering superior performance compared to existing techniques.

Key Innovations

Angora's design incorporates several innovative components aimed at solving path constraints efficiently:

1. Context-Sensitive Branch Coverage: Unlike AFL's context-insensitive branch coverage, Angora includes context in its branch coverage metric. This enhancement enables more comprehensive exploration of program states, as context information helps distinguish between different executions of the same branch.
2. Scalable Byte-Level Taint Tracking: By identifying specific input bytes that impact path constraints through byte-level taint tracking, Angora focuses mutations on relevant sections of the input, optimizing the search space and improving efficiency.
3. Gradient Descent-Based Search: Instead of symbolic execution, Angora utilizes gradient descent—a method rooted in machine learning—to navigate path constraints. This approach offers computational efficiency and versatility in solving complex constraints.
4. Type and Shape Inference: Angora includes mechanisms to identify and group bytes that form single values in the program, allowing gradient descent to adjust these values as coherent units rather than separate bytes.
5. Input Length Exploration: The fuzzer also detects when the length of an input string influences path constraints and systematically adjusts length to ensure comprehensive state exploration.

Performance Evaluation

Angora's evaluation benchmarks its performance against state-of-the-art fuzzers using the LAVA-M data set and several open-source programs. The LAVA-M data set, characterized by injected bugs, allows a precise measurement of Angora's ability to discover software defects:

• On the LAVA-M data set, Angora detected nearly all injected bugs, outperforming other fuzzers dramatically. Notably, it found eight times as many bugs in the program "who" compared to the next best fuzzer.
• Beyond synthetic data, Angora's capabilities extend to real-world software. In mature open-source programs like "file", "jhead", and "nm", Angora discovered numerous new bugs, evidencing its practical significance.

Implications and Future Directions

The strong numerical results indicate Angora's potential to fundamentally improve the efficiency and effectiveness of software testing processes. Its design principles—particularly in constraint solving via gradient descent—demonstrate the utility of adapting methodologies from machine learning to solve long-standing challenges in software testing. Angora's methodologies could inspire further research into machine learning applications in fuzzing and program analysis, offering pathways to enhance automated testing in increasingly complex software environments.

Future advancements might focus on refining and extending the gradient descent method for broader application in fuzzing, possibly integrating more sophisticated machine-learning techniques to anticipate edge cases and enhance coverage. Additionally, exploring further optimizations in taint tracking and context sensitivity could fuel continued improvements in fuzzing methodologies.

In conclusion, Angora represents a substantial advancement in fuzzing technology, showcasing how principled, methodical approaches can drive significant improvements in software testing. Its success sets a new standard for mutation-based fuzzing and provides a solid foundation for the next generation of automated software testing tools.