WFD: Open-Source Benchmark for REST APIs
- WFD is a standardized open-source benchmark for REST API fuzzing featuring 36 JVM-based APIs with reproducible scaffolding.
- It includes Web Fuzzing Commons (WFC) which provides declarative schemas, libraries, and reporting conventions to normalize authentication and fault categorization.
- WFD enables comparative evaluation through automated test generation, containerized deployment, and detailed fault analysis, addressing fragmentation in prior API fuzzing studies.
Searching arXiv for papers on Web Fuzzing Dataset (WFD), WFC/WFD, and related web/API fuzzing benchmarks. Web Fuzzing Dataset (WFD) is an open-source benchmark corpus for REST API fuzzing that, in its specific 2025 formulation, consists of 36 open-source APIs together with the scaffolding needed to run reproducible experiments, especially when combined with Web Fuzzing Commons (WFC), a companion set of schemas, libraries, and reporting conventions for authentication and fault categorization. The term also sits within a broader research context in which “web fuzzing datasets” may denote public APIs, self-developed benchmarks, purposely vulnerable applications, recorded request/response corpora, or seed-template collections. In that broader literature, WFD is best understood as a standardization effort for empirical REST API fuzzing, introduced after earlier work had explicitly identified the absence of a widely accepted benchmark and stable evaluation conventions as a central obstacle to progress (Sahin et al., 1 Sep 2025, Dharmaadi et al., 2024).
1. Historical emergence and problem setting
A central motivation for WFD was the state of web API fuzzing evaluation documented in the 2024 survey “Fuzzing Frameworks for Server-side Web Applications: A Survey” (Dharmaadi et al., 2024). That survey reports that, for empirical evaluations, 43% of studies used publicly available web applications, 39% developed new benchmarks, and 18% employed third-party benchmarks. It also states: “There is neither a widely accepted benchmark for web API fuzzers nor established security-relevant metrics in comparing the performances of the fuzzers.” The consequence was a fragmented evaluation landscape in which objective comparison, reproducibility, and cumulative progress were difficult.
Before WFD, several benchmark-like resources already existed, but they were heterogeneous in scope and format. “Deriving Semantics-Aware Fuzzers from Web API Schemas” assembled a public suite of 16 distinct open-source web services implemented in Python, JavaScript, Go, and Rust, spanning Flask, Django, Express, Tornado, Rust Tide, and Go net/http, with schemas in OpenAPI and GraphQL (Hatfield-Dodds et al., 2021). The 2024 survey also identified public online APIs, self-developed benchmarks such as the EvoMaster Benchmark, and third-party vulnerable applications such as WebGoat, Gruyere, DVWA, and NodeGoat as the dominant evaluation choices (Dharmaadi et al., 2024). WFD emerged from this environment as a dataset intended not merely to collect systems under test, but to normalize the surrounding experiment infrastructure.
The official WFD was introduced together with WFC in “WFC/WFD: Web Fuzzing Commons, Dataset and Guidelines to Support Experimentation in REST API Fuzzing” (Sahin et al., 1 Sep 2025). In later work, WFD is described as a curated, open-source corpus of REST APIs “previously known as EMB,” indicating a lineage from prior EvoMaster-centric benchmarks toward a more explicitly standardized community resource (Sahin et al., 1 Apr 2026).
2. Corpus composition and technical profile
As defined in the WFC/WFD work, WFD contains 36 open-source JVM-based REST APIs written in Java or Kotlin (Sahin et al., 1 Sep 2025). These APIs are sourced from government, commercial, and sample/example projects and vary in size, complexity, and authentication requirements. Aggregate statistics reported for the corpus are 6,465 source files, 657,162 lines of code, and 1,487 endpoints, with 15 APIs requiring authentication (Sahin et al., 1 Sep 2025). A subsequent study using the same corpus reports that 25 of the 36 APIs use a SQL or NoSQL database (Sahin et al., 1 Apr 2026).
| Aspect | WFD characteristic |
|---|---|
| APIs | 36 distinct REST APIs |
| Implementation | Java or Kotlin |
| Aggregate scale | 6,465 source files, 657,162 LOC, 1,487 endpoints |
| Operational features | 15 require authentication; 25 use SQL or NoSQL databases |
The corpus spans both small example APIs and large real-world APIs, including public-sector systems (Sahin et al., 1 Apr 2026). All APIs are accompanied by Docker Compose-based dependency management, which is part of the dataset’s reproducibility design. For APIs requiring authentication, credentials are provided through YAML files following the WFC format (Sahin et al., 1 Apr 2026).
Availability is an explicit part of the design. WFD and the associated WFC assets are maintained as open-source resources, with versioned releases using Zenodo at DOI 10.5281/zenodo.18196038 (Sahin et al., 1 Apr 2026). The dataset repository is provided at github.com/WebFuzzing/Dataset, and the companion WFC repository at github.com/WebFuzzing/Commons (Sahin et al., 1 Sep 2025).
3. Web Fuzzing Commons and the scaffolding model
WFD is not only a corpus of APIs; it is coupled to WFC, which provides declarative schemas and libraries intended to make experiments comparable across tools (Sahin et al., 1 Sep 2025). WFC includes an authentication schema in auth.yaml, a standard fault catalog in fault_categories.json, and a unified result-report schema in report.yaml. It also provides HTML, JavaScript, and CSS assets for report visualization, and uses JSON Schema-based generation to produce parsing classes in languages such as Java and Python (Sahin et al., 1 Sep 2025).
The authentication schema is designed to express both static and dynamic authentication modalities. The paper explicitly describes support for basic authentication, static tokens, token-based login flows, header and cookie extraction, extraction of tokens from arbitrary JSON fields via JSON Pointer syntax, and multiple user accounts per API (Sahin et al., 1 Sep 2025). This is significant because authentication had been singled out as one of the three major issues hindering REST API fuzzing research, alongside fault cataloging and case-study selection (Sahin et al., 1 Sep 2025).
The dataset-side scaffolding is equally concrete. Each API is instrumented for code-coverage measurement via JaCoCo, runs behind a mitmproxy instance for request/response logging, exposes parameterized TCP ports for parallel experimentation, and comes with Bash and Python scripts for setup, execution, result collection, and teardown (Sahin et al., 1 Sep 2025). The stated aim is repeatability: fixed API versions, fixed database states, fixed schemas, and reusable authentication configuration.
The fault-catalog component of WFC standardizes oracle naming. Examples explicitly listed include F100 for HTTP 500 status returned, F101 for response invalid according to the API schema, and F204–F206 for access-policy violations (Sahin et al., 1 Sep 2025). This codification is intended to reduce ambiguity in cross-tool comparisons, especially when different fuzzers support different classes of oracle.
4. Benchmarking methodology and role in comparative evaluation
WFD is presented as infrastructure for fair and repeatable comparative evaluation rather than as a single-tool benchmark (Sahin et al., 1 Sep 2025). The WFC/WFD paper states that EvoMaster was compared with ARAT-RL, EmRest, LLamaRestTest, RESTler, and Schemathesis, and emphasizes that “any fuzzer can benefit from WFC and WFD” (Sahin et al., 1 Sep 2025). This generality is a defining feature: the corpus is meant to be tool-agnostic, while WFC serves as a lingua franca for configuration and result reporting.
The associated guidelines are methodological as much as technical. Tool selection should favor open-source or retrievable systems, with installation treated as a one-time cost and with minimal required command-line parameters such as schema path, host and port, time limit, output directory, authentication configuration, and local LLM configuration if applicable (Sahin et al., 1 Sep 2025). Authentication information should not be hardcoded. Output should be standardized, for example as executable Python tests. API selection should rely on a community-driven, ever-expanding corpus, and cherry-picking subsets is explicitly discouraged unless scientifically justified (Sahin et al., 1 Sep 2025).
A particularly important methodological point concerns what should be measured. The WFC/WFD paper argues that experiment reporting should be based on generated tests, not only on behavior observed during fuzzing sessions, because code coverage observed during fuzzing can overestimate the coverage achieved by the resulting regression tests (Sahin et al., 1 Sep 2025). The same paper also notes self-reporting bias in fault detection and recommends cross-validation through code-coverage and HTTP-log analysis (Sahin et al., 1 Sep 2025). In this respect, WFD is not merely a dataset but a proposal for evaluation discipline.
This methodological framing responds directly to the concerns articulated in the 2024 survey, which had called for a benchmark with diverse web applications under test, automation scripts for deployment and measurement, and a structure enabling fair comparison (Dharmaadi et al., 2024). WFD operationalizes those requirements for the REST API setting.
5. Use of WFD in security-oracle research
A concrete demonstration of WFD’s use appears in “Enhancing REST API Fuzzing with Access Policy Violation Checks and Injection Attacks” (Sahin et al., 1 Apr 2026). That work used the 36 WFD APIs in white-box mode with EvoMaster, fuzzing each API for 1 hour and repeating each experiment 10 times, for a total of 360 hours devoted to the WFD portion of the study. The setup read authentication credentials from WFC YAML files, using at least two users per API if available, and automatically reset database state between stateful test executions (Sahin et al., 1 Apr 2026).
The study introduced nine automated oracles integrated as a post-processing “security testing” phase after fuzzing. These are F204 Existence Leakage, F205 Not Recognized Authentication, F206 Missed Authorization Checks, F900 Ignore Anonymous, F901 Anonymous Modifications, F902 Leaked Stack Trace, F903 Hidden Accessible Endpoints, F200 SQL Injection, and F201 Cross-Site Scripting (Sahin et al., 1 Apr 2026). The paper reports that the WFD experiments found hundreds of security faults across the 36 APIs and that the additional post-processing overhead was lightweight, with median post-processing around 14 seconds and average time below 3 minutes except for outliers (Sahin et al., 1 Apr 2026).
The same study reports a median 2xx-coverage of 53.3% across the WFD APIs, while noting that some APIs had much lower coverage, such as 2.8%, due to input constraints or fuzzing limitations (Sahin et al., 1 Apr 2026). This makes WFD useful not only for fault counting but also for diagnosing the interaction between corpus complexity, authentication, and search effectiveness.
One of the oracles is specified with a simple timing condition for time-based SQL injection detection:
with typical values seconds and seconds (Sahin et al., 1 Apr 2026). The tool output in this work consists of minimized, executable test suites in Java, Kotlin, Python, and JavaScript, which aligns with WFC/WFD’s emphasis on artifact-level reproducibility rather than only run-time observations (Sahin et al., 1 Apr 2026).
6. Relation to adjacent dataset forms and scope boundaries
In the literature, WFD should be distinguished from several adjacent forms of web-fuzzing data. EDEFuzz, for example, generated datasets consisting of recorded API requests and responses, configuration files for browser replay, per-field deletion mutations, DOM snapshots, and manual labels for excessive fields, applied to 8 Australian websites and 69 applicable targets from the Alexa Top-200; the paper explicitly gives no evidence of public dataset release (Pan et al., 2023). ChatHTTPFuzz, in turn, constructs and enriches HTTP “seed templates” for IoT devices through LLM-guided protocol variable annotation and code-guided seed expansion, evaluating on 14 real-world devices and documenting the seed structure, but not explicitly stating public release of a WFD artifact (Yang et al., 2024).
A different neighboring direction is WAF-evasion fuzzing. WAFFLED generated 373,670 HTTP requests across five major WAFs and multiple frameworks, identifying 1,207 unique bypasses and releasing code, grammars, and datasets at github.com/sa-akhavani/waffled/, with some sensitive requests temporarily withheld (Akhavani et al., 13 Mar 2025). Although that corpus satisfies many properties one would expect from a principled web-fuzzing dataset, its target problem is differential parsing and WAF bypass rather than benchmark standardization for REST API fuzzers.
These comparisons matter because a common misconception is to treat “Web Fuzzing Dataset” as a generic label for any collection of web-fuzzing inputs or results. In current usage, the official WFD of WFC/WFD is a specific corpus of 36 open-source REST APIs plus standardized scaffolding (Sahin et al., 1 Sep 2025). By contrast, the broader field still includes public online APIs, self-developed benchmarks, purposely vulnerable educational applications, recorded traffic corpora, and protocol-aware seed datasets (Dharmaadi et al., 2024). This suggests that WFD standardizes one especially important segment of the problem space—REST API fuzzing with reproducible experimentation—rather than exhausting the full range of web-fuzzing dataset needs.
7. Significance, strengths, and remaining research pressures
The principal significance of WFD lies in its attempt to make REST API fuzzing evaluation cumulative. By fixing case studies, packaging authentication declaratively, instrumenting coverage, logging HTTP behavior, standardizing fault codes, and encouraging report schemas tied to generated tests, it addresses several empirical pitfalls that had previously made comparisons difficult or misleading (Sahin et al., 1 Sep 2025). In that sense, WFD and WFC collectively provide an infrastructure layer for experimentation, not just a benchmark list.
Its strengths are correspondingly concrete. The APIs are open source, containerized, versioned, and locally deployable. Authentication is reusable across fuzzers. Coverage and faults can be measured with common instrumentation. Parallel experiments are facilitated through parameterized ports. Report formats and visualization assets promote traceability from detected faults to the tests that reveal them (Sahin et al., 1 Sep 2025). These are exactly the sorts of operational details that were missing when the 2024 survey concluded that web API fuzzing lacked a widely accepted benchmark and established security-relevant metrics (Dharmaadi et al., 2024).
At the same time, the scope of WFD is intentionally narrow in some respects. The corpus is JVM-based and consists of Java/Kotlin REST APIs (Sahin et al., 1 Sep 2025). Earlier benchmark work in web API fuzzing had included Python, JavaScript, Go, Rust, OpenAPI, and GraphQL services (Hatfield-Dodds et al., 2021). The survey literature also emphasized broader desiderata such as injected or annotated vulnerabilities and diversity across monolithic and microservice architectures (Dharmaadi et al., 2024). A plausible implication is that WFD should be viewed as a major standardization step for REST API fuzzing infrastructure, while broader benchmark diversity remains an ongoing research objective.
In this sense, WFD occupies a distinct place in the evolution of web fuzzing research. Earlier work documented the absence of common benchmarks; intermediate work assembled heterogeneous suites of open-source services; WFD then combined a fixed corpus with authentication schemas, fault taxonomies, instrumentation, and experimental guidelines (Dharmaadi et al., 2024, Hatfield-Dodds et al., 2021, Sahin et al., 1 Sep 2025). Its value is therefore not only in the 36 APIs themselves, but in the attempt to turn web API fuzzing into a benchmarked, reportable, and extensible experimental discipline.