Validating HTTP Semantics in REST APIs With Constructed Call Sequence Scenarios

Abstract: Context: REST APIs are widely used in industry. These APIs use HTTP for their communications. Failures in following the specifications of HTTP can lead to confusing and hard to use APIs, with possibly serious software faults with dire consequences. Objectives: Define novel automated techniques to automatically find HTTP semantics-level faults in existing REST APIs. Methods: We extended the state-of-the-art fuzzer EvoMaster with 9 new oracles to detect HTTP semanticslevel faults. Once the standard fuzzing process is finished generating N test cases, a new phase is executed in which these N tests are used as a starting point to create new scenarios (i.e., new sequences of HTTP calls) aimed at validating specific HTTP properties defined in these 9 oracles. Results: Experiments on 9 artificial APIs with inject faults show that our novel techniques can successfully detect all of them. Further experiments on 36 APIs from the WFD corpus show that our novel techniques can automatically find 166 existing faults in these real-world APIs. Conclusion: REST APIs use HTTP, and, as such, they need to follow its semantics to avoid misleading their clients and introducing subtle software faults. The novel techniques presented in this paper are shown to be effective at automatically finding several of this type of faults.

• The paper introduces nine automated HTTP-semantics oracles and deterministic call-sequence construction, extending REST API fuzzing beyond HTTP 500 errors without manual schema changes.
• The evaluation detected all injected faults across nine benchmark APIs and found 166 manually verified defects across 36 real-world APIs, including 133 invalid Allow headers and non-idempotent PUT operations.
• The post-processing approach adds little cost, with a median overhead of 0.5 seconds, but its effectiveness depends on the underlying fuzzer achieving sufficient successful 2xx endpoint coverage.

Motivation and problem statement

REST API fuzzing research has largely converged on a narrow fault oracle: the HTTP 500 (Server Error) response. This oracle misses an entire class of defects in which the server responds with a plausible status code while violating the semantics of the HTTP protocol itself — for example, returning 201 (Created) on every repeated PUT, or leaving a resource retrievable after a successful DELETE. Such faults mislead clients, break assumptions embedded throughout the HTTP stack (proxies, gateways, and client libraries routinely re-execute idempotent requests), and can produce subtle business-logic corruption that manifests only sporadically.

Static analyzers such as SCOAS can flag some of these problems by inspecting OpenAPI schemas, but many rules are only observable dynamically: they require issuing specific sequences of HTTP calls and observing how resources change. Prior dynamic work is limited — RESTler defined four compliance rules [atlidakis2020checking], and IcePick's Glacier-based contract checking proved so restrictive that it could be applied to only two of the 36 APIs in the EMB/WFD corpus, after manual schema modification. The paper's contribution is a set of nine automated oracles, plus a deterministic scenario-construction phase that builds targeted call sequences from an existing fuzzing-derived test suite, applicable out-of-the-box to all 36 WFD APIs without any manual schema edits.

The nine oracles

Each oracle is identified by a WFC-format code in the 9xx range and grounded in RFC 9110 (HTTP Semantics), RFC 5789 (PATCH), or RFC 7386 (JSON Merge Patch):

Code	Name	Basis
900	Non-Working Delete	RFC 9110
901	Side-Effects Failed Modification	RFC 9110
902	Repeated Create PUT	RFC 9110
903	Misleading Create PUT	RFC 9110
904	Partial Update PUT	RFC 9110
905	Non-Idempotent PUT	RFC 9110
906	Invalid Merge-Patch	RFC 5789/7386
907	Invalid Location	RFC 9110
908	Invalid Allow	RFC 9110

The construction machinery rests on three reusable operations. Slice truncates a test case after a target call $\alpha$, keeping tests short while preserving state-setting calls before $\alpha$ (the authors deliberately do not prune preceding GETs, having observed APIs that wrongly create resources via GET). bindAccess attaches new calls to the same dynamically created resource as $\alpha$, reusing the fuzzer's path-resolution logic for nested resources and copying authentication credentials to avoid spurious 401/403 responses. bindQueries strips non-required query parameters from new GETs and copies matching parameter values from $\alpha$, sampling schema-valid values otherwise.

The oracles themselves follow a common pattern: locate a seed test in $N$ exhibiting a needed precondition (e.g., a 2xx DELETE), construct a minimal sequence around it, execute, and flag violations. Notable design decisions include:

• Non-Working Delete (900): a successful GET before the DELETE confirms resource existence; a successful GET after it indicates the deletion failed.
• Side-Effects Failed Modification (901): compares GET responses before and after a failed (4xx) PUT/PATCH, with case-specific handling for 401 (strip credentials), 403 (use a different credential), and 404 (no content to compare). To suppress flakiness-induced false positives, a differing field $f$ is only reported if its value equals what the failed request payload sent ($A.f = M.f \neq B.f$).
• Repeated Create PUT (902) / Misleading Create PUT (903): two complementary checks that a PUT returning 201 actually created something new — one by re-executing the PUT, the other by first confirming existence via GET.
• Partial Update PUT (904): verifies full replacement semantics, comparing only fields declared in both the PUT input and GET output schemas to avoid false positives from write-only and server-generated fields.
• Non-Idempotent PUT (905): duplicates a PUT–GET pair and compares responses, restricting comparison to numeric/boolean fields and array sizes; string fields are ignored entirely because timestamps and similar values dominate flakiness. The authors argue this oracle targets a particularly severe fault class: middleware may legitimately replay idempotent PUTs, so a non-idempotent implementation (e.g., a deposit endpoint implemented as PUT) can silently duplicate side effects roughly once every $K$ calls, where $K$ might be 10,000 — a failure mode the authors report having encountered directly during EvoMaster development.
• Invalid Merge-Patch (906): exploits the null-versus-undefined distinction of RFC 7386, which is lost when JSON payloads map onto statically typed DTOs; fields absent (not null) in the PATCH payload must remain unchanged between the two bracketing GETs. If no application/merge-patch+json PATCH exists, the oracle falls back to plain application/json PATCHes on the assumption developers used the wrong media type unknowingly.
• Invalid Location (907): follows Location headers, matching relative URLs against schema endpoints with verb priority GET > DELETE > POST > PUT > PATCH, and flags 404/405/500/501 outcomes.
• Invalid Allow (908): issues OPTIONS on each schema-declared path and diffs the Allow header against declared verbs (ignoring HEAD/OPTIONS); extra verbs may even constitute security exposure.

Integration into the fuzzing workflow

All nine oracles are implemented in EvoMaster as a post-processing phase executed after the main search completes. Given the minimized suite $N$, scenarios are constructed deterministically, so the phase's cost depends on $\alpha$0 rather than on additional search. A pre-emptive timeout capped at 10% of the fuzzing budget bounds worst-case overhead. Fault-revealing tests are appended to $\alpha$1 and emitted as executable test cases (Java, Kotlin, Python, JavaScript) with inline comments marking the detected fault type. The approach is fuzzer-agnostic in principle: any source of seed tests, including manually written ones, suffices.

Empirical evaluation

RQ1 — injected faults. Nine artificial SpringBoot/Kotlin APIs, one per oracle, were each fuzzed five times for up to one minute. All injected faults were detected in all runs, without exception. These APIs have been promoted to end-to-end regression tests inside EvoMaster's own CI pipeline.

RQ2 — real-world faults. On WFD 4.3.0 (36 open-source APIs, 1,487 endpoints, up to 183 endpoints and 143k LOC per API, including authentication and diverse databases), with five one-hour runs per API (at least 180 CPU-hours total), the techniques found 166 faults spanning 7 of the 9 types, all manually verified as genuine rather than false positives.

Fault type	Sum over 36 APIs	# APIs affected
900 Non-Working Delete	13.4	3
902 Repeated Create PUT	0.8	1
903 Misleading Create PUT	0.8	1
904 Partial Update PUT	9.9	9
905 Non-Idempotent PUT	4.6	1
907 Invalid Location	1.2	2
908 Invalid Allow	133.0	14

Invalid Allow dominates (133 of 166 findings across 14 APIs). Two oracles found nothing: Side-Effects Failed Modification (901) and Invalid Merge-Patch (906). The authors offer explanations but concede they cannot distinguish technique weakness from fault absence: PATCH adoption is simply rare in WFD, and atomic-transaction implementations make post-validation side effects uncommon in this corpus.

Two findings illustrate the value of the approach. In features-service, the excludes-constraint endpoint returns a Location header pointing to /products/{name}/constraint/{id} — missing the "s" present in the sibling requires endpoint's /constraints/ path — a genuine typo caught as an Invalid Location fault, while the awkward-but-legal DELETE-only constraint endpoint was correctly not flagged. In tracking-system, a PUT to /app/api/assignments/update appends to a collection, so replaying it grows the returned array — a Non-Idempotent PUT that could silently insert duplicates at random intervals under legitimate middleware retries.

RQ3 — overhead. The median overhead across the 36 APIs is 0.5 seconds, with an average of 5.3 seconds. Only webgoat (22 s, 204 endpoints) and proxyprint (104 s, attributable to individually slow HTTP calls rather than scenario count) exceeded 10 seconds; no API approached the 6-minute cap. Overhead correlates with 2xx endpoint coverage (average 65%, median 70.5%), since successful calls are the raw material for scenario construction — reinforcing that achieving 2xx coverage remains the binding constraint on this class of techniques.

Limitations and threats to validity

The authors identify several constraints plainly. The nine rules rest on their interpretation of the relevant RFCs, and misinterpretation cannot be fully excluded despite peer review. False-positive screening was manual and therefore error-prone. Results depend on a specific implementation, mitigated by thorough testing and open-source availability. Generalization beyond open-source APIs is unverified: whether these fault densities hold in closed-source industrial systems is an open question, which the authors propose to address through practitioner feedback from industrial deployments of EvoMaster. A further structural limitation is the dependence on the underlying fuzzer's ability to produce successful 2xx calls; endpoints that resist coverage yield no scenarios, and the two silent oracles (901, 906) leave unresolved whether the techniques or the corpus are responsible.

Conclusion

This paper extends REST API fuzzing beyond crash-based oracles with nine dynamically checked HTTP-semantics rules, realized as a deterministic, low-overhead post-processing phase requiring no manual schema preparation. The evaluation supports three claims: reliable detection of all injected faults in controlled settings; discovery of 166 verified real-world faults across all 36 WFD APIs, including critical non-idempotent PUT implementations; and negligible computational cost (median 0.5 seconds). The main open questions left by the work are the real-world prevalence of the two undetected fault categories and the transferability of these results to industrial, closed-source APIs.