MASTOR: A Multi-Agent Approach to Semantic Test Oracle Generation for RESTful APIs

Abstract: Existing automated RESTful API testing approaches commonly rely on simple checks (e.g., HTTP status codes, schema conformance), which are insufficient for detecting semantic faults, business logic violations, and state-dependent inconsistencies. To address this, we propose MASTOR, a Multi-Agent approach for generating Semantic Test Oracles for RESTful APIs based on implementation source code. MASTOR consists of two phases: source analysis and oracle generation. The former employs a source extraction agent to construct a source context for each endpoint operation by analyzing a transitive import closure of relevant source files. The latter employs two parallel oracle-generation paths over the collected contexts: a single-operation path producing status and field oracles per operation, and a multi-operation path generating behavioral consistency oracles for operation sequences by leveraging cross-operation semantic associations. Both paths apply a challenger-agent review, where a dedicated reviewer identifies weaknesses and issues improvement hints to guide targeted regeneration, followed by oracle normalization to filter out structurally invalid oracles. We evaluated MASTOR on a benchmark of 13 open-source RESTful API projects (296 operations, 251,303 lines of code) from the WFD and PRAB datasets. MASTOR achieved an average mutation score of 75.4%, generating 10,022 oracles. These oracles were translated into executable assertions via ToJUnit and ToPostmanAssertify, and into human-readable descriptions via ToReadable. In a baseline comparison on 50 selected operations, MASTOR outperformed Direct Prompting by 30.1 percentage points (69.9% vs. 39.8%) and SATORI by 49.4 percentage points (69.9% vs. 20.5%).

• The paper presents a multi-agent system (MASTOR) that grounds semantic test oracle generation directly in Java source code analysis for REST APIs.
• It employs specialized agents to extract code constraints and synthesize both per-endpoint and cross-operation oracles, improving mutation scores compared to conventional approaches.
• Empirical results demonstrate high fault detection efficacy and practical cost efficiency, supporting CI/CD integration and enhanced API testing.

MASTOR: Multi-Agent Semantic Test Oracle Generation for RESTful APIs

Motivation and Problem Setting

Test oracle generation remains a core challenge in automated RESTful API testing. Automated approaches traditionally employ superficial checks—such as HTTP status codes, runtime crashes, or schema validation—that are insufficient for detecting business logic faults, complex state-dependent behavior, or semantic inconsistencies. Semantic oracles, grounded in implementation-specific logic, are essential for accurate detection of subtle faults but require extracting non-trivial, distributed constraints from complex codebases.

Existing specification-driven or black-box methods (e.g., SATORI, AGORA+) either rely on incomplete/underspecified OpenAPI contracts or dynamic traces and are unable to synthesize assertions that faithfully represent the implementation's actual behavior, particularly with cross-operation state or non-trivial control/data dependencies. This work introduces MASTOR, a novel multi-agent system that grounds semantic test oracle generation directly in source code analysis, exclusively for Java-based REST APIs.

MASTOR Architecture and Multi-Agent Design

MASTOR implements a blackboard-style multi-agent architecture. Specialized agents handle discrete, context-bounded subtasks, sharing structured intermediate data via a centralized OutputStore, and coordinated by a deterministic orchestrator (MastorAgent). The pipeline is divided into two principal phases:

• Source Analysis:
  • SourceExtractionAgent constructs source "bundles" for each endpoint by forming the transitive import closure (controller, service, data model, etc.), extracting concrete request/response constraints and cross-referencing them against OAS entries.
  • SemanticExtractAgent identifies inter-operation semantic associations (e.g., identifier flow, nested dependencies, resource lifecycles).
• Oracle Generation:
  • SingleOpOracleAgent synthesizes per-endpoint status and field oracles along four targeted strategies (forward/backward, valid/invalid) using branch-sensitive prompts.
  • MultiOpOracleAgent produces behavioral consistency oracles for operation sequences, exploiting associations such as identifier passing through captures and binds.
  • Pairwise ChallengerAgents (SingleOpChallengerAgent, MultiOpChallengerAgent) review oracle sets, providing actionable hints for one regeneration pass.
  • Oracle normalization is applied to remove structurally invalid or unsupported oracles (e.g., untraced assertions, hallucinated parameters).

This design enforces evidence grounding at every step: only constraints directly extracted from code are used for oracle generation, and each assertion is linked back to auditable code fragments.

Figure 1: MASTOR workflow: source code is decomposed into bundles, source and semantic context are extracted for each operation, and single/multi-op agents generate oracles with challenger review in parallel.

Oracle Generation Strategies

MASTOR's oracle synthesis is explicitly diversified. Per-endpoint oracles are generated along the axes of validity (valid/invalid inputs) and reasoning direction (forward: input-to-response; backward: from desired constraint to input). This ensures comprehensive guard condition coverage (e.g., boundary values, negative checks, error handling paths) and increases path sensitivity versus monolithic LLM prompt approaches.

Cross-operation oracles address correctness in multi-step scenarios (e.g., resource lifecycle, identifier reuse) using captured response fields wired as request parameters in follow-up operations. This capability is orthogonal to per-endpoint oracles and critical for verifying business invariants that span multiple API calls.

Figure 2: Experimental procedure: subject APIs are fed into MASTOR, producing an oracle suite for mutation testing and cost analysis; comparative baselines are included for reference.

Empirical Evaluation

Benchmarks and Setup

MASTOR is evaluated on 13 open-source Java REST APIs (296 endpoints, 251,303 LoC) covering various domains and architectural patterns. Experiments measure fault detection via mutation score (MS), number of synthesized oracles, and cost (tokens, time, and monetary cost), with comparisons against Direct Prompting (DP) and SATORI baselines.

Fault Detection Effectiveness

• MASTOR achieves an average mutation score of 75.4% (range: 69.0%–95.9%), generating 10,022 total oracles.
• Status oracles account for ~61%, field oracles ~36%, and multi-operation oracles ~3%.
• Performance is best on stateless/static-data APIs and lowest on APIs with heavy authentication or complex interactive state.

Comparative Results:

• On 50 representative endpoints, MASTOR outperforms Direct Prompting by 30.1 percentage points (69.9% vs. 39.8%) and SATORI by 49.4 points (69.9% vs. 20.5%), using status/field oracles only.
• MASTOR's source analysis enables it to generate assertions for endpoints lacking OAS schemas, where SATORI fails completely.

Architecture Component Contributions

• Multi-Operation Oracle Generation: Disabling this component (i.e., single-operation only) reduces MS by 3.7 points on average, even though multi-op oracles comprise only 2.6% of oracles—demonstrating high marginal efficacy in cross-operation semantic fault detection.
• ChallengerAgent Review: Removing review reduces MS by 7.2 points, revealing that semantic error filtering and guided regeneration are critical for maintaining oracle precision. It prevents the propagation of structurally correct but logically vacuous assertions.
• Removing both simultaneously results in an 11.2 point decrease.

Computational Cost

• MASTOR's median API cost is $0.56, with costs scaling linearly by endpoint count and codebase size. The largest projects (proxyprint/tracking-system) incur the highest cost.
• The majority of cost (75%+) is spent on single-operation generation and challenger review.
• While 10–46$\times$ more expensive per API than Direct Prompting, the improvements in MS per unit oracle are substantial and the absolute costs are practical even at scale.

Implications and Future Directions

MASTOR demonstrates that a multi-agent, evidence-grounded architecture for semantic oracle generation achieves significant gains in RESTful API fault detection. The architectural choices—modular context decomposition, challenger-guided refinement, dual-phase oracle generation, and code-based grounding—systematically reduce both LLM hallucination and superficial assertion bias compared to single-pass prompting or spec-only approaches.

Practical implications include:

• Higher-quality, source-traced oracles facilitate automation in CI/CD testing and reproducible regression testing.
• Integration with existing systems is feasible via format converters (JUnit, Postman, human-readable).

Limitations and Future Work:

• Current approach is restricted to synchronous, statically-typed Java services with explicit call graphs; dynamic/reflection-heavy or asynchronous/event-driven frameworks require advanced call graph analysis.
• Extending multi-operation associations beyond static patterns (e.g., dynamic workflow discovery, stateful user journeys) could further improve cross-cutting semantic fault coverage.
• Cost/quality trade-offs could be further explored with adaptive agent orchestration or model selection.

Conclusion

MASTOR provides a robust, scalable framework for the automated generation of semantic test oracles in RESTful APIs, addressing the limits of interface-centric and black-box techniques. Its strong empirical performance and open-source release position it as a solid baseline for future research on oracle generation and semantic API verification.

---

References

For full methodology, detailed oracle formats, implementation, and empirical data, see "MASTOR: A Multi-Agent Approach to Semantic Test Oracle Generation for RESTful APIs" (2606.10465).