---
title: 'RestTSLLM: Automated REST API Testing'
url: https://www.emergentmind.com/topics/resttsllm
type: topic
---

# RestTSLLM: Automated REST API Testing

RestTSLLM is an approach for automating REST API testing by combining Test Specification Language (TSL) with large language models (LLMs) to generate test cases from OpenAPI specifications. It targets two core challenges in REST API test automation: the creation of test scenarios and the definition of appropriate input data. The approach uses prompt engineering techniques together with an automated pipeline to evaluate multiple LLMs on their ability to transform OpenAPI specifications into structured TSL artifacts and then into executable integration tests, with evaluation centered on success rate, test coverage, and mutation score [2509.05540].

## 1. Concept and scope

RestTSLLM addresses a setting in which REST API testing is described as vital but difficult because of system complexity, the volume of possible test scenarios, and limited time for test design. Exhaustive testing of all input combinations is treated as impractical, often resulting in undetected failures, high manual effort, and limited test coverage. Within that problem setting, RestTSLLM proposes automatic generation of integration tests for REST APIs using LLMs and TSL, with OpenAPI serving as the foundational specification artifact [2509.05540].

In RestTSLLM, TSL functions as an intermediate representation between API specification and test code. It is described as an intermediate, structured, human-readable format that specifies test cases, including inputs, behaviors, and expected outputs. This design decouples business logic and scenario reasoning from code synthesis. The LLMs are then used for two distinct transformations: OpenAPI to TSL, and TSL to executable integration tests. That two-step decomposition is central to the method’s organization and evaluation [2509.05540].

A plausible implication is that the intermediate TSL layer is intended not merely as a serialization format, but as a control point for preserving scenario semantics across generation stages. The paper’s emphasis, however, remains on practical test generation and comparative evaluation rather than on a formal semantics of TSL itself [2509.05540].

## 2. Workflow and system structure

The RestTSLLM workflow integrates prompt engineering, staged generation, and automation. The paper describes a prompt structure in three stages—Behavior, Examples, and Action—used across the two main transformations. In the Behavior stage, the LLM is prompted to embody an expert tester or developer familiar with REST APIs and testing practices such as Arrange-Act-Assert and boundary testing. In the Examples stage, the model receives worked examples for OpenAPI-to-TSL and TSL-to-integration-test generation. In the Action stage, the model performs those same transformations on the target project [2509.05540].

The operational flow can be summarized as follows:

| Step | Input | Output |
|---|---|---|
| Behavior | System prompt | Expert tester and developer role setup |
| Example | OpenAPI and TSL examples | Sample TSL and sample test code |
| Action | Real OpenAPI, then generated TSL | TSL for the API and executable integration tests |

The OpenAPI specification is the foundation of the pipeline. It provides endpoints, request and response schemas, and business rules. The LLM is prompted to interpret OpenAPI, extract business logic through the provided examples, and generate structured TSL test scenarios. The downstream TSL-to-test-code translation is described as ensuring that coverage and context in the generated test suite closely mirror the REST API contract defined by OpenAPI [2509.05540].

The paper also notes a practical mechanism for handling output-length limitations. When outputs may be truncated because of LLM token limits, prompts are split by API tags or methods. This indicates that RestTSLLM is designed as a production-oriented prompting pipeline rather than a single monolithic generation call [2509.05540].

## 3. TSL mediation and prompt engineering

The use of TSL is the distinctive methodological component of RestTSLLM. TSL is described as specifying test cases through fields such as identifiers, grouping, endpoint, method, preconditions, request body, and expected response. The provided example includes a login scenario with a request body containing email and password, and an expected response containing a status code and semantic assertions about response fields such as `userId`, `token`, and `refreshToken` [2509.05540].

The transformation from TSL to executable tests is exemplified with generated integration test code in C# using xUnit. The sample follows an Arrange-Act-Assert pattern, including setup of input data, invocation of API operations such as login, and assertions over the response. This reflects the paper’s claim that the approach produces executable integration tests rather than only abstract scenarios [2509.05540].

Prompt engineering combines decomposed prompting and few-shot learning. Decomposed prompting breaks a complex task into more manageable subtasks, while few-shot learning supplies worked examples for both transformation steps. The paper specifies the four prompts as: Prompt 1 for OpenAPI to TSL example generation, Prompt 2 for TSL to integration test example generation, Prompt 3 for real OpenAPI to TSL generation, and Prompt 4 for generated TSL to executable integration tests [2509.05540].

This suggests that RestTSLLM treats test generation as a controlled sequence of specification transformations rather than as direct end-to-end synthesis. The paper frames that decomposition as helpful for preserving contextual coherence and producing business-meaningful as well as technically robust tests [2509.05540].

## 4. Experimental setting and evaluation methodology

The empirical study evaluates eight LLMs spanning proprietary and open-source systems, including US and international models, as well as code-focused models. The tested models are GPT-4o, Llama 3.2 90b, Claude 3.5 Sonnet, Gemini 1.5 Pro, Deepseek R1, Mistral Large, Qwen 2.5 32b, and Sabiá 3. Sabiá 3 is specifically noted for Portuguese support [2509.05540].

The benchmark consists of six open-source .NET REST API projects, all with OpenAPI specifications and with various integrations covering databases and external APIs. The automation pipeline itself is implemented with a Python script that handles prompt submission, collection of outputs, and metric evaluation for multiple LLMs [2509.05540].

The study uses three quantitative metrics: Success Rate, Coverage, and Mutation Score. Success Rate is defined as the percentage of tests executed without failure. Coverage is branch coverage, defined as the percent of code branches executed by the generated tests. Mutation Score is the proportion of artificially mutated code branches, treated as faults, that are detected by the tests. A composite score \( S \) is also reported using the TOPSIS MCDM technique:

$$
S = w \cdot SuccessRate + w \cdot Coverage + w \cdot MutationScore
$$

with \( w = 33.\overline{3}\% \) for each metric [2509.05540].

Additional tracking includes the number of generated tests, cost per generation, and failure categorization. The paper’s methodology figures are described as showing, respectively, the RestTSLLM workflow and the broader experimental flow involving model selection, project selection, prompt engineering, execution, and evaluation [2509.05540].

## 5. Comparative results

The reported results indicate that all evaluated LLMs produced functional, contextually appropriate tests. The best-performing models are Claude 3.5 Sonnet, Deepseek R1, Qwen 2.5 32b, and Sabiá 3, which are described as consistently producing robust and contextually coherent REST API tests. Among them, Claude 3.5 Sonnet outperformed all other models across every metric and is identified in the study as the most suitable model for this task [2509.05540].

The paper reports the following excerpted results:

| Model | Score (S) | Success | Coverage |
|---|---:|---:|---:|
| Claude 3.5 Sonnet | 70.9% | 100% | 71.7% |
| Deepseek R1 | 67.1% | 97.0% | 67.5% |
| Qwen 2.5 32b | 65.8% | 95.5% | 68.7% |
| Sabiá 3 | 65.5% | 97.5% | 64.3% |

For the same models, the reported mutation scores are 40.8% for Claude 3.5 Sonnet, 36.9% for Deepseek R1, 33.3% for Qwen 2.5 32b, and 34.7% for Sabiá 3. The corresponding average generation costs are reported as \$0.47, \$0.78, \$0.09, and \$0.08, respectively [2509.05540].

The lower-scoring models are Mistral Large, Gemini 1.5 Pro, GPT-4o, and Llama 3.2, although the paper notes that these still scored above 62% on the composite metric \( S \). Only 2.38% of generated tests failed overall. Out of 1,635 generated tests, 39 failed, and Claude 3.5 Sonnet had zero failed tests. The paper characterizes the failed tests as largely interpretable and correctable in post-processing [2509.05540].

A plausible implication is that the performance differences arise not only from raw coding ability, but also from variation in how effectively models infer business rules and constraints from OpenAPI descriptions. The study itself emphasizes observed outcomes rather than a causal decomposition of model capabilities [2509.05540].

## 6. Failure modes, significance, and limitations

The failure analysis identifies several recurring error types: boundary values outside the allowed range, misinterpretation of specification logic, missing or incorrect authentication, incorrect usage of required or optional fields, missing required characters such as password composition requirements, and JSON deserialization failures. These categories situate the observed failures at the boundary between specification comprehension and executable test synthesis [2509.05540].

The study argues that RestTSLLM has several practical implications for development workflows. It is described as enabling rapid, consistent, and spec-driven test generation; allowing LLMs to understand API contracts directly from OpenAPI; reducing manual effort; and increasing coverage. The TSL intermediate step is presented as helping to guide generation toward both business-meaningful and technically robust tests. The paper further notes that automated pipelines of this kind can be integrated into CI/CD and adapted to other technologies, projects, and languages through generalized prompting and script infrastructure [2509.05540].

The approach is also characterized as cost-effective, with very low per-project cost, especially for open-source models. The artifacts—code, scripts, prompts, and results—are reported as openly available, supporting replication and extension [2509.05540].

The limitations are explicitly bounded. Evaluation is limited to six .NET OpenAPI-based projects, so industrial or legacy contexts may introduce untested challenges. The method also depends on well-crafted OpenAPI specifications and prompt design; incomplete or ambiguous specifications limit output quality. The stated future directions are expansion to more languages, technologies, and larger or more complex projects, along with addressing prompt and token limitations, developing IDE plugins, and improving automated error handling [2509.05540].

Source: https://www.emergentmind.com/topics/resttsllm