---
title: 'Gilesi: Java API Compatibility Testing'
url: https://www.emergentmind.com/topics/gilesi
type: topic
---

# Gilesi: Java API Compatibility Testing

Searching arXiv for the Gilesi paper and closely related compatibility-testing work.
Gilesi is a prototype Java framework for client–library compatibility testing based on **API interaction snapshots**. Its purpose is to detect **behavioral breaking changes (BBCs)** in evolving libraries: changes that do not alter signatures and therefore do not necessarily cause compilation or linkage failures, but do change run-time behavior in ways that can silently break clients. The framework exploits existing client test suites as execution drivers, but shifts observation from developer-written assertions to the **client-library API boundary**, where it records invocation protocol, input and output values, exceptions, and related run-time interaction data, then compares these snapshots across library versions to identify perturbations in the client-library contract [2507.20814].

## 1. Problem setting and scope

Modern software development heavily relies on third-party libraries, and those libraries evolve. In this setting, a library maintainer may preserve the syntactic API while still changing run-time behavior in a way that breaks clients. The paper positions Gilesi specifically against such **behavioral breaking changes**, rather than compilation failures or linkage failures.

The motivating diagnosis is that ordinary client-side regression tests are often poor detectors of BBCs for two reasons. First, tests may not cover the library execution paths affected by a change. Second, even when they reach the changed code, their assertions may be too weak or too far away from the library boundary to observe the perturbation. The paper emphasizes this “distance” problem: a behavioral difference must propagate through client code and remain observable until some assertion checks it, and along that path the difference may be swallowed, ignored, or only partially checked.

Within this framing, Gilesi is not presented as a replacement for tests. Its intended role is to **augment** existing tests by extracting a more explicit behavioral contract from executions that already exist. A plausible implication is that the framework is most relevant for dependency-upgrade scenarios in which clients already have regression suites but still lack confidence that those suites encode sufficiently strong checks on library behavior.

## 2. API interaction snapshots as the compatibility model

The central abstraction in Gilesi is the **API interaction snapshot**. Rather than treating a client test as valuable only through its assertions, the framework treats the test as an execution trace over the client-library boundary. During that execution, Gilesi records what actually happens at the API boundary and stores the resulting sequence as a snapshot.

The paper formalizes a single API interaction as

\[
I = \langle m, o, \langle p_1, \dots, p_n \rangle, r \rangle
\]

where \(m\) is the API method, \(o\) is the receiver object, \(\langle p_1, \dots, p_n \rangle\) are the input parameters, and \(r\) is the result. The result is typed and may be either a normal return value or an exception. A snapshot \(S\) is then an ordered sequence of such interactions produced by one test execution [2507.20814].

This model is intentionally concrete. It records not merely that a test passed, but which API methods were invoked, on which objects, with which arguments, in what order, and with what outputs or exceptions. The paper’s representative example uses `StringTokenizer#getTokenList`. In the original execution, the method returns `ArrayList(apple, banana)`; after a library behavior change, it returns `Arrays$ArrayList(apple, banana)`. Even if a client test checks only size and element order and still passes, the snapshots differ because the concrete returned object differs. The framework therefore treats the behavior as perturbed.

This comparison criterion encodes a view of compatibility that is narrower than full semantic equivalence and broader than signature preservation. Gilesi checks whether the run-time contract actually exercised by a client remains stable at the API boundary. The paper accordingly describes any detected perturbation as a **potential BBC**, recognizing that not every observed difference must be client-harmful in all downstream contexts.

## 3. Instrumentation architecture and execution workflow

Gilesi is implemented for the **Java ecosystem**. The workflow begins by identifying the relevant part of the library API for a given client. Instead of instrumenting the entire library, the framework computes a **client-specific syntactic usage footprint** using **UCov**. This footprint contains the API symbols the client can reach and depend on, so instrumentation is restricted to the subset of exported API elements actually used by that client.

Instrumentation is performed with **Java instrumentation APIs** and **Byte Buddy** through a **run-time agent**. The agent intercepts class-loading events, identifies footprint-relevant methods, and injects probes at method entry and exit. These probes capture the receiver object, input arguments, and result. The paper explicitly states that, aside from the added probes, the method’s behavior remains unchanged [2507.20814].

The client’s existing test suite is first run against a baseline version of the library. Every direct or indirect call to an instrumented API method triggers the probes. Gilesi records the sequence of interactions observed during each test, including invocation ordering, receiver object identity, inputs, outputs, exceptions, and, more generally, any runtime interaction data observable at the API boundary.

After this baseline phase, the same test suite is rerun against the upgraded library version. Gilesi records a new set of snapshots and compares them against the baseline set. The workflow is therefore execution-driven, version-relative, and client-specific. It does not attempt to infer the complete semantics of the library; it records the concrete behavior actually exercised by the client suite.

## 4. Snapshot representation, object identity, and comparison semantics

A practical challenge in the design is serialization. Some values observed at the API boundary are straightforward to serialize, while others are complex, nested, or library-defined. For standard JDK types such as primitives and collections, Gilesi uses **XStream**. For complex objects, it may record only an object identifier rather than eagerly serializing the entire internal state. Detailed observation can then be deferred until later API calls on that object expose simpler and serializable data.

This design is closely tied to the framework’s compatibility model. The paper argues that when a library returns a complex object, what matters to the client is the behavior exposed when that object is subsequently used through API calls. To support that view, Gilesi assigns unique identifiers to objects so that interactions on the same receiver can be correlated across the execution. The result is a trace model that preserves both **protocol** and **identity** information without requiring a complete internal snapshot of arbitrary object graphs.

Comparison proceeds in two stages. Gilesi first checks whether the interaction ordering is identical. A protocol mismatch is itself treated as a perturbation. If ordering matches, interactions are compared pairwise to determine whether the same method on the same object with the same inputs yields the same outputs or exceptions. Differences in protocol, return values, or exceptional behavior are then reported as signs that the client-library contract has been perturbed [2507.20814].

The paper attributes several benefits to this run-time grounding. Because Gilesi records actual invocations, it naturally captures reflective calls. It can also detect problems that originate in transitive dependencies, provided their effects propagate to the direct library API boundary used by the client. In addition, before/after snapshots provide a concrete debugging artifact rather than leaving maintainers to infer the cause of a compatibility failure from a distant test assertion.

## 5. Case study and observed effectiveness

The evaluation is presented as a **preliminary case study** over two Java libraries: **Jsoup** with **6 clients** and **Commons-Lang3** with **21 clients**, for a total of **27 client-library pairs**. The study does not use a large corpus of naturally occurring historical BBCs. Instead, it applies a mutation-based procedure in which a second Java agent introduces **extreme mutations** into reached API methods one at a time. These mutations replace the entire implementation with a trivial default return such as `null` for reference types or zero for integers.

After filtering out flaky or failing clients, the authors run tests against the original library version twice to verify test and snapshot stability and to record which API methods are reached. For each reached API method mutant, client tests are rerun. A mutant is considered killed by the client test suite if any test fails; it is considered killed by Gilesi if any resulting snapshot differs from the baseline snapshot [2507.20814].

The aggregated results reported in Table I are: **commons-lang3** with **106 mutants**, killed by tests **100** and killed by Gilesi **105**; **jsoup** with **52 mutants**, killed by tests **41** and killed by Gilesi **46**. Across both libraries, this yields **158 mutants total**, with **client tests killed 89%** and **Gilesi killed 96%**. The paper further states two summary findings: **all mutants killed by client tests were also killed by Gilesi**, and **Gilesi detected 10 extreme mutations missed by client tests**.

Two examples clarify the nature of these wins. In a **devops-comdor** client using **Commons-Lang3**, the client calls `ExceptionUtils.getStackTrace(Throwable)` when building a GitHub issue body. Under mutation, the method returns `null` instead of the stack-trace string. The client test still passes because it checks only parts of the generated issue body, but Gilesi records the changed API output and reports a perturbation. In a **chyxion-table-to-xls** client using **Jsoup**, mutation changes the behavior of `attr` during HTML parsing. The test only checks that conversion completes without crashing, whereas Gilesi detects both a different first interaction result and a changed number of interactions. This demonstrates sensitivity not only to output differences but also to protocol differences.

## 6. Limitations, interpretation, and position within compatibility testing

The paper is explicit that Gilesi’s evidence base is preliminary. It does not provide a full precision/recall analysis on real historical BBCs, detailed performance measurements, or scalability curves for large systems. The use of **artificially seeded extreme mutations**, the relatively small number of libraries and clients, and the exclusion of flaky tests and snapshots all limit how far the results can be generalized.

False negatives are discussed more concretely than false positives. Seven mutants were not detected by Gilesi. According to the paper, these missed cases arose from mutations that changed **library-side I/O side effects** without affecting observed return values, especially on `void` methods, and from **equivalent behavior** from the client’s perspective because the original method already returned the same default value. The framework also currently struggles with nondeterminism, concurrency, and side effects not cleanly reflected at the API boundary [2507.20814].

A common misconception would be to treat snapshot comparison as equivalent to complete semantic equivalence checking. The paper does not make that claim. Gilesi reports **potential BBCs**, and some snapshot differences may be tolerable to a client even if they are real behavioral perturbations. Conversely, if client tests do not execute the changed behavior, no snapshot can reveal it. Coverage dependence is therefore fundamental: Gilesi is stronger than ordinary assertions at the boundary it observes, but it cannot infer unexecuted behavior.

Within that scope, the framework occupies a specific niche. It is aimed at **public-facing library APIs used by a given client**, not arbitrary internal methods. Its best use case is a dependency-management workflow in which existing tests are already present, but additional sensitivity to behavior changes is needed before accepting a library upgrade. In that role, Gilesi provides a concrete operationalization of client-library compatibility: compare the actual boundary behavior exercised by the client before and after the upgrade, and treat perturbations in protocol, values, or exceptions as compatibility warnings requiring review.

Source: https://www.emergentmind.com/topics/gilesi