Gilesi: Java API Compatibility Testing

• Gilesi is a Java framework that captures API interaction snapshots to detect runtime behavioral changes not caught by traditional tests.
• It instruments client tests using Java instrumentation APIs and Byte Buddy to record API method calls, arguments, and outcomes at the library boundary.
• Gilesi augments existing regression suites by identifying perturbations in client-library contracts, thereby increasing confidence during dependency upgrades.

Searching arXiv for the Gilesi paper and closely related compatibility-testing work. Gilesi is a prototype Java framework for client–library compatibility testing based on API interaction snapshots. Its purpose is to detect behavioral breaking changes (BBCs) in evolving libraries: changes that do not alter signatures and therefore do not necessarily cause compilation or linkage failures, but do change run-time behavior in ways that can silently break clients. The framework exploits existing client test suites as execution drivers, but shifts observation from developer-written assertions to the client-library API boundary, where it records invocation protocol, input and output values, exceptions, and related run-time interaction data, then compares these snapshots across library versions to identify perturbations in the client-library contract (Monce et al., 28 Jul 2025).

1. Problem setting and scope

Modern software development heavily relies on third-party libraries, and those libraries evolve. In this setting, a library maintainer may preserve the syntactic API while still changing run-time behavior in a way that breaks clients. The paper positions Gilesi specifically against such behavioral breaking changes, rather than compilation failures or linkage failures.

The motivating diagnosis is that ordinary client-side regression tests are often poor detectors of BBCs for two reasons. First, tests may not cover the library execution paths affected by a change. Second, even when they reach the changed code, their assertions may be too weak or too far away from the library boundary to observe the perturbation. The paper emphasizes this “distance” problem: a behavioral difference must propagate through client code and remain observable until some assertion checks it, and along that path the difference may be swallowed, ignored, or only partially checked.

Within this framing, Gilesi is not presented as a replacement for tests. Its intended role is to augment existing tests by extracting a more explicit behavioral contract from executions that already exist. A plausible implication is that the framework is most relevant for dependency-upgrade scenarios in which clients already have regression suites but still lack confidence that those suites encode sufficiently strong checks on library behavior.

2. API interaction snapshots as the compatibility model

The central abstraction in Gilesi is the API interaction snapshot. Rather than treating a client test as valuable only through its assertions, the framework treats the test as an execution trace over the client-library boundary. During that execution, Gilesi records what actually happens at the API boundary and stores the resulting sequence as a snapshot.

The paper formalizes a single API interaction as

$$I = \langle m, o, \langle p_1, \dots, p_n \rangle, r \rangle$$

where $m$ is the API method, $o$ is the receiver object, $\langle p_1, \dots, p_n \rangle$ are the input parameters, and $r$ is the result. The result is typed and may be either a normal return value or an exception. A snapshot $S$ is then an ordered sequence of such interactions produced by one test execution (Monce et al., 28 Jul 2025).

This model is intentionally concrete. It records not merely that a test passed, but which API methods were invoked, on which objects, with which arguments, in what order, and with what outputs or exceptions. The paper’s representative example uses StringTokenizer#getTokenList. In the original execution, the method returns ArrayList(apple, banana); after a library behavior change, it returns Arrays$ArrayList(apple, banana). Even if a client test checks only size and element order and still passes, the snapshots differ because the concrete returned object differs. The framework therefore treats the behavior as perturbed.

This comparison criterion encodes a view of compatibility that is narrower than full semantic equivalence and broader than signature preservation. Gilesi checks whether the run-time contract actually exercised by a client remains stable at the API boundary. The paper accordingly describes any detected perturbation as a potential BBC, recognizing that not every observed difference must be client-harmful in all downstream contexts.

3. Instrumentation architecture and execution workflow

Gilesi is implemented for the Java ecosystem. The workflow begins by identifying the relevant part of the library API for a given client. Instead of instrumenting the entire library, the framework computes a client-specific syntactic usage footprint using UCov. This footprint contains the API symbols the client can reach and depend on, so instrumentation is restricted to the subset of exported API elements actually used by that client.

Instrumentation is performed with Java instrumentation APIs and Byte Buddy through a run-time agent. The agent intercepts class-loading events, identifies footprint-relevant methods, and injects probes at method entry and exit. These probes capture the receiver object, input arguments, and result. The paper explicitly states that, aside from the added probes, the method’s behavior remains unchanged (Monce et al., 28 Jul 2025).

The client’s existing test suite is first run against a baseline version of the library. Every direct or indirect call to an instrumented API method triggers the probes. Gilesi records the sequence of interactions observed during each test, including invocation ordering, receiver object identity, inputs, outputs, exceptions, and, more generally, any runtime interaction data observable at the API boundary.

After this baseline phase, the same test suite is rerun against the upgraded library version. Gilesi records a new set of snapshots and compares them against the baseline set. The workflow is therefore execution-driven, version-relative, and client-specific. It does not attempt to infer the complete semantics of the library; it records the concrete behavior actually exercised by the client suite.

4. Snapshot representation, object identity, and comparison semantics

A practical challenge in the design is serialization. Some values observed at the API boundary are straightforward to serialize, while others are complex, nested, or library-defined. For standard JDK types such as primitives and collections, Gilesi uses XStream. For complex objects, it may record only an object identifier rather than eagerly serializing the entire internal state. Detailed observation can then be deferred until later API calls on that object expose simpler and serializable data.

This design is closely tied to the framework’s compatibility model. The paper argues that when a library returns a complex object, what matters to the client is the behavior exposed when that object is subsequently used through API calls. To support that view, Gilesi assigns unique identifiers to objects so that interactions on the same receiver can be correlated across the execution. The result is a trace model that preserves both protocol and identity information without requiring a complete internal snapshot of arbitrary object graphs.

Comparison proceeds in two stages. Gilesi first checks whether the interaction ordering is identical. A protocol mismatch is itself treated as a perturbation. If ordering matches, interactions are compared pairwise to determine whether the same method on the same object with the same inputs yields the same outputs or exceptions. Differences in protocol, return values, or exceptional behavior are then reported as signs that the client-library contract has been perturbed (Monce et al., 28 Jul 2025).

The paper attributes several benefits to this run-time grounding. Because Gilesi records actual invocations, it naturally captures reflective calls. It can also detect problems that originate in transitive dependencies, provided their effects propagate to the direct library API boundary used by the client. In addition, before/after snapshots provide a concrete debugging artifact rather than leaving maintainers to infer the cause of a compatibility failure from a distant test assertion.

5. Case study and observed effectiveness

The evaluation is presented as a preliminary case study over two Java libraries: Jsoup with 6 clients and Commons-Lang3 with 21 clients, for a total of 27 client-library pairs. The study does not use a large corpus of naturally occurring historical BBCs. Instead, it applies a mutation-based procedure in which a second Java agent introduces extreme mutations into reached API methods one at a time. These mutations replace the entire implementation with a trivial default return such as null for reference types or zero for integers.

After filtering out flaky or failing clients, the authors run tests against the original library version twice to verify test and snapshot stability and to record which API methods are reached. For each reached API method mutant, client tests are rerun. A mutant is considered killed by the client test suite if any test fails; it is considered killed by Gilesi if any resulting snapshot differs from the baseline snapshot (Monce et al., 28 Jul 2025).

The aggregated results reported in Table I are: commons-lang3 with 106 mutants, killed by tests 100 and killed by Gilesi 105; jsoup with 52 mutants, killed by tests 41 and killed by Gilesi 46. Across both libraries, this yields 158 mutants total, with client tests killed 89% and Gilesi killed 96%. The paper further states two summary findings: all mutants killed by client tests were also killed by Gilesi, and Gilesi detected 10 extreme mutations missed by client tests.

Two examples clarify the nature of these wins. In a devops-comdor client using Commons-Lang3, the client calls ExceptionUtils.getStackTrace(Throwable) when building a GitHub issue body. Under mutation, the method returns null instead of the stack-trace string. The client test still passes because it checks only parts of the generated issue body, but Gilesi records the changed API output and reports a perturbation. In a chyxion-table-to-xls client using Jsoup, mutation changes the behavior of attr during HTML parsing. The test only checks that conversion completes without crashing, whereas Gilesi detects both a different first interaction result and a changed number of interactions. This demonstrates sensitivity not only to output differences but also to protocol differences.

6. Limitations, interpretation, and position within compatibility testing

The paper is explicit that Gilesi’s evidence base is preliminary. It does not provide a full precision/recall analysis on real historical BBCs, detailed performance measurements, or scalability curves for large systems. The use of artificially seeded extreme mutations, the relatively small number of libraries and clients, and the exclusion of flaky tests and snapshots all limit how far the results can be generalized.

False negatives are discussed more concretely than false positives. Seven mutants were not detected by Gilesi. According to the paper, these missed cases arose from mutations that changed library-side I/O side effects without affecting observed return values, especially on void methods, and from equivalent behavior from the client’s perspective because the original method already returned the same default value. The framework also currently struggles with nondeterminism, concurrency, and side effects not cleanly reflected at the API boundary (Monce et al., 28 Jul 2025).

A common misconception would be to treat snapshot comparison as equivalent to complete semantic equivalence checking. The paper does not make that claim. Gilesi reports potential BBCs, and some snapshot differences may be tolerable to a client even if they are real behavioral perturbations. Conversely, if client tests do not execute the changed behavior, no snapshot can reveal it. Coverage dependence is therefore fundamental: Gilesi is stronger than ordinary assertions at the boundary it observes, but it cannot infer unexecuted behavior.

Within that scope, the framework occupies a specific niche. It is aimed at public-facing library APIs used by a given client, not arbitrary internal methods. Its best use case is a dependency-management workflow in which existing tests are already present, but additional sensitivity to behavior changes is needed before accepting a library upgrade. In that role, Gilesi provides a concrete operationalization of client-library compatibility: compare the actual boundary behavior exercised by the client before and after the upgrade, and treat perturbations in protocol, values, or exceptions as compatibility warnings requiring review.