CSRO: Code-Based Oracles in AI, SMT & Testing

• CSRO is a design pattern that defines oracles as executable code, replacing opaque parameters with human-readable source code in policy synthesis.
• In SMT and synthesis, CSRO integrates external computation by compiling and executing code to generate assumptions and constraints for verification.
• For automated testing, CSRO derives Boolean test oracles from natural-language contracts, enabling precise and auditable unit tests.

Code-Space Response Oracles (CSRO) denotes a class of oracle constructions in which the oracle’s operative object is executable code rather than only opaque parameters or a purely symbolic theory. In recent arXiv usage, the term appears in three closely related but distinct settings. In multi-agent learning, CSRO names a variant of Policy-Space Response Oracles (PSRO) in which a LLM generates best-response policies directly as human-readable source code (Hennes et al., 10 Mar 2026). In satisfiability and synthesis modulo oracles, a CSRO is an oracle whose implementation compiles and/or executes code and returns assumptions or constraints to an SMT-based engine (Polgreen et al., 2021). In automated software testing, the term is also used for executable Boolean test oracles synthesized from natural-language Javadocs by an LLM and embedded into a unit-test suite (Jiang et al., 2024).

1. Terminological scope and core idea

Across these usages, the shared idea is that the oracle “lives in code space”: it either produces code, executes code, or is itself expressed as executable code. What changes across domains is the surrounding search procedure and the semantic role of the oracle.

Context	CSRO meaning	Response form
Multi-agent learning	LLM-based oracle replacing the RL best-response step in PSRO	Executable policy code
SMTO/SyMO	Black-box oracle that compiles and/or runs code	Assumptions and constraints
Javadoc-based testing	Executable oracle in a unit-test suite	Boolean verdict or exceptional-behavior check

This terminological variation matters. In the multi-agent formulation, the emphasis is on interpretable policy synthesis; in SMTO/SyMO, it is on integrating external computation into satisfiability and synthesis; in software testing, it is on deriving executable checks from natural-language contracts. A plausible implication is that “code-space” functions less as a single standardized formalism than as a recurring design pattern for oracle construction.

2. CSRO in multi-agent learning

In the multi-agent setting, CSRO is introduced for finite games

$$G = \bigl(N,\{A_i\}_{i\in N},\{u_i\}_{i\in N}\bigr),$$

with emphasis on two-player symmetric zero-sum games satisfying

$$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$

The framework builds directly on PSRO, which maintains a growing policy set $P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$, computes an empirical payoff matrix $U_{ij}=u(\pi_i,\pi_j)$, solves for a meta-equilibrium mixed strategy $\sigma\in\Delta^{|P^k|}$, and then computes a best response

$$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$

CSRO replaces that oracle step with code generation:

$$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$

The prompt encodes a natural-language description of the game rules, a minimal API for a policy class or function, the current meta-strategy $\sigma$, and instructions to produce well-commented, human-readable code (Hennes et al., 10 Mar 2026).

The paper studies three oracle constructions. ZeroShot uses a single prompt with no feedback loop. LinearRefinement evaluates the generated policy against the current meta-strategy and, when the expected return is negative, re-prompts the LLM with feedback such as “Your code lost with expected return $u$. Please improve.” up to a refinement budget $M$. AlphaEvolve introduces a distributed, population-based evolutionary loop over program variants, with fitness

$$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$0

together with mutation, selection, clustering, and parallel mutation threads.

The conceptual shift is explicit: instead of optimizing opaque policy parameters, the oracle synthesizes algorithmic behavior as source code. The stated motivation is interpretability, trust, and debuggability, together with the possibility that pretrained LLM knowledge may discover complex, human-like strategies.

3. Oracle interfaces in satisfiability and synthesis modulo oracles

In the SMTO/SyMO literature, a CSRO is defined through the general oracle interface

$$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$1

where $$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$2 is the query domain, $$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$3 is the response codomain, $$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$4 is an assumption generator, and $$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$5 is a constraint generator. Given concrete query values $$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$6, the black-box oracle $$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$7 returns response values $$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$8, after which the system records $$u_1(a_1,a_2) = -\,u_2(a_2,a_1), \qquad A_1=A_2=\Pi.$$9 and $P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$0 in the proof state (Polgreen et al., 2021).

A definitional SMTO problem is written as $P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$1, where $P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$2 are oracle function symbols and $P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$3 is a quantifier-free formula in the background theory $P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$4. The satisfiability judgment $P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$5 means that $P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$6 is satisfiable modulo $P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$7 and the oracles $P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$8. In the synthesis setting, the judgment

$P^k=\{\pi_1,\dots,\pi_{|P^k|}\}\subset\Pi$9

states that, under synthesis constraints $U_{ij}=u(\pi_i,\pi_j)$0 and oracles $U_{ij}=u(\pi_i,\pi_j)$1, the synthesizer produces candidate definitions $U_{ij}=u(\pi_i,\pi_j)$2 satisfying $U_{ij}=u(\pi_i,\pi_j)$3.

Within that framework, a CSRO is “simply an instance” of $U_{ij}=u(\pi_i,\pi_j)$4 whose black-box implementation actually compiles and/or executes user-provided code, or accesses a system library, image-processing pipeline, protocol checker, or similar external artifact. The contrast with a theory oracle is explicit: a theory oracle is white-box because its semantics are fully inside $U_{ij}=u(\pi_i,\pi_j)$5, whereas a CSRO is black-box because the SMT solver cannot peek inside and only learns through assumptions and constraints. In definitional SMTO, the main recorded fact is often an equality such as $U_{ij}=u(\pi_i,\pi_j)$6; in SyMO, $U_{ij}=u(\pi_i,\pi_j)$7 becomes central because it drives search.

The associated SMTO algorithm alternates between SMT solving over $U_{ij}=u(\pi_i,\pi_j)$8, partial evaluation under the current model, and resolution of remaining oracle applications by querying or reusing stored equalities. If oracle domains are finite and the background theory is decidable, termination is guaranteed because the assumption set eventually saturates. If oracle domains are infinite, termination is not guaranteed.

4. Executable test oracles from natural-language contracts

In automated testing for Java libraries, the term CSRO is used for executable oracle functions synthesized from Javadocs. Let a method have signature

$U_{ij}=u(\pi_i,\pi_j)$9

and let $\sigma\in\Delta^{|P^k|}$0 be the semantic properties described in its Javadoc. A Code-Space Response Oracle for a property $\sigma\in\Delta^{|P^k|}$1 is defined as

$\sigma\in\Delta^{|P^k|}$2

with $\sigma\in\Delta^{|P^k|}$3 exactly when execution on those inputs satisfies the natural-language requirement $\sigma\in\Delta^{|P^k|}$4. The full oracle suite is

$\sigma\in\Delta^{|P^k|}$5

Each oracle is realized in Java as a method returning boolean, or as an exceptional-behavior check using try-catch (Jiang et al., 2024).

The construction pipeline has four components wrapped around Javadoc partitioning. First, the Javadocs are partitioned at method level, augmented by descriptions of closely related methods found in “See Also” sections. Second, the LLM is framed with an assistant role such as “You are a Software Testing Engineer…”. Third, one to three few-shot examples teach output format and style. Fourth, chain-of-thought prompting is used to separate feature identification from oracle generation: enumerate semantic features, generate one oracle per feature, and handle exceptional and boundary cases with try-catch.

The generated examples illustrate the intended form. For java.lang.Object.equals(Object), the system produces oracles such as checkReflexive, checkSymmetric, and checkEqualsHashCodeConsistency. For String.codePointAt(int), it produces an oracle that returns the expected Boolean condition when IndexOutOfBoundsException is or is not thrown. For Map.forEach, it generates a concurrent-modification exception oracle that mutates the map during iteration and checks for ConcurrentModificationException. The emphasis is on executable, human-readable assertions rather than natural-language test plans.

5. Empirical performance and interpretability

The multi-agent CSRO paper evaluates repeated Rock-Paper-Scissors (RRPS) against 43 hand-coded competition bots and Repeated Leduc hold’em poker against a CFR+ Nash policy, AlwaysCall, and AlwaysFold. The metrics are

$\sigma\in\Delta^{|P^k|}$6

$\sigma\in\Delta^{|P^k|}$7

and

$\sigma\in\Delta^{|P^k|}$8

The baselines include PSRO with IMPALA-LSTM oracle (“PSRO-IMPALA”), an LLM-agent playing one-step best-response to predicted next move, Tabular Q-learning, Contextual RM, and CFR+ (Hennes et al., 10 Mar 2026).

In RRPS, the reported scores are heterogeneous rather than uniformly dominant. CSRO-LinearRefinement attains $\sigma\in\Delta^{|P^k|}$9, $$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$0, and $$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$1; CSRO-ZeroShot attains $$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$2, $$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$3, and $$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$4; and CSRO-AlphaEvolve attains $$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$5, $$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$6, and $$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$7. The same table reports $$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$8 for ContRM and $$\pi^* \in \arg\max_{\pi\in\Pi}\;\mathbb{E}_{j\sim\sigma}\bigl[u(\pi,\pi_j)\bigr].$$9 for LLM-Agent (70B), while PSRO-IMPALA records $$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$0. In Repeated Leduc, CSRO-AlphaEvolve records $$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$1, $$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$2, and $$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$3, exceeding CFR+ at $$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$4 and substantially outperforming PSRO-IMPALA at $$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$5.

Interpretability is analyzed through concrete policy excerpts. An RRPS policy generated by LinearRefinement exposes an ensemble of named experts such as markov2, freq, and meta_imitation, an exponential decay factor 0.985, and a vote weighting rule score^5; one can inspect which predictor contributed to each action. A Repeated Leduc policy generated by AlphaEvolve reduces decision-making to two documented subroutines, _estimate_equity and _model_fold_prob, together with explicit expected-value comparisons between RAISE and CALL. The paper’s claim is not merely that these policies are executable, but that they are auditable as code.

The Javadoc-oracle paper reports evaluation on five core JDK types: java.lang.Object with 11 methods, java.lang.String with 77, java.util.Set with 20, List with 32, and Map with 25. Using GPT-4 with temperature $$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$6 via ChatGPT, it generated 428 oracles; 418 compiled without change, and 423 were semantically correct even if a helper class was required. Across all five types, 390 documented properties yielded 352 generated oracles, corresponding to $$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$7 recall, with 338 correct checks and $$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$8 precision. For documented exception cases, 182 cases yielded 180 generated oracles, corresponding to $$\pi' = \mathrm{LLM\_Oracle}\bigl(\mathrm{Prompt}(G,\sigma,P)\bigr).$$9 recall, with 175 correct exception detections and $\sigma$0 precision. The reported code-quality characteristics include human-readable method names, embedded Javadoc comments, and logical structure using ternary expressions and try-catch (Jiang et al., 2024).

6. Limitations, design trade-offs, and recurring misconceptions

The SMTO/SyMO treatment makes the principal theoretical limitation explicit: if a CSRO has an infinite domain, termination is not guaranteed. The paper also identifies a cost–informativeness trade-off, because each query may trigger expensive external execution. Designing $\sigma$1 and $\sigma$2 is described as an art: they must be strong enough to prune search but light enough to keep SMT agile. Non-deterministic or stateful code must be wrapped carefully, for example by fixed-seed drivers (Polgreen et al., 2021).

The Javadoc-based testing work identifies a different set of failure modes: minor type errors such as long↦int mismatches, missing parentheses in complex Boolean expressions, non-compilable oracles when the LLM invents a hypothetical helper class, and only partial coverage of concurrency exceptions. The paper also reports that removing assistant creation, few-shot examples, chain-of-thought prompting, or Javadoc partitioning causes major drops in coverage and pushes the model toward natural-language plans or hard-coded examples rather than general oracle methods.

A recurring misconception is to treat “code-space” as synonymous with full symbolic transparency. The literature does not support that simplification. In SMTO/SyMO, the oracle is black-box to the solver even though it operates by compiling or running code. In multi-agent learning, the relevant property is not solver visibility but the human readability of synthesized policies. In software testing, the oracle is neither a best-response computation nor an SMT-side black-box function, but an executable assertion mechanism in the test suite. What unifies these cases is the relocation of oracle behavior into executable artifacts; what differentiates them is whether code is the object being synthesized, the mechanism being queried, or the checker being produced.