Oracle-Driven Self-Debugging

• Oracle-driven self-debugging is a method that separates candidate generation from ground-truth adjudication using an external, immutable oracle.
• It employs iterative control loops with techniques such as numerical equivalence tests, SMT constraint checks, and entailment queries to refine and validate repairs.
• The approach is applied in diverse domains like deep learning migration, ontology debugging, pipeline repair, and DBMS testing to enhance diagnostic accuracy and efficiency.

Oracle-driven self-debugging denotes a class of debugging and repair procedures in which an external or immutable oracle provides authoritative information that constrains diagnosis, test synthesis, query selection, or repair generation. In the cited literature, the oracle may take the form of a serialized execution record from a source PyTorch module, an entailment-answering oracle over candidate ontology repairs, an SMT-based remediation oracle for AutoML pipelines, or a formally verified equivalence oracle for DBMS testing. Across these settings, the shared pattern is iterative: construct a candidate, evaluate it against oracle-backed criteria, use the resulting discrepancy signal to prune diagnoses or revise the candidate, and repeat until a stopping condition is met (Liang et al., 14 Jun 2026, Shchekotykhin et al., 2010, Dolby et al., 2022, Mang et al., 8 Oct 2025).

1. Core concept and architectural pattern

Oracle-driven self-debugging is characterized by the separation of candidate generation from ground-truth adjudication. The candidate may be translated code, a diagnosis hypothesis, a repaired ML pipeline, or a semantically equivalent SQL transformation. The oracle supplies information that the generator is not trusted to infer reliably on its own. In the PyTorch-to-JAX migration setting, this is explicit: the system avoids asking the LLM to deduce mathematical outputs and instead runs the source PyTorch modules to obtain actual dynamic tensor states, yielding an “unchangeable execution oracle” (Liang et al., 14 Jun 2026).

The architecture is usually closed-loop. A candidate is produced; a validation phase invokes compilation checks, shape checks, numerical equivalence tests, entailment queries, satisfiability checks, or equivalence proofs; then the resulting tracebacks, mismatches, or posterior updates are fed back into the next iteration. In ontology debugging, the loop maintains a set of candidate diagnoses and asks an oracle whether the target ontology should entail a query; the answer updates the sets $P$ or $N$, prunes incompatible diagnoses, and recomputes probabilities (Shchekotykhin et al., 2010). In Maro, a failing pipeline is validated against an SMT constraint system, candidate repairs are synthesized, re-evaluated, and ranked under a mixed minimal-edit and predictive-performance objective (Dolby et al., 2022).

This suggests that oracle-driven self-debugging is not a single algorithmic technique but a family of control architectures organized around trusted external signals. The oracle may be human, symbolic, executable, or hybrid, but its role is the same: to anchor debugging decisions in a source of truth that is harder to hallucinate, approximate, or misuse than the candidate generator itself.

2. Oracle representations

Different subfields instantiate the oracle with different formal objects.

Domain	Oracle representation	Primary role
PyTorch $\rightarrow$ JAX migration	$$\mathcal O=\{W_{\text{pt}},\{x_i\},\{y_i^{\text{pt}}\}\}$$	Numerical ground truth
Ontology debugging	yes/no oracle over entailments	Diagnosis disambiguation
AutoML pipeline repair	SMT-based “remediation oracle”	Constraint-based repair synthesis
DBMS testing	verified equivalent CAQ pair	Sound logic-bug detection

In “Agentic Framework for Deep Learning workload migration via In-Context Learning,” the oracle is formalized as the tuple

$$\mathcal O=\bigl\{W_{\text{pt}},\{x_i\},\{y_i^{\text{pt}}\}\bigr\},$$

serialized into a “.pkl” file. It includes the initialized weight dictionary, input tensors, and resulting output activations of the original PyTorch module $f_{\text{pt}}$ under representative inputs (Liang et al., 14 Jun 2026). The essential property is immutability: the outputs are fixed numerical anchors.

In ontology debugging, the oracle is interactive rather than serialized. Queries have the form “Should the target ontology entail $\alpha$?” and partition current diagnoses into $D^+$, $D^-$, and $D^0$. The answer, supplied by a domain expert, test suite, or automated extractor, eliminates diagnoses incompatible with that answer (Shchekotykhin et al., 2010). Here the oracle does not directly repair the ontology; it resolves ambiguity among minimal diagnoses.

Maro uses the phrase “remediation oracle” differently. There the oracle is a formal validity and repair mechanism encoded by an SMT constraint system $N$0, augmented by failure constraints extracted from traces. It answers a more constructive question: what minimal edits to a pipeline restore validity while optimizing predictive performance (Dolby et al., 2022)? By contrast, Argus treats a DBMS test oracle as a procedure that maps an SQL query $N$1 to an equivalent variant $N$2, such that disagreement in results implies a logic bug. Its “equivalent CAQ pair” formalism makes the oracle reusable across instantiations while a SQL equivalence prover ensures soundness (Mang et al., 8 Oct 2025).

A common misconception is that an oracle must be a human expert. The literature explicitly includes human experts, automated extractors, runtime executions, SMT solvers, and equivalence provers. The decisive feature is not the oracle’s embodiment but its authority over the debugging loop.

3. Iterative control loops and oracle-efficient search

Oracle-driven systems differ substantially in how they spend oracle interactions. Some use direct execution-based validation, whereas others optimize question selection because oracle access is expensive or cognitively burdensome.

The PyTorch-to-JAX migration loop is a four-stage validation cascade: compilation, shape tests, numerical equivalence tests, and then LLM repair if any stage fails. The pseudocode returns immediately when all checks pass and otherwise prompts the LLM with the failure status and traceback or numerical discrepancy. Termination occurs either when all compile, shape, and numerical checks pass or when T_max is reached; empirically, 3–5 iterations are usually sufficient, and in practice the loop typically converges in 2–4 iterations for Level 1–2 modules (Liang et al., 14 Jun 2026).

Ontology debugging formulates query choice as an information-theoretic optimization problem. Given diagnosis probabilities $N$3, one chooses the query $N$4 maximizing expected information gain. The scoring function

$N$5

is minimized, with $N$6 representing diagnoses that make no prediction (Shchekotykhin et al., 2010). A related formulation in the interactive ontology literature compares this entropy-based strategy with a “split-in-half” baseline, again emphasizing that the cost of debugging depends not only on diagnosis quality but on how oracle questions are scheduled (Shchekotykhin et al., 2011).

The execution-tree literature pushes this idea further. “Optimal Divide and Query” shows that earlier divide-and-query strategies for algorithmic debugging were not optimal and could be incomplete. The new criterion selects a node whose split best balances

$N$7

or, absent an exact split, minimizes

$N$8

This is equivalent to maximizing $N$9 (Insa et al., 2011). Although this work is framed as algorithmic debugging rather than LLM-based repair, it supplies a rigorous theory of oracle-efficient query selection: the objective is to minimize the expected or worst-case number of subsequent oracle questions.

A plausible implication is that oracle-driven self-debugging has two coupled design problems: constructing a trustworthy oracle and allocating oracle interactions efficiently. The former determines correctness; the latter determines practicality.

4. Execution-oracle self-debugging for cross-framework migration

The most explicit recent instantiation appears in “Agentic Framework for Deep Learning workload migration via In-Context Learning” (Liang et al., 14 Jun 2026). The target problem is automated migration from PyTorch’s flexible, object-oriented design to JAX’s functional, stateless setup. The paper attributes failure of naive automation to strict and dynamic API alignment and to LLM mistakes on exacting operations.

The system combines three elements. First, it curates an ICL context that serves as a strict reference for idiomatic JAX styling and test case generation. The prompt contains a small set of 4–6 reference pairs, each pairing a PyTorch snippet with its idiomatic JAX/Flax translation and accompanying minimal “silver” test cases consisting of compile-and-shape checks plus numpy.allclose with $\rightarrow$0. The prompt also enforces a style guide: flax.linen Module subclassing, explicit jax.random.PRNGKey splits, and jax.lax.scan for loops where necessary. Second, the execution oracle supplies fixed weights, inputs, and output activations from the source module. Third, an autonomous agentic loop synthesizes tests from the oracle, executes them repeatedly, and feeds the traceback back into the LLM for self-correction.

The numerical equivalence criterion is defined per input $\rightarrow$1 as

$\rightarrow$2

with a module passing when

$\rightarrow$3

Over a suite of $\rightarrow$4 modules, the numerical equivalence rate is

$\rightarrow$5

The ablation study reports the following results.

Configuration	Level 1 Comp/Shape/NumEq	Level 2 Comp/Shape/NumEq
Baseline Only	44% / 44% / 44%	18% / 18% / 9%
Instruction Only	67% / 67% / 44%	36% / 36% / 18%
Instruction + Self-Debugging	89% / 89% / 89%	73% / 55% / 27%
Full Pipeline (Oracle-Driven Ours)	100% / 100% / 100%	91% / 91% / 91%

The paper summarizes the central outcome as 91% numerical equivalence on neural modules, compared to 9% for the baseline and 27% for instruction plus self-debugging, and reports validation across several state-of-the-art models including SAM (segment anything), T5, Code Whisper amongst others (Liang et al., 14 Jun 2026). Oracle generation requires exactly one forward pass through the PyTorch module on representative inputs, each self-debugging iteration involves one JAX compile plus forward pass on the same inputs, and in practice the full pipeline adds only 2–3 additional backend runs per module. The stated interpretation is that oracle grounding plus ICL anchoring yields a dramatic gain in “true” numerical fidelity without excessive computational overhead.

5. Domain-specific manifestations

In ontology debugging, oracle-driven self-debugging is primarily a fault-localization procedure. The system starts from a faulty ontology $\rightarrow$6, optional background theory $\rightarrow$7, and sets $\rightarrow$8 and $\rightarrow$9 encoding required and forbidden entailments. Conflict sets are computed, minimal diagnoses are obtained as minimal hitting sets, and HS-TREE interleaves QUICKXPLAIN calls to compute minimal conflict sets on demand. The oracle-driven stage then asks whether the target ontology should entail selected axioms, updating $$\mathcal O=\{W_{\text{pt}},\{x_i\},\{y_i^{\text{pt}}\}\}$$0 or $$\mathcal O=\{W_{\text{pt}},\{x_i\},\{y_i^{\text{pt}}\}\}$$1 and eliminating diagnoses accordingly (Shchekotykhin et al., 2010). Experimental results show that the entropy method reduces the number of required queries by ∼50% versus “Split-in-half” on generated tests, and on real-world ontologies mildly biased priors yielded ≈20–50% fewer queries in “good” and “average” cases. The companion interactive study reports that entropy-based selection used on average 50–60% fewer queries than split-in-half and random on controlled benchmarks, and that on large stress tests entropy+CKK needed only 3–8 queries, each computed in under a minute (Shchekotykhin et al., 2011).

Maro applies the idea to pipeline repair rather than diagnosis selection. An ML pipeline is modeled as

$$\mathcal O=\{W_{\text{pt}},\{x_i\},\{y_i^{\text{pt}}\}\}$$2

and the debugging loop validates it against constraints over hyperparameters, operator interfaces, and global properties. Hyperopt proposes candidates, SMT checks satisfiability, runtime evaluation confirms success, and the best feasible repair is chosen under

$$\mathcal O=\{W_{\text{pt}},\{x_i\},\{y_i^{\text{pt}}\}\}$$3

The empirical study covers 150 real and synthetic pipelines over 10 standard UCI and Kaggle datasets and 8 operator libraries. Reported outcomes are an 85 % fix rate, mean +1.2 ppt accuracy delta, median 12 s time to resolution, and a drop from 85 % to 47 % in successful compile-time fixes when SMT checking is disabled (Dolby et al., 2022). Here the oracle is not merely adjudicative; it is generative and optimizing.

Argus realizes oracle-driven self-debugging as oracle discovery for DBMS testing. A test oracle is defined as a procedure that produces a semantically equivalent query variant $$\mathcal O=\{W_{\text{pt}},\{x_i\},\{y_i^{\text{pt}}\}\}$$4 for a given $$\mathcal O=\{W_{\text{pt}},\{x_i\},\{y_i^{\text{pt}}\}\}$$5; disagreement in results reveals a silent logic bug. Argus uses LLMs to generate equivalent Constrained Abstract Query pairs, a SQL equivalence solver to prove them sound, and a large offline snippet corpus to instantiate placeholders at scale. The prover is “sound but incomplete,” so a positive discharge guarantees no false-positive oracles (Mang et al., 8 Oct 2025). On five extensively tested DBMSs—Dolt, DuckDB, MySQL, PostgreSQL, TiDB—the system reports 40 previously unknown bugs, 35 of which are logic bugs, with 36 confirmed and 26 already fixed by the developers. The reported cost model includes N_{\mathit{CAQ}}\approx 400 discovery calls for about \$\mathcal O=\{W_{\text{pt}},\{x_i\},\{y_i^{\text{pt}}\}\}$612</code>, and instantiation cost of <code>≈ \$1 per 1 000 tests.

These cases show that oracle-driven self-debugging spans at least four distinct operational regimes: numerical equivalence enforcement, interactive diagnosis narrowing, constraint-based remediation, and metamorphic contract discovery. The invariant is not the debugging object but the existence of an oracle-backed criterion that closes the loop.

6. Limitations, failure modes, and methodological significance

The literature repeatedly emphasizes that oracle-driven methods do not remove all uncertainty. The PyTorch-to-JAX migration framework explicitly states that it does not prove formal convergence guarantees, even though the loop typically converges quickly in practice (Liang et al., 14 Jun 2026). Ontology debugging shows that entropy-based query selection depends on prior quality: with “bad” priors it can sometimes lose out to split-in-half, though only slightly in one study, and another study reports that split-in-half can marginally outperform entropy for very small numbers of required queries before entropy catches up (Shchekotykhin et al., 2010, Shchekotykhin et al., 2011).

Formal oracles may also be incomplete or costly to engineer. Argus relies on a sound but incomplete prover, so the verification stage guarantees soundness only when it succeeds (Mang et al., 8 Oct 2025). Maro acknowledges that its prototype cannot yet fix deep semantic logic errors such as mislabeled targets or concept drift, relies on hand-coded operator interface constraints, and faces performance bottlenecks when pipelines exceed ~20 operators due to SMT complexity (Dolby et al., 2022). The execution-tree literature adds a different warning: even longstanding oracle-query heuristics that appear intuitive can be suboptimal or incomplete, which is why “Optimal Divide and Query” replaces half-closest subtree heuristics with a formally optimal Up/Down balance criterion (Insa et al., 2011).

A plausible implication is that oracle-driven self-debugging shifts the principal research burden from ad hoc repair heuristics to oracle construction, oracle scheduling, and interface design. Once an oracle is trustworthy, the remaining question is how much structure can be imposed on the search process without making the oracle itself the new bottleneck. The cited work answers this differently in each domain—ICL anchoring and dynamic execution traces for deep learning migration, entropy-based querying for ontology diagnosis, SMT-guided candidate repair for AutoML pipelines, and prover-backed equivalence synthesis for DBMS testing—but all treat the oracle as the mechanism that turns debugging from unconstrained generation into evidence-driven iteration.