Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence

Abstract: Coding agents increasingly act as codebase-scale collaborators that can assist with codebase conversion, but this progress has exposed a critical weakness: agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks while violating the semantic contracts users actually care about. This problem is especially acute in codebase conversion, where prior evaluation is largely outcome-driven and therefore unstable: two implementations can match on a shallow outcome, such as a single forward loss, while diverging in gradients, optimizer behavior, or short-horizon training dynamics. We introduce T2J-Bench, a benchmark for codebase conversion that reformulates conversion as transfer under a fixed equivalence contract. A fixed verifier then compares source and converted codebases through three ordered stages: Spec (interface admissibility), Numeric (forward outputs, losses, gradients, and objective-specific tensors), and Behavioral (short training dynamics under fixed seeds). Across 355 blind conversion attempts, the best system reaches only 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%; a 4.7x token-budget spread yields only a 2.2x pass-rate spread; and all systems overestimate success by 66.6--97.8 points relative to the fixed evaluator. This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.

• The paper demonstrates that outcome-based self-validation overstates conversion success, with overall pass rates as low as 26.7% despite high specification-stage metrics.
• It introduces T2J-Bench, a multi-stage evaluation framework assessing specification, numerical, and behavioral equivalence to ensure rigorous cross-framework codebase conversion.
• The study highlights that preserving numerical and behavioral equivalence is critical, urging future agents to adopt contract-driven validation beyond superficial checks.

Rigorous Evaluation of Codebase Conversion via Observational Equivalence

Motivation and Problem Definition

As AI-driven coding agents increasingly automate large-scale software tasks, a prominent application is the automated conversion of machine learning codebases across major frameworks—particularly from PyTorch to JAX. However, existing evaluation protocols for code-conversion agents are primarily outcome-driven, focusing on shallow artifacts such as successful script execution, minimized loss on a batch, or superficial correctness metrics. This assumption of equivalence based on local success is unreliable: functionally distinct implementations may satisfy narrow local checks while fundamentally violating the true semantic contract of the codebase. This ambiguity is acute in end-to-end training codebases, where minor misalignments—such as mismatched gradients, optimizer state, or training dynamics—can result in divergent learning behavior. This paper introduces T2J-Bench as a principled, artifact-driven benchmark for codebase conversion, grounded in the concept of finite observational equivalence.

T2J-Bench: Design and Workflow

T2J-Bench formalizes codebase conversion as transfer under a fixed series of externally observable contracts, moving away from the standard practice of output-based or self-reported success. A source PyTorch codebase, implementing SFT, DPO, or PPO training for one of 15 contemporary model families, is paired with a bounded configuration. Agents receive the source along with a minimum constraint prompt $p^\star$ specifying the external training interface, but withholding evaluator internals.

Converted repositories are then verified by a fixed, external three-stage evaluator that compares source and translation through strictly bounded interactions:

1. Spec (Specification): Interface admissibility, including schema, parameter trees, and data pipeline contracts.
2. Numeric: Semantic fidelity on forward outputs, losses, computed gradients, and method-specific tensors, quantitatively compared with precise tolerance values.
3. Behavioral: Short-horizon training behavior, measured by replaying training steps under fixed seeds, and, for SFT tasks, generation agreement.

Only candidates passing all stages are credited as successful conversions; a single violation is sufficient for failure. This process ensures that conversions are not merely syntactically or structurally plausible, but semantically equivalent on externally observable axes.

Figure 1: T2J-Bench workflow, showing initial curation of codebases, conversion by agent, and rigorous multi-stage equivalence evaluation.

Experimental Framework and Main Results

The benchmark comprises 45 tasks (15 model families × 3 objective types), each subjected to conversion attempts from frontier LLM agents, including Claude, Gemini, Codex, Qwen3-Coder, and GLM5 in settings with carefully controlled scaffolding. Controlled agent attempts averaged 0.26 hours and consumed 3.1M tokens per attempt, mapping the benchmark to realistic codebase-scale migration effort.

Results reveal that overall end-to-end pass rates remain low, despite high surface-level Spec success:

• Best systems achieve only 26.7%–28.9% overall pass@1 (first-attempt) rates, even with Spec-stage rates as high as 91.1%.
• Allowing up to three attempts only raises the best overall pass rate to 46.7%.
• Preserving numerical equivalence is the primary bottleneck, as many conversions that pass Spec do not survive deeper Numeric checks.

These findings demonstrate that current coding agents systematically overstate their success: self-reported pass rates exceed external verifications by 66.6–97.8 percentage points, indicating severe overconfidence rooted in shape- rather than semantics-based validation.

Figure 2: Per-task mean token cost vs. source-codebase size shows negligible correlation between cost and tractability; agent failures are not explained by codebase size or structural difficulty.

Error Taxonomy and Failure Analysis

A detailed error analysis over 355 blind conversion attempts exposes the prevailing cross-agent failure modes:

• Interface/admissibility failures: Runtime initialization errors, parameter topology mismatches, batch-schema drift.
• Cross-paradigm errors: Type confusion at JAX/Torch boundaries, device and dtype mismatches, contract drift, missing or malformed artifacts, tensor shape errors.
• Numeric disagreement: Forward/method loss mismatches, gradient deviations, and token/generation disparities, often arising from method- and artifact-specific contract confusion.
• Behavioral divergence: Training trajectory or generation deviations, especially with divergent optimizer state or unaligned replay logic.

Crucially, neither token budget nor backbone capability alone explains outcomes. Compute budget buys more activity, not correctness. Within a given task, failing attempts do not systematically consume more tokens than successful ones, and repeated local experimentation does not improve external pass rates. The principal source of wasted effort is agent looping when local validators return positive signals on semantically flawed candidates, rather than genuine progress toward contract-true conversions.

Implications and Future Directions

This work delivers several key implications for research on coding agents and codebase migration:

• Local Validators Are Misaligned: The prevalence of surface-level self-testing results in chronic overconfidence and silent semantic regressions. Future agents must internalize source-to-translation comparisons across all external artifacts, not just internal plausibility checks.
• Numerical and Behavioral Equivalence Are Critical: Outcome-only benchmarks are fundamentally insufficient. Future efforts should focus on contract-level verification that extends deep into quantitative and dynamic codebase properties.
• Framework Conversion Dominated by Seam Errors: Most numerical and behavioral failures stem from boundary layer issues (type, device, dtype) at the intersection of source and target frameworks. Explicitly typed conversion layers and disciplined contract enforcement are prerequisites for robust cross-framework conversion systems.
• Scaling Alone Is Insufficient: Improvements in model backbone, prompt size, or token budget are not sufficient to bridge the semantic validation gap; system and protocol design must prioritize contract-mirroring.

Theoretically, T2J-Bench clarifies the practical undecidability of codebase equivalence but demonstrates that rigorously bounded, externally contract-driven evaluation is both practical and discriminative. Practically, including fixed observational benchmarks in codebase migration pipelines provides a clear diagnostic path for agent debugging and progress measurement.

Conclusion

T2J-Bench redefines the standard for codebase conversion evaluation by shifting the locus of correctness from local plausibility to fixed, externally observable contracts, rigorously enforced across specification, numerical, and behavioral dimensions. Results indicate that, in its current regime, the dominant limitation is not raw capacity or compute, but fundamental misalignment between local validation and user-desired contract equivalence. For future AI coding agents, progress will require contract-calibrated self-verification, cross-paradigm boundary discipline, and a rigorous embrace of observable semantics as the core evaluation axis.