---
title: 'RTCE: RoundTrip Code Evaluation Framework'
url: https://www.emergentmind.com/topics/roundtripcodeeval-rtce
type: topic
---

# RTCE: RoundTrip Code Evaluation Framework

RoundTripCodeEval (RTCE) is a benchmarking framework for evaluating the mechanistic code reasoning and invertibility capacities of large language models (LLMs) through execution-free, bijection-based round-trip code execution tasks. By explicitly assessing whether models preserve a strict one-to-one mapping between encoding and decoding operations over canonical compression algorithms, RTCE exposes the limits of current Code-LLMs’ consistency and internal coherence, surfacing phenomena that standard code generation or I/O prediction benchmarks fail to capture [2601.13398].

## 1. Foundational Principles: Round-Trip Consistency and Bijection

RTCE is predicated on the exact recovery property inherent to bijections: for a bijective transformation between input domain $\mathcal X$ and code domain $\mathcal Z$ via an encoder $enc$ and decoder $dec$, true round-trip consistency demands
\[
\forall x\in\mathcal X,\quad dec\bigl(enc(x)\bigr)\;=\;x
\]
RTCE evaluates whether an LLM can simulate both $enc$ and $dec$—even in inversion directions—without ever executing code, requiring that its predictions satisfy this property exactly, not merely up to functional or semantic equivalence.

Traditional code evaluation frameworks focus on I/O mapping or execution reasoning, constraining assessment to forward prediction. RTCE, by contrast, tests bidirectional reasoning: the ability to mentally invert transformations and verify full-cycle fidelity. This paradigm exposes superficial pattern-matching and memorization, demanding instead mechanistic understanding of algorithmic bijections.

## 2. RTCE Benchmark Task Suite

The RTCE benchmark consists of four execution reasoning tasks centered on canonical compression schemes: Run-Length Encoding (RLE), LZW, Arithmetic Encoding, and Huffman Coding. For each, reference $enc$ and $dec$ functions are provided. The tasks—explicitly formalized—are:

| Task                   | Input Provided             | Function Access | Required Prediction                             |
|------------------------|---------------------------|----------------|-----------------------------------------------|
| Output Prediction      | $x$                       | $enc$          | $\hat z \approx enc(x)$                      |
| Input Prediction-Inv   | $z$                       | $enc$          | $\hat x' \approx dec(z)$ (must invert $enc$) |
| Output Prediction-Inv  | $x$                       | $dec$          | $\hat z \approx enc(x)$ (must invert $dec$)  |
| Input Prediction       | $z$                       | $dec$          | $\hat x' \approx dec(z)$                     |

Tasks 2 and 3 require the model to perform mental inversion of the provided function, testing its ability to “walk back” nontrivial algorithms. All tasks are single-step; the combined sequence forms a round-trip: $x \xrightarrow{enc} z \xrightarrow{dec} x'$ [2601.13398].

## 3. Metrics, Evaluation, and Differentiation

RTCE employs a suite of metrics to diagnose specific failure modes and gauge bidirectional fidelity:

- **Exact Match (EM):**
  \[
  EM \;=\;\frac1N\sum_{i=1}^N \mathbb{1}[\hat y_i=y_i]
  \]
  Demands strict equivalence of predicted and ground-truth codes or reconstructions.
- **Edit Similarity (ES):** Quantifies normalized Levenshtein similarity, useful for identifying “near misses.”
- **Pass@5:** Probability that at least one of the top-5 sampled completions exactly matches the reference.

All evaluation is execution-free: LLMs are never permitted to run or step through code; outputs are compared to reference artifacts (strings, tuples, bit-packed values) [2601.13398].

This differentiates RTCE from:
- I/O-prediction (forward-only) benchmarks which allow massive leeway for overfitting,
- Execution-reasoning tests, which do not probe invertibility,
- Natural-language round-trips (e.g., code → comment → code) that assess semantic but not syntactic or algorithmic fidelity.

The bijection test inherent to RTCE enforces that only perfect, mechanistic reasoning can guarantee success.

## 4. Experimental Setup and Quantitative Findings

A broad sweep of 20+ open-source Code-LLMs was evaluated, from 1B to 70B parameters, including Llama-3, Phi-3, Qwen2.5, CodeLlama, Mistral, and others. Three principal protocols were tested:

- **Zero-Shot Prompting:** Models receive the reference function and an explicit “self-consistency” instruction for reasoning.
- **Self-Reflection / Multi-Turn Revision:** The model critiques and edits its own outputs via two JSON-guided rounds, terminating on EM.
- **Supervised Fine-Tuning:** QwQ-32B was fine-tuned (LoRA, rank 8, 3 epochs) on natural language execution trace chains derived from reference solutions.

### Performance Hierarchy and Error Analysis

Empirically, forward Output Prediction (given $x$, predict $enc(x)$) is the easiest; Output Prediction-Inv, Input Prediction, and Input Prediction-Inv increase in difficulty, with inversion of decoding (Input Prediction-Inv) being hardest.

- Models $\mathbf{< 3B}$: near-zero EM on all tasks.
- $7{-}9$B: moderate ES on RLE (up to 50%), but nearly zero EM for LZW, Arithmetic, and Huffman.
- $14{-}32$B: up to 57% EM (RLE), $\approx$30% (LZW), $<10\%$ (AE), and $\approx$0% (Huffman).
- $70$B: minor gains, but Huffman remains wholly unsolved (EM$\approx$0).

Self-reflection yields modest absolute gains (e.g., AE zero-shot EM 0% $\rightarrow$ two-round EM $\approx$8%) but does not close the gap for hard tasks. Fine-tuning on execution traces boosts Pass@5 (RLE: $\sim$57%→80%, LZW: $\sim$24%→78%, AE: $\sim$15%→84%) but fails to render models invertible on Huffman [2601.13398].

## 5. Pathologies and Key Insights

RTCE surfaces new classes of mechanistic reasoning failures:

- Models largely memorize or pattern-match forward transformations, lacking the internal representations needed for reliable inversion.
- Inversion tasks reveal pronounced asymmetries: models can compress but not reconstruct, or vice versa.
- Huffman Coding, requiring tree-based reasoning and bit-level fidelity, is unsolved at all current scales—with EM$\approx$0 even at 70B parameters.
- Self-reflection corrects superficial errors (e.g., off-by-one issues) but cannot induce the algorithmic state-tracking required for true bijection.
- Even extensive fine-tuning on step-by-step execution traces fails to guarantee round-trip consistency, pointing to a dissociation between procedural recall and algorithmic coherence.

These findings reveal a fundamental deficiency in current Code-LLMs’ ability to reason mechanistically over algorithmic bijections, a gap not evident in forward-only benchmark settings [2601.13398].

## 6. Dataset Construction, Reference Implementations, and Release Strategy

The RTCE dataset consists of 1,000 synthetic instances (250 per algorithm), spanning:

- Patterned strings (palindromes, pangrams, repetition, and noise),
- Structured logs (e.g., Apache access records),
- YAML-like configs (Helm, Docker, Ansible),
- Tabular CSV/TSV data (heterogeneous, sparse repeat patterns).

Each algorithm’s data family is stored as raw files and unified JSON Lines, with metadata, reference input-output pairs, and algorithm identifiers. Reference encoder and decoder implementations are included.

The release plan provides open access to tasks, prompt templates, an inference harness (vLLM), and fine-tuning scripts under an MIT license. Directory structure includes `/data/{ae,lzw,rle,huffman}/`, `/prompts/`, `/eval/`, and `/finetune/` [2601.13398].

## 7. RTCE in the Context of Broader RTC Methodology

RTCE is a rigorous extension of Round-Trip Correctness (RTC) methodology [2402.08699, 2407.16557], which generally evaluates whether forward and backward model passes (e.g., code↔natural language or patch generation↔review) preserve semantic identity. RTCE distinguishes itself by enabling execution-free, algorithm-specific, exact-match evaluation in the code domain, enforcing stricter bijection constraints than RTC formulations utilizing test-based or semantic similarity metrics. While prior RTC works have shown high empirical correlation ($r \approx 0.95$) with pass@1 and downstream accuracy on code synthesis, RTCE demonstrates that true bidirectional code understanding is a substantially more challenging task, requiring advances beyond current scaling, prompting, or SFT strategies [2601.13398, 2402.08699, 2407.16557].

## 8. Future Directions and Open Problems

RTCE highlights open research directions:

- Developing Code-LLMs with robust mechanistic invertibility and internal consistency,
- Closing the large performance gaps on complex, tree-based or bit-level algorithms,
- Integrating execution-free bijection diagnostics as a standard metric for code understanding,
- Scaling dataset complexity and breadth (multi-module, multi-step transformations).

*A plausible implication is* that improvements in algorithmic state-tracking and symbolic manipulation will be necessary to achieve nonzero RTCE scores on the hardest tasks, especially Huffman Coding. RTCE thus serves as a diagnostic and developmental benchmark for trustworthy code-oriented LLMs and the broader evaluation of round-trip reasoning in complex symbolic domains [2601.13398].

Source: https://www.emergentmind.com/topics/roundtripcodeeval-rtce