Compiler Feedback and Control Signals

• Compiler feedback is a set of signals—from binary validity checks to detailed diagnostics—that guide iterative repair, optimization, and proof refinement.
• It underpins generate–validate–repair systems by converting raw compiler outputs into actionable insights for tasks like code synthesis and performance tuning.
• It functions as both a machine-readable control signal and a human-facing debugging aid, emphasizing the need for precise, structured feedback in error correction.

Compiler feedback is the use of diagnostics, validation results, and execution-derived signals produced by a compiler, static analyzer, verifier, optimizer, or instrumented toolchain to guide subsequent reasoning or action. In the literature considered here, the term ranges from raw compiler output and unit-test results in programming education, to pylint diagnostics in Python repository-level generation, to Lean error messages plus local goal states in theorem proving, to optimization remarks and runtime counters in compiler autotuning, and to assembler, linker, and execution failures in neural compilation (Jacobs et al., 25 Jun 2025, Bi et al., 2024, Wang et al., 3 Feb 2026, Li et al., 18 Jun 2026, Zhang et al., 26 May 2025). Across these settings, compiler feedback functions less as a single data type than as a family of control signals for iterative repair, search, ranking, and pedagogy.

1. Signal forms and feedback representations

A first technical distinction is between binary validity signals and structured diagnostics. CompCoder uses compiler feedback as a Boolean indicator of whether generated code is compilable, formalized as $\mathds{1}_{\operatorname{Compiler}(c)}$, and then reuses that signal both as an RL reward and as labels for a compilability discriminator (Wang et al., 2022). By contrast, the LLVM code-size optimization work “Compiler generated feedback for LLMs” constructs a richer report after an LLM proposes optimization passes: pass validity, source and target instruction-count correctness, generated IR compilability, compiler error messages, and BLEU between generated optimized IR and compiler-produced IR (Grubisic et al., 2024). The practical difference is that binary feedback can separate valid from invalid outputs, whereas structured feedback can expose how a proposal diverged from compiler reality.

A second distinction concerns what counts as “compiler” feedback in dynamic or proof-oriented languages. In ProCoder, the term is explicitly “slightly overloaded” because the implementation uses pylint rather than an ahead-of-time compiler; the resulting feedback nevertheless behaves like compile-time diagnostics by identifying UNDEF, API, OBJECT, FUNC, and OTHER error classes, with static categories tied to concrete pylint codes such as E0401, E1101, E0611, E1121, E1120, E1102, and E1136 (Bi et al., 2024). Lean proof repair similarly treats compiler output as a combination of error messages, failure locations, and local goal states rather than a simple parse/typecheck outcome (Wang et al., 3 Feb 2026). In educational settings, Tutor Kai surfaces compiler feedback as “compiler output” after code execution, alongside unit-test results, and positions it as the first post-execution information source before GenAI feedback can be requested (Jacobs et al., 25 Jun 2025).

A third distinction is between human-oriented prose remarks and machine-oriented actionable analysis. AutoPass consumes LLVM optimization remarks from -Rpass, -Rpass-missed, and -Rpass-analysis, together with IR snapshots, compact structural features, and measured runtime evidence (Li et al., 18 Jun 2026). “AI Coding Agents Need Better Compiler Remarks” argues that current optimization remarks are often “unstructured, lossy” and designed for “human intuition rather than machine logic”; the paper contrasts ambiguous remarks such as “value that could not be identified as reduction is used outside the loop” with precise dependence remarks that include dependence class, source locations, and a suggested refactoring (Deo et al., 15 Apr 2026). This supports a general taxonomy in which compiler feedback differs not only in source and granularity, but also in whether it is actionable enough to support automated repair.

2. Feedback-loop architectures

Most modern systems use compiler feedback inside an explicit generate–validate–repair loop. In the ABAP benchmark, the loop is: send a task prompt to an LLM, insert the returned code into a newly created class in the SAP server, save and activate it, create a predefined ABAP unit test class, execute syntax/activation checks and then unit tests, return the SAP error message to the model if an error occurs, and repeat for up to five feedback iterations; each task is repeated ten times, yielding $10 \times 180 = 1800$ runs per model (Wallraven et al., 21 Jan 2026). ReFuzzer follows an analogous but fuzzing-oriented loop: compile each generated C test with Clang at -O0, collect a compilation log, run sanitizer-based dynamic checks, feed the invalid program plus the log into a local LLM, revalidate the revised program, and repeat for two refuzzing attempts per test program (Shree et al., 5 Aug 2025). LEGO-Compiler generalizes this pattern across multiple intermediate artifacts and a final assembly stage, with self-correction triggered when any earlier step fails and a retry budget set to $k=5$ during evaluation (Zhang et al., 26 May 2025).

Several papers refine the loop by changing the representation of feedback before it returns to the model. “Compile to Compress” injects Lean error messages directly into the proof text with <error> ... </error> tags, preserving local code context and precise error location rather than appending diagnostics globally at the end (Li et al., 13 Mar 2026). HintPilot accumulates compilation and profiling outcomes in a feedback history $H_{feedback}$, then rebuilds the next prompt as $$Prompt_t \leftarrow ConstructPrompt(P_{src}, S_{struct}, D_{rag}, H_{feedback})$$, so later hint synthesis sees both the program and summarized “bad hint sets and bad logs” (Jiang et al., 16 Apr 2026). WhiteFox uses optimization-triggering tests as feedback examples: once instrumented logging shows that a generated program triggered the target optimization, that program becomes a candidate few-shot exemplar for future rounds, and Thompson Sampling over Beta posteriors selects which successful examples to include in the next prompt (Yang et al., 2023).

These architectures differ in whether they are single-shot with verification, multi-round with prompt augmentation, or policy-optimization loops. The unifying feature is that compiler or verifier outputs are not terminal judgments; they are transformed into new conditioning context, new search bias, or new gradient signal.

3. Program synthesis, repair, and translation

In repository-aware code generation, compiler feedback often serves as an error-directed retrieval signal rather than merely a pass/fail filter. ProCoder begins with a natural-language requirement and surrounding code, generates a candidate, checks it in project context, identifies context-related errors, retrieves additional repository information targeted to those errors, and iterates. On CoderEval project-runnable tasks with GPT-3.5-Turbo, Pass@10 rises from 13.04 for Direct and 21.74 for RepoCoder to 39.13 for ProCoder; averaged across CoderEval, adding compiler feedback improves Direct from 29.13 to 40.43, ReACC from 43.48 to 46.52, and RepoCoder from 42.17 to 48.26 (Bi et al., 2024). The paper is equally clear that compiler feedback alone is insufficient: on project-runnable GPT-3.5-Turbo tasks, “w/ CF” and “w/ CF+SQL” both reach 30.43 at Pass@10, below “w/ CF+Semantic” at 34.78 and full ProCoder at 39.13 (Bi et al., 2024).

In low-resource or enterprise languages, compiler feedback frequently acts as the main route from weak one-shot generation to usable code. The ABAP study shows cumulative success rising from 19.28% at Round 0 to 77.11% at Round 5 for GPT-5, from 24.11% to 74.67% for Claude-Sonnet-4, and from 1.44% to 46.17% for GPT-OSS-120B, with the largest gains in the earliest rounds and diminishing returns afterward (Wallraven et al., 21 Jan 2026). The paper also shows why this works: most failures are syntax-stage failures, and error-type breakdowns are dominated by declaration, lexical/token, structural, and type/conversion problems rather than deep algorithmic defects (Wallraven et al., 21 Jan 2026). A similar pattern appears in generic code correction on LeetCode challenge problems: Compile Error in Java is fixed 88.2% overall, Runtime Error in Python 82.8%, but Wrong Answer only about 33.6% in Python and 35.1% in Java, while Time Limit Exceeded is corrected only 18.6% and 21.4% respectively (Zhang et al., 16 Jun 2026).

Compiler feedback has also been used as a direct RL signal. CoTran translates whole programs between Java and Python and defines a fine-grained compiler reward $\omega_{\text{compiler}}(\widehat{\mathbf{t}})$ that is $+2$ if the generated program compiles and otherwise depends on the position of the first syntax error normalized by output length, with an additional Gaussian penalty for length mismatch (Jana et al., 2023). The final system reaches 48.68% FEqAcc and 76.98% CompAcc for Python-to-Java, improving over the CodeT5 baseline by +14.89% FEqAcc and +8.14% CompAcc (Jana et al., 2023). RLCF similarly coarse-tunes code LLMs by treating the compiler as a grounding function: if the generated Java does not compile, the rollout is truncated at the earliest error location and assigned reward $-1$; only compilable outputs are then scored by a separate discriminator model (Jain et al., 2023). On MBJP, RLCF raises CodeGen 350M Comp@1 from 60.68 to 71.82 and Pass@10 from 8.64 to 10.15, while making smaller models competitive with much larger ones (Jain et al., 2023).

The RL literature also highlights limitations of naive compiler-feedback training on long programs. StepCoder argues that full-program RL suffers from sparse terminal rewards and imprecise credit assignment, then introduces CCCS and FGO to address those issues. Its reward function is explicitly tiered: $+1$ for passing all unit tests, $-0.3$ for failing any unit test, $10 \times 180 = 1800$0 for runtime error, and $10 \times 180 = 1800$1 for compile error (Dou et al., 2024). FGO then computes a token mask $10 \times 180 = 1800$2 indicating whether each token in the generated solution was actually executed, and masks out unexecuted tokens during optimization (Dou et al., 2024). On APPS+ pass@1, StepCoder reaches 36.1 overall, compared with 31.7 for vanilla PPO, 32.1 for PPOCoder, and 32.7 for RLTF (Dou et al., 2024).

Even coarse binary feedback can be highly effective when integrated carefully. CompCoder improves average compilation rate in Python code completion from 44.18 to 89.18 relative to CodeGPT, and improves text-to-code generation compilation rate from 70.3 to 96.2, using a three-stage pipeline of language-model fine-tuning, compilability reinforcement, and compilability discrimination (Wang et al., 2022). JCoffee pushes the idea further toward program repair of incomplete artifacts: it repeatedly compiles partial Java snippets, interprets compiler diagnostics, and synthesizes missing classes, fields, methods, constructors, wrappers, and annotations until the fragment becomes compilable or a maximum number of iterations is reached (Gupta et al., 2020).

4. Formal verification and proof repair

In theorem proving, compiler feedback becomes verifier feedback: exact diagnostics about why a proof attempt fails, where it fails, and what the local proof state is at that location. APRIL frames Lean proof repair as supervised learning over tuples containing an erroneous proof, compiler feedback, a repaired proof, and a natural-language diagnosis. The dataset contains 260,125 supervised tuples derived from 39,492 unique correct Lean theorems, with theorem mutations, tactic mutations, line mutations, and multi-line mutations (Wang et al., 3 Feb 2026). Finetuning on APRIL raises Qwen3-4B from 1.1% to 27.4% pass@1 repair accuracy, and the finetuned 4B model slightly exceeds Goedel-Prover-V2-32B at 26.8% under the same single-shot protocol (Wang et al., 3 Feb 2026). The same paper shows a trade-off between explanation generation and raw repair: specializing exclusively on repair raises Qwen3-4B pass@1 from 27.4% to 31.2%, but APRIL-style explanations substantially improve a downstream stronger model’s repair success from 4% to 29% in a small pipeline experiment (Wang et al., 3 Feb 2026).

“Compile to Compress” provides a complementary theoretical lens. It formalizes the verifier as a many-to-one map $10 \times 180 = 1800$3, from Lean program space $10 \times 180 = 1800$4 to compiler-message space $10 \times 180 = 1800$5, and defines the corresponding equivalence class of programs with identical feedback as $10 \times 180 = 1800$6 (Li et al., 13 Mar 2026). The paper’s key claim is that self-correction induces a chain of feedback-conditioned distributions $10 \times 180 = 1800$7, with refinement states $10 \times 180 = 1800$8, and that treating refinement as Markovian avoids the cost of ever-growing context histories (Li et al., 13 Mar 2026). Empirically, this learning-to-refine setup yields strong gains on hard theorem-proving benchmarks: on PutnamBench at budget 256, Kimina Value solves 25 problems and Goedel Value solves 110, improving substantially over their base provers (Li et al., 13 Mar 2026).

These proof-repair systems sharpen a broader point about compiler feedback. In Lean, the compiler is not merely a checker of validity; it is a compressor of failure structure. Many syntactically different failed proofs collapse to a smaller family of recurring messages such as unsolved goals, unknown identifier 'id', failed to synthesize, or type mismatch, and those failure modes can be learned as repair targets (Li et al., 13 Mar 2026). That same logic reappears outside theorem proving wherever diagnostics partition a huge search space into a smaller family of actionable repair classes.

5. Optimization, autotuning, and compiler interfaces

Compiler feedback also drives optimization itself, though the feedback objects are different: pass validity, optimization remarks, profile counters, and measured runtime. In LLVM code-size optimization, “Compiler generated feedback for LLMs” lets an LLM predict optimization passes, instruction counts, and optimized IR, then compiles the input with the predicted passes and returns a structured report. A single feedback round improves the original model from 2.87% to 3.40% improvement over -Oz, an added gain of 0.53%, but the same paper reports that simple sampling reaches about 86% of autotuner performance with 10 samples and about 98% with 100 samples, outperforming iterative feedback at higher sample counts (Grubisic et al., 2024). This is an important negative result: compiler feedback is useful as a local correction signal, but not automatically better than diverse search.

AutoPass turns compiler internals into a much richer feedback surface. It exposes LLVM optimization remarks under -Rpass, -Rpass-missed, and -Rpass-analysis, compact IR-native structural features, raw LLVM IR for selected functions, and runtime evidence including hardware-counter measurements (Li et al., 18 Jun 2026). The Evaluation Agent measures candidate pipelines and accepts $10 \times 180 = 1800$9 only if $k=5$0, where $k=5$1 is the mean runtime of pipeline $k=5$2 over three runs (Li et al., 18 Jun 2026). This feedback-driven search achieves geometric-mean speedups of $k=5$3 over LLVM -O3 on x86-64 and $k=5$4 on ARM64, while ablations show that removing the Evaluation Agent causes later-round instability and removing the Analysis Agent weakens early decisions (Li et al., 18 Jun 2026). HintPilot applies a similar philosophy at source level by synthesizing compiler hints rather than rewriting code directly, combining retrieval over a curated set of 46 semantics-preserving hints with profiling-guided iterative refinement. On HumanEval_CPP and PolyBench, it reports up to $k=5$5 geometric mean speedup over -Ofast while preserving program correctness through compiler validation and test execution (Jiang et al., 16 Apr 2026).

The quality of the feedback interface itself can dominate model capability. “AI Coding Agents Need Better Compiler Remarks” shows that on 151 TSVC loops, Qwen2.5-Coder 7B improves vectorization success from 0.80% to 2.68% with Clang remarks at $k=5$6, a 3.3× increase, and from 2.38% to 6.95% with Intel remarks, a 2.9× increase (Deo et al., 15 Apr 2026). The paper’s more pointed finding is that precise dependence remarks such as ReadAfterWrite or WriteAfterRead, enriched with source locations and suggested fixes, produce far larger gains than ambiguous legacy remarks, while some vague remarks are actively detrimental because they trigger semantic-breaking hallucinations (Deo et al., 15 Apr 2026). WhiteFox extends the idea from remark consumption to white-box fuzzing: it instruments compiler optimizations, treats pass activation as the primary trigger feedback signal, and then uses successful optimization-triggering tests as few-shot feedback examples selected by Thompson Sampling. Across PyTorch, TensorFlow-XLA, TensorFlow Lite, and LLVM, WhiteFox can exercise up to 8× more optimizations than baselines and reports 101 bugs, with 92 confirmed as previously unknown and 70 fixed (Yang et al., 2023).

A related systems thread appears in neural compilation. LEGO-Compiler decomposes high-level-to-assembly translation into verifiable steps and control blocks, then uses assembler diagnostics, runtime errors, and unit test failures as self-correction signals for up to five rounds (Zhang et al., 26 May 2025). Its formal composability condition, $k=5$7, justifies local block translation plus global rebuild, while the empirical results show 99.375% on ExeBench for the full 17,121-case evaluation and 97.9% on AnsiBench (Zhang et al., 26 May 2025). Here compiler feedback is not only corrective; it is the mechanism that makes a staged neural compilation workflow verifiable at all.

6. Human-facing compiler feedback

Compiler feedback is not only a machine-facing optimization signal. In Tutor Kai, it is a human-facing debugging resource whose actual use can be measured with eye-tracking, think-aloud data, and interviews (Jacobs et al., 25 Jun 2025). The interface has four areas of interest: task description, code editor, GenAI feedback, and compiler feedback. Compiler output appears in the bottom-right area after code execution, alongside unit-test results, and GenAI feedback becomes available only after execution via a separate button. This structural design was intended to make compiler feedback the first post-execution information source, with GenAI as a secondary, on-demand layer (Jacobs et al., 25 Jun 2025).

The observed behavior diverged sharply from that intended workflow. Across all students, fixation time was 53.91% on the code editor, 23.79% on GenAI feedback, 7.00% on compiler feedback, and 15.29% on the task description (Jacobs et al., 25 Jun 2025). Experience mattered: experienced students spent 9.35% of fixation time on compiler feedback versus 5.05% for inexperienced students, while inexperienced students spent 30.71% on GenAI feedback versus 15.49% for experienced students (Jacobs et al., 25 Jun 2025). The “immediately before requesting GenAI” analysis sharpened the pattern: experienced students were looking at compiler feedback in 44 of 64 requests (68.75%), whereas inexperienced students looked at compiler feedback in only 39 of 107 requests (36.45%) and more often requested GenAI directly from the code editor (Jacobs et al., 25 Jun 2025).

Usefulness coding showed the same asymmetry. Out of 287 compiler feedback messages, only 17.4% were coded as having helped, compared with 49.7% of 171 GenAI feedback outputs (Jacobs et al., 25 Jun 2025). Compiler feedback was 25.9% helpful for experienced students but only 7% for inexperienced students, and 39.5% of the 129 compiler feedback messages received by inexperienced students were not read at all (Jacobs et al., 25 Jun 2025). The paper’s interpretation is not that compiler feedback was irrelevant, but that its usability, interpretability, and place in help-seeking routines made it easy to bypass once a more readable natural-language alternative was available (Jacobs et al., 25 Jun 2025). This has a broader implication for compiler-feedback research: interface design and prior knowledge can determine whether a feedback channel is even entered into the reasoning loop.

7. Limits, controversies, and future directions

Across the literature, compiler feedback is strongest when failure is local, diagnosable, and structurally aligned with the available signal. ABAP results show that iterative feedback “primarily overcomes syntactic hurdles,” with stronger models climbing from about 20% one-shot success to about 75% after five rounds but still plateauing below full correctness (Wallraven et al., 21 Jan 2026). The LeetCode study shows the same hierarchy: Compile Error and Runtime Error are far more tractable than Wrong Answer or Time Limit Exceeded, and reasoning models benefit more than non-reasoning models from the same feedback loop (Zhang et al., 16 Jun 2026). Repository-aware generation likewise finds that compiler feedback is highly effective for UNDEF and API errors but cannot repair FUNC errors inside the static-feedback loop, and some programs still compile while remaining semantically wrong (Bi et al., 2024). In theorem proving, not all frequent errors are equally repairable; failed to synthesize and some local tactic failures are more tractable than unsolved goals or linarith failed to find a contradiction (Li et al., 13 Mar 2026).

A second recurring limit is that feedback quality matters as much as feedback presence. The optimization-remarks study argues that the bottleneck is “the interface, not the agent,” because replacing ambiguous remarks with precise dependence-level analysis unlocks much larger gains for the same 7B model (Deo et al., 15 Apr 2026). Tutor Kai similarly shows that a feedback channel can remain present yet functionally displaced if students do not find it interpretable or worth consulting (Jacobs et al., 25 Jun 2025). This suggests that future compiler systems should expose more “structured, actionable feedback” rather than relying on legacy prose diagnostics alone (Deo et al., 15 Apr 2026).

A third limit is that feedback does not uniformly dominate other search strategies. The LLVM code-size study reports a real single-round gain from compiler-generated feedback, but also finds that simple sampling is stronger when the budget allows 10 or more samples (Grubisic et al., 2024). This suggests that compiler feedback is best viewed as a high-value, domain-dependent control signal rather than a universal replacement for diversity, search, or retrieval. A plausible implication is that the most effective systems will continue to combine compiler feedback with other mechanisms: retrieval in ProCoder, explanation-conditioned repair in APRIL, profiling-guided hint synthesis in HintPilot, or staged decomposition in LEGO-Compiler (Bi et al., 2024, Wang et al., 3 Feb 2026, Jiang et al., 16 Apr 2026, Zhang et al., 26 May 2025).

The forward-looking agenda is already visible in the papers themselves. Educational tools are urged to adapt feedback to prior knowledge and individual informational needs, and to teach students how to engage productively with both compiler and GenAI feedback (Jacobs et al., 25 Jun 2025). Proof-repair work calls for richer compiler-aware representations and more realistic failure trajectories (Wang et al., 3 Feb 2026). Optimization-interface work calls for compiler outputs designed explicitly for autonomous performance engineering, with machine-readable, dependence-level, actionable analysis (Deo et al., 15 Apr 2026). Taken together, these studies define compiler feedback not as a narrow implementation detail, but as a central interface problem at the boundary between formal tooling, human reasoning, and learned systems.