Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents

Abstract: Interactive LLM agents are becoming part of daily work, but they do not reliably become easier to work with over time: a correction remembered in one session may still be violated in the next. We study this gap between preference access and preference compliance. In tasks derived from anonymized real-user friction cases, Mem0 memory still leaves 57.5% of applicable preference checks violated. We introduce Test-time Rule Acquisition and Compiled Enforcement (TRACE), a drop-in skill-layer pipeline for coding-agent runtimes that mines user corrections, rewrites them as atomic rules, and compiles them into runtime checks that must pass before an agent completes future tasks. Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections. We evaluate TRACE with simulated user-in-the-loop experiments on ClawArena coding-agent tasks and MemoryArena-derived memory-intensive tasks. On ClawArena, TRACE reduces held-out preference violation from 100.0% to 37.6% on in-distribution tasks and from 100.0% to 2.0% on out-of-distribution tasks. On MemoryArena-derived tasks, TRACE reduces in-distribution violation from 100.0% to 60.5% while matching or exceeding the strongest memory baseline on task pass. These results suggest that compiling corrections into runtime enforcement can address a repeated-friction failure mode that memory alone does not reliably solve, reducing the need for users to restate the same correction across future sessions. Experiment code is available at https://github.com/YujunZhou/TRACE_exp, and the deployable skill is available at https://github.com/YujunZhou/tellonce.

• The paper demonstrates that compiling natural-language user corrections into runtime rules significantly improves coding agent compliance, achieving up to 70.1% adherence.
• It details a novel TRACE pipeline that includes detection, extraction, and compilation of corrections, reducing violation rates across in-distribution and out-of-distribution tasks.
• Experimental evaluations reveal that TRACE minimizes repeated user interventions and maintains efficiency, paving the way for dynamic and robust agent personalization.

Compiling User Corrections into Runtime Enforcement for Coding Agents

Introduction

The paper "Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents" (2606.13174) introduces a novel approach for bridging a substantial gap in coding-agent personalization pipelines: converting user corrections from retrievable, natural-language memory artifacts into strong, runtime-enforceable constraints. Conventional memory-based personalization paradigms in LLM coding agents enable retrieval of user-specific preferences and corrections, but empirical failures highlight that the presence of a preference in context does not guarantee agent compliance. This work formalizes the access-compliance gap and proposes Trace, a pipeline that operationalizes user feedback as executable rules, providing binding runtime enforcement at inference.

Figure 1: A repeated-correction failure in daily-use interactive agents; visibility of a correction does not guarantee compliance. Trace addresses this by converting corrections into runtime-enforced checks.

Quantifying the Access–Compliance Gap

Memory systems for LLM agents (e.g., Mem0, Hindsight, ReMe-Light) focus on access: retrieving historically stored user feedback and inserting it as context in the prompt or via an external memory backend. The hypothesis behind these systems is that access leads to improved adherence, under the assumption that prompting is sufficient for behavioral alignment. To empirically test this, the authors curated a diagnostic benchmark from 32 long-context real-world coding-agent transcripts, extracting 142 correction-conflict records and deriving 29 preference checks over 19 held-out evaluation tasks. Five experimental conditions were compared: context-free (No Rules), prompt-injected (All Rules), memory retrieval (Mem0), narrow prompt-based (Relevant Rules), and compiled constraint-enforced (Compiled Rules).

Results are summarized as follows:

• No Rules: 31.6% preference compliance.
• All Rules (prompt-context): 55.0%.
• Mem0: 42.5%.
• Relevant Rules (filtered prompt): 54.0%.
• Compiled Rules (runtime executable): 70.1%.

  Figure 2: Empirical setup and compliance rates across different representation and enforcement strategies, highlighting the substantial benefit of compiled enforcement.

These results demonstrate that access, while beneficial, is insufficient for binding agent behavior: agents frequently retrieve and even acknowledge stored corrections but ultimately ignore them during task execution, optimizing for immediate coding output and requiring user re-intervention.

TRACE: Test-time Rule Acquisition and Compiled Enforcement

TRACE is designed as a drop-in skill-layer for coding agent runtimes. Its central operation is to mine user correction signals from conversation, rewrite corrections as atomic, per-user rules with explicit applicability boundaries, and compile these into runtime artifacts (skills, hooks, verifiers) that must pass before agent completion. The pipeline consists of:

1. Detection: Every user message is scanned for correction signals via a lightweight LLM (Gemma 4 31B). Only durable corrections (not transient feedback) proceed.
2. Rule Extraction: Upon detection, corrections are atomized into concise actionable rules, paired with applicability conditions (when to trigger, when to abstain).
3. Lifecycle Management: Each candidate rule undergoes semantic resolution against the user’s library, deciding among Noop, Update, Supersede, Split, or New, to address overlapping, refining, or contradictory corrections.
4. Compilation: The rule is compiled into one of three enforcement tiers (deterministic, semantic, or intent-level) with (a) an applicability matcher, (b) context-injected instructions, and (c) executable runtime verifiers.
5. Execution: At inference, only the relevant per-user rules are loaded, and the agent is required to pass all registered verifiers to complete the task.

   Figure 3: Trace pipeline for converting real-time user corrections into runtime-enforceable rules, with enforcement artifacts loaded as runtime gates during subsequent tasks.

This pipeline instantiates an ability for agents to transform user instructions from advisory advice to explicit preconditions for task completion.

Experimental Evaluation

In-Distribution (ID) Performance

The evaluation leverages ClawArena and MemoryArena benchmarks for coding and memory-intensive tasks, utilizing simulated user-in-the-loop protocols to provide repeatable feedback. Evaluation metrics include task pass rates, violation rates (preference non-compliance), and the average number of user corrections.

On ClawArena ID tasks:

• Violation rate: Trace reduces from 100% (baseline) to 37.6%.
• All memory baselines exceed 50%, some over 94%.
• Task pass rate: Remains in a narrow band across all methods, indicating enforcement gains are not traded for task performance degradations.

On MemoryArena-derived tasks:

• Trace yields the highest task pass (17.3%) and lowest violation (60.5%) among all competitive baselines.

  Figure 4: In-distribution evaluation on ClawArena and MemoryArena, showing that Trace substantially lowers repeated preference violations compared to memory-based pipelines.

Out-of-Distribution (OOD) Generalization

When evaluated on OOD families:

• ClawArena violation rate: Trace achieves 2.0% (nearly two orders of magnitude below next baseline), with some models reaching perfect compliance.
• Task pass rate remains stable across scenarios, demonstrating generalization without adverse side effects.
• MemoryArena OOD: Absolute violation remains high (best at 97.0%), but Trace is still the only method improving over memory-only baselines, indicative of partial cross-task generalization.

  Figure 5: Out-of-distribution results on ClawArena and MemoryArena tasks, with Trace sharply reducing violation rates in unseen task families.

Efficiency and User Interventions

Test-time efficiency metrics show that Trace does not introduce substantial wall-clock overhead:

• User turns (correction interventions): Reduced from 2.00 (no memory) to 1.37 (ID) and 1.02 (OOD) with Trace.
• Wall-clock times: Trace averages 42.5s (ID) and 41.8s (OOD), matching the fastest baselines.
• Some memory retrieval-based baselines (e.g. ReMe-Light) introduce substantial latency.

  Figure 6: Test-time efficiency, showing Trace reduces the need for repeated user corrections while maintaining low runtime overhead.

Implications and Future Directions

This work fundamentally challenges the sufficiency of memory-based personalization in agent alignment and collaboration, especially for coding environments where user corrections are precise and frequent. TRACE demonstrates that runtime enforcement—not just context access—reduces repeated friction by making corrections binding, thereby reducing the user's cognitive and operational burden in repetitive collaborative settings.

From a practical perspective, the pipeline is skill-agnostic, runtime-agnostic (compatible with Codex CLI and Claude Code), and does not require developer-authored rule repositories. All rules are dynamically induced from user interactions, supporting per-user policy evolution and refinement.

From a theoretical perspective, the study reframes agent alignment: not just as a problem of preference inference or memory retrieval, but as a pipeline of (1) detection, (2) representation, (3) enforcement—where execution constraints take precedence over narrative recall. This approach unifies personalization-by-memory and personalization-by-enforcement in a compositional, incremental, and lifecycle-aware manner.

Speculative future directions include:

• Automated rule synthesis and abstraction: Generalizing individual rules into broader policy schemas or synthesizing higher-order constraints from atomic rules.
• Preference conflict resolution: Scaling up to support multi-user or organizational settings with conflicting corrections.
• Broader modalities: Extending compiled enforcement from code/workspace domains to multi-modal or open-ended creative tasks.
• Integration with advanced RL alignment pipelines: Combining runtime enforcement with process supervision (Liu et al., 20 Dec 2025) and policy optimization (2509.23095).
• Continual and federated learning: Evolving enforcement artifacts as agent deployments grow in scale, diversity, and span.

Conclusion

The distinction between remembering and following user corrections is empirically vital in LLM coding agents. TRACE provides strong evidence that compiling natural-language feedback into runtime-enforceable rules, rather than passively storing and retrieving it, sharply improves compliance without sacrificing efficiency or task completion. The findings advocate for a paradigm shift: effective personalization for LLM agents must integrate memory and enforcement, not treat them as alternatives. Future research in agent alignment and collaboration should therefore consider enforcement as a primary substrate for preference binding, opening the path to richer, less frictional, and more robust human–AI interaction.