A Rubric-Supervised Critic from Sparse Real-World Outcomes

Abstract: Academic benchmarks for coding agents tend to reward autonomous task completion, measured by verifiable rewards such as unit-test success. In contrast, real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse. How can we bridge this gap? In this paper, we propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling. Specifically, we introduce Critic Rubrics, a rubric-based supervision framework with 24 behavioral features that can be derived from human-agent interaction traces alone. Using a semi-supervised objective, we can then jointly predict these rubrics and sparse human feedback (when present). In experiments, we demonstrate that, despite being trained primarily from trace-observable rubrics and sparse real-world outcome proxies, these critics improve best-of-N reranking on SWE-bench (Best@8 +15.9 over Random@8 over the rerankable subset of trajectories), enable early stopping (+17.7 with 83% fewer attempts), and support training-time data curation via critic-selected trajectories.

• The paper introduces a rubric-supervised critic that leverages behavioral rubrics and outcome proxies for segment-level success prediction in real-world coding tasks.
• It employs a multi-task, semi-supervised training method that improves AUC performance with finer-grained code survival proxies, achieving notable gains.
• Empirical results demonstrate practical benefits in inference-time scaling and SFT data selection, including a +15.9% best-of-8 improvement and an 83% reduction in sampling attempts.

Rubric-Supervised Critic Models from Sparse Real-World Outcomes

Motivation and Problem Formulation

This paper addresses the challenge of evaluating and improving LLM-based coding agents in real-world, human-in-the-loop environments—a context fundamentally distinct from autonomous, benchmark-based evaluations. Benchmarks such as SWE-bench rely on concrete metrics like unit-test pass rates, which fail to capture nuanced human interaction dynamics and outcomes prevalent in production settings. Real-world signals are sparse, delayed, and noisy, exemplified by indirect proxies such as pull request merges and subjective user feedback, which complicate credit assignment and supervision.

To bridge this gap, the authors propose a rubric-supervised critic—a learned evaluator model trained from production traces, leveraging a novel supervision hierarchy grounded in human-agent interaction segments, behavioral rubrics, and outcome proxies (Figure 1).

Figure 1: Overview of the proposed method: segments annotated with Critic Rubrics and outcome proxies enable semi-supervised, multi-task critic training for segment-level success prediction and behavioral feature recognition.

Data Structuring: Segment Grounding and Outcome Attribution

The paper formalizes agent-user interactions as sequences of segments—atomic units bounded by user-initiated task requests and agent completion signals. This segmentation isolates minimal, self-contained work units for credit assignment. In benchmarks, segments correspond to single-turn, directly-verified episodes, whereas real-world deployments induce multi-turn, multi-segment trajectories with overlapping objectives and follow-ups.

Supervision in production is organized hierarchically: pull requests (PRs) serve as coarse outcome signals, commits are attributed to specific segments, and code survival quantifies the fraction of segment-authored code that persists in the final merged diff. This structure enables both coarse and fine-grained outcome proxies, as depicted in Figure 2.

Figure 2: Production hierarchy: PRs provide outcome signals, commits are mapped to segments, enabling dense segment-level feedback via rubrics and fine-grained code survival labels.

Critic Rubrics: Design and Annotation Framework

To overcome outcome sparsity, the rubric-supervised framework introduces 24 behavioral features, termed Critic Rubrics, capturing both agent failure modes and user follow-up patterns (e.g., misunderstood intent, insufficient testing, user correction). These features are observable within segment traces and are scalable to annotate using LLM-based prompting, thus transforming previously unlabeled data into dense signals for critic training.

Rubric design groups features into agent behavioral issues, user follow-up patterns, and infrastructure faults, providing broad coverage of common software agent failure modes and user dissatisfaction signals. Annotation leverages batch LLM calls, ensuring context-adaptive labeling while preventing outcome leakage.

Rubric regression analysis demonstrates that these features correlate strongly with verified outcomes on benchmarks, and—though attenuated—maintain predictive validity in real-world interaction traces (Figure 3).

Figure 3: Rubric feature effects: benchmarks exhibit robust negative associations for core agent failures, while real-world data shows proxy-dependent, interaction-sensitive effects with weaker signal.

Semi-Supervised Critic Model Training

The critic model architecture employs a multi-task head initialized from Qwen3-4B-Instruct, jointly predicting rubric features and segment success probability using semi-supervised losses. Two primary objectives are compared: Success-Only (outcome label prediction) and Success+Rubrics (joint rubric and outcome prediction), trained on both PR merge and code survival proxies.

Training on real-world segment-level signals is essential; critics trained only on benchmarks do not transfer (AUC ≈ 0.45–0.48 on real-world outcomes). Code survival offers finer-granularity and segment-attributable supervision than PR merge (AUC improvement from 0.58 to 0.69). Rubric supervision fosters cross-backbone robustness and transferability, mitigating backbone-specific overfitting observed in outcome-only critics.

Downstream Utility: Inference-Time and Training-Time Improvements

Critic scores enable compute-efficient inference-time scaling via best-of-$K$ reranking and early stopping. Rubric-supervised critics consistently improve best-of-8 selection by +15.9 percentage points over random ($K=8$), and early stopping can reduce sampling attempts by 83% while maintaining higher success rates. Cross-backbone evaluation reveals that Success-Only critics overfit backbone idiosyncrasies (e.g., Claude Sonnet vs. Opus), whereas rubric-supervised critics generalize (Best@8 improvement on both; no degradation below random).

Critic scores are actionable for training-time SFT data selection: critic-selected segments yield a solved rate improvement (+1.2 percentage points over random SFT, +3.8 when using code survival proxy). Random SFT on real-world data provides negligible improvement, confirming critic-guided curation as an effective signal for agent improvement.

Implications and Future Directions

The rubric-supervised critic architecture supports practical, segment-level evaluators for human-in-the-loop coding agents. Empirical results underscore the necessity of real-world supervision, the utility of fine-grained survival proxies, and the generalization benefits of dense rubric supervision. These critics enable robust inference-time selection policies with substantial compute savings, as well as actionable training-time supervision for SFT curation.

Practically, these findings suggest reward modeling for interactive LLM agents should prioritize segment-level behavioral annotation via rubrics, and rely on fine-grained outcome proxies for credit assignment. Theoretical implications are that process supervision and multi-objective critic training lead to better transfer and generalization properties, particularly in settings with sparse or noisy real outcomes and heterogeneous LLM backbones.

Further research could explore expanding rubric taxonomies for broader agent tasks, adaptive critic-annotator cycling, and reinforcement learning from segmented feedback. Deployment at scale warrants careful monitoring for drift, reward hacking, and cross-context generalization.

Conclusion

This work establishes a principled approach for learning rubric-supervised critics from sparse, noisy production traces. Structuring agent-user interactions as segments and annotating them via a behavioral rubric enables semi-supervised critic training with actionable signal for real-world evaluation and improvement. Empirical results confirm that rubric supervision delivers robust scoring functions for inference-time scaling, cross-backbone generalization, and training-time SFT selection, advancing the practical evaluation and optimization of interactive LLM coding agents.