FARE-Code: Specialized Code Evaluator

• FARE-Code is a specialized variant of foundational automatic reasoning evaluators, fine-tuned for assessing code and test-case quality.
• It employs iterative supervised fine-tuning with rejection sampling on ~15K annotated code samples to enhance diagnostic test-case judgments.
• The evaluator outperforms baseline models by up to 65%, offering improved reliability for code generation and reinforcement learning pipelines.

FARE-Code is a variant within the Foundational Automatic Reasoning Evaluators (FARE) family specialized for evaluating code and, in particular, test-case quality in code-centric reasoning tasks. Emerging directly from the paradigm of scalable multi-task generative evaluator training (Xu et al., 20 Oct 2025), it exemplifies how continued domain-specific adaptation atop a strong generalist base can produce automatic evaluators with significant improvements for programming-related assessment.

1. Model Architecture and Initialization

FARE-Code inherits the architecture of LLMs used throughout the FARE family, deploying billions of parameters (notably in the 8B, 20B, and higher ranges). These architectures, such as SFR‑FRJ‑20B, are originally trained for multi-task evaluation across diverse reasoning domains. Crucially, FARE-Code is not simply a code-specific model from scratch; it is initialized from a general-purpose FARE checkpoint and then continually fine-tuned on a code-relevant dataset. The weights, therefore, combine reasoning-centric generality with specialized learning targeting programming language syntax, code structure, and test-case semantics.

2. Iterative Supervised Fine-Tuning via Rejection Sampling

The FARE-Code training regime employs an iterative supervised finetuning process with rejection sampling ("RS‑SFT"):

$$\theta_{t+1} = \arg\max_\theta \sum_{(x,y) \in \mathcal{D}_t} \log \pi_\theta(y \mid x)$$

where $\pi_\theta$ is the evaluator policy, and $(x,y)$ are input–output pairs in the code evaluation domain. This objective is applied domain-specifically using a dataset of approximately 15,000 annotated code samples—for instance, pairwise test-case judgments from AceCoder. By repeatedly sampling candidate outputs and only retaining those deemed "correct" or "diagnostic," the model’s learning signal is concentrated on code correctness and evaluation fidelity.

The continual fine-tuning phase further leverages domain-sensitive prompting strategies, including "direct judgment" formats and curricula ordered by empirical difficulty (measured by the number of correct outputs per sample), to sharpen outcome-focused reasoning. Unlike standard chain-of-thought inputs, intermediate outputs may be dropped to focus exclusively on test-case verdicts.

3. Domain Adaptation and Error Pattern Sensitivity

FARE-Code’s specialized fine-tuning yields several domain-relevant enhancements absent in generic evaluators:

• Programming Error Pattern Recognition: The evaluator develops sensitivity to code-specific error modes, including type mismatches, argument omissions, or syntax errors (e.g., malformed JSON in code tool calls).
• Granularity in Test-Case Quality Judgment: Rather than binary “good/bad” classification, the model discriminates the diagnostic value of test-cases—such as whether a test actually distinguishes between correct and flawed code snippets.
• Outcome-Focused Signal Extraction: By isolating training data explicitly on correctness outcomes (and not intermediate reasoning steps, unless appropriate), FARE-Code provides high-resolution assessment of test-case robustness.

A plausible implication is increased reliability when using automatic evaluators for both model selection and verification tasks in code generation pipelines.

4. Performance Benchmarks and Comparative Evaluation

On test-case quality tasks, FARE-Code outperforms the base gpt-oss-20B model by approximately 65%. Additional experiments on benchmarks such as CodingJudgeBench report improvements exceeding 11 absolute points over baseline evaluators, and further gains in comparison to models with much larger parameter counts (e.g., gpt-oss-120B). These metrics are strictly quantitative and reflect the model's enhanced ability to evaluate programming tasks; for example, FARE-Code is shown to distinguish subtle code flaws and robust test-case design more effectively than prior (even larger) generative evaluators.

5. Practical Utility in Code Generation Ecosystems

By specializing in test-case judgment, FARE-Code fulfills several critical roles:

• Inference-Time Reranking: When used as a reranker on code generation benchmarks (such as on MATH or CodingJudgeBench), it approaches oracle-level selection of correct outputs.
• Reinforcement Learning Integration: As a verifier within RL training loops, FARE-Code increases downstream model performance (as assessed by held-out test-case success) compared to string-matching verifiers.
• Reduced Artifact Bias: Fine-tuning on direct judgment datasets reduces positional bias (the tendency to prefer first answers in pairwise ranking), which is crucial for robust evaluation and fairness.

6. Data Efficiency and Deployment Considerations

FARE-Code achieves domain-specialist improvement with only a modest code-specific data (≈15K samples). This data efficiency makes it practical for deployment in real-world programming pipelines, where annotating substantially larger datasets for every programming task may be infeasible. Its strong evaluator performance—from continual code-centric adaptation—is competitive even against much larger models trained on primarily natural language data, suggesting an effective blueprint for specialized automatic evaluator development.

7. Summary and Outlook

FARE-Code exemplifies the transformation of generalist multi-task evaluators into highly specialized automatic judges for code and test-case assessment. Through iterative rejection-sampling supervised fine-tuning, direct judgment prompting, and curriculum ordering, it learns to identify programming-specific flaws, evaluate test-case robustness at high resolution, and minimize common unwanted artifacts such as positional bias.

Its strong performance metrics—up to 65% improvement against baseline—demonstrate that targeted domain adaptation atop foundational evaluator architectures can yield robust, data-efficient, and reasoning-centric assessment systems for software engineering. Applications span code inference tasks, benchmark evaluation, and reinforcement learning pipelines, establishing FARE-Code as an authoritative evaluator for programming outputs in both research and practical deployment (Xu et al., 20 Oct 2025).