Small Reward Models via Backward Inference

Abstract: Reward models (RMs) play a central role throughout the LLM (LM) pipeline, particularly in non-verifiable domains. However, the dominant LLM-as-a-Judge paradigm relies on the strong reasoning capabilities of large models, while alternative approaches require reference responses or explicit rubrics, limiting flexibility and broader accessibility. In this work, we propose FLIP (FLipped Inference for Prompt reconstruction), a reference-free and rubric-free reward modeling approach that reformulates reward modeling through backward inference: inferring the instruction that would most plausibly produce a given response. The similarity between the inferred and the original instructions is then used as the reward signal. Evaluations across four domains using 13 small LLMs show that FLIP outperforms LLM-as-a-Judge baselines by an average of 79.6%. Moreover, FLIP substantially improves downstream performance in extrinsic evaluations under test-time scaling via parallel sampling and GRPO training. We further find that FLIP is particularly effective for longer outputs and robust to common forms of reward hacking. By explicitly exploiting the validation-generation gap, FLIP enables reliable reward modeling in downscaled regimes where judgment methods fail. Code available at https://github.com/yikee/FLIP.

• The paper introduces FLIP, a backward inference method that reconstructs instructions from responses to compute rewards instead of using direct judgment.
• It leverages the F1 similarity between inferred and original instructions to achieve up to a 79.6% improvement over judgment-based models for small LMs.
• FLIP enhances robustness against adversarial prompts and scaling challenges, offering a cost-efficient alternative for RLHF pipelines.

Backward Inference Enables Small Models to Serve as Effective Reward Models

Introduction and Motivation

Reward models (RMs) are integral components in LLM (LM) development, impacting reinforcement learning from human feedback (RLHF), preference optimization, automatic evaluation, and response reranking. Traditional RM paradigms predominantly adopt the "LLM-as-a-Judge" approach, in which large-scale LLMs are prompted directly for scalar judgment, preference, or critique. However, this direct judgment framework incurs significant computational cost, scales poorly when model parameters are reduced, and is sensitive to prompt variations and adversarial manipulation. Critically, performance deteriorates as model sizes decrease—small LLMs (SLMs, ≤8B parameters) yield unreliable judgments, underperforming by 41% on the RewardBench2 leaderboard relative to large models.

The paper "Small Reward Models via Backward Inference" (2602.13551) proposes FLIP (FLipped Inference for Prompt reconstruction), a generative, reference-free, and rubric-free RM paradigm based on backward inference: given an output, infer the most plausible underlying instruction, and define reward as the similarity between this inferred instruction and the original. FLIP exploits the observation that SLMs often retain strong generative capacity (i.e., prompt completion) even as their discriminative (judgment) abilities diminish—a direct consequence of the generation–validation gap.

Methodology: FLIP and Backward Inference

The FLIP approach diverts from conventional judgment-centric evaluation by reframing the RM problem as a generative inference task. The main steps are:

1. For each instruction–response ($x,y$) pair, a LLM infers the most plausible instruction $\hat{x}$ that could have produced $y$.
2. The reward $r(x,y)$ is computed as a similarity score $s(x,\hat{x})$ between the inferred and reference instructions, with the F1 score over words being the default metric due to its empirical effectiveness and computational simplicity.

The method composition is outlined below:

• Sampling the Inferred Instruction: $\hat{x} \sim p_\phi(x'|y)$, where $p_\phi$ is the reward model's conditional generation probability.
• Reward Computation: $r(x, y) = s(x, \hat{x})$, operationalized as the F1 score.

Figure 1 presents the graphical model contrast: while LLM-as-a-Judge samples reward directly given $(x, y)$, FLIP samples an inferred instruction $x'$ given $y$ and bases the reward only on $x$ and $x'$.

Figure 1: Comparison of conditional structures in LLM-as-a-Judge (direct reward inference) and FLIP (reward via backward-inferred instruction).

FLIP is reference- and rubric-free, requiring only $(x,y)$ pairs, thus increasing accessibility and reducing annotation costs. The analysis formalizes the approach under a Bayesian latent-variable formulation, marginalizing over possible instructions $\hat{x}$ that could generate $y$.

Workflow details are depicted in Figure 2. After inference, similarity is measured between inferred and reference instructions; high similarity signals better instruction-following.

Figure 2: FLIP infers the most plausible instruction for a given response and uses similarity as the reward.

Experimental Results

Intrinsic Evaluation

FLIP was benchmarked against pointwise, listwise, and pairwise LLM-as-a-Judge baselines on RewardBench2, across four domains and 13 SLMs. FLIP outperformed judgment-based baselines by an average of 79.6%, often boosting small models from below-random to above-random performance on four-way response selection tasks.

Notably:

• The improvement is most pronounced in the smallest models: 75% gain for ~1B models, 27% for ~4B/7B models.
• On the "Focus" subset (detecting on-topic/quality responses), SLMs with FLIP match large commercial LLMs with direct judgment.
• Gains are particularly high for long, information-rich responses, as shown in Figure 3, demonstrating the advantage of generative inference when more output context is available.

  Figure 3: FLIP yields larger performance gains as response length increases, leveraging greater contextual information for backward inference.

Parallel Test-Time Scaling

FLIP was evaluated in parallel test-time scaling (Best-of-N sampling), a standard inference strategy for leveraging response diversity. Across OLMo 2 and Llama 3 families and four benchmarks (including AlpacaEval, MATH, IFEval, and Human Interest), FLIP consistently outperforms LLM-as-a-Judge and other baselines in both performance and stability, as shown in Figures 4, 7, and 8.

Figure 4: FLIP delivers robust test-time scaling improvements across model families and benchmarks, consistently surpassing direct judgment baselines.

Reinforcement Learning

Under Group Relative Policy Optimization (GRPO) RL, adopting FLIP-based small reward models improves final policy performance by up to 2.5 absolute points over baselines, and even outperforms the final policy following RLVR on certain tasks. FLIP is especially effective in instruction-following and math evaluations, aligning with its conceptual advantage for tasks demanding precise output adherence.

Robustness Analysis

• Adversarial Prompting: FLIP is substantially more robust than LLM-as-a-Judge under adversarial prompt-injection attacks. Where direct judgment models are manipulated to assign artificially high scores, FLIP’s backward inference is not easily susceptible, as shown in Figure 5.

  Figure 5: FLIP resists adversarial prompt injection more effectively than judgment-based models, maintaining accuracy.

• Prompt Variation Robustness: FLIP exhibits the lowest performance variance across prompt formulations, further emphasizing the stability of backward-inference-based reward determination.
• Validation–Generation Gap: FLIP explicitly exploits the gap that emerges as validation (judgment) skill in SLMs deteriorates faster than generation skill with parameter downscaling. This phenomenon is substantiated both qualitatively and in prior work.
• Qualitative Capabilities: FLIP accurately penalizes off-topic, factuality-violating, and under/over-addressed outputs, since misalignment between response and instruction leads to low instruction-similarity scores (Figure 6).

  Figure 6: FLIP identifies diverse error patterns (off-topic, instruction-misaligned, false) that mislead LLM judges but are captured via backward inference.

Theoretical and Practical Implications

FLIP challenges the prevailing assumption that direct judgment is the gold standard for automatic reward modeling. Its success shifts focus toward generative approaches to reward estimation, particularly in resource-constrained environments or where model scaling is costly. Practically, FLIP reduces the reliance on large LMs for reward computation, potentially enabling more democratized and cost-efficient RLHF pipelines.

Theoretically, the work substantiates the decoupling between generative and discriminative capabilities as models scale down—echoing broader themes in discriminative vs. generative modeling [Ng & Jordan, 2001]—and extends the concept of generation–validation gap to the reward modeling domain.

Limitations and Directions for Future Research

Potential limitations include sensitivity to instruction paraphrasing and rare confounds when responses explicitly repeat instructions. The reliance on F1 as a similarity metric, though robust in practice, may break down with cross-linguality and extreme paraphrase. Future avenues include:

• Extending FLIP’s backward inference to vision, audio, and robotics, exploring generalization beyond text.
• Integrating critic-based refinement, where disagreement between original and inferred instructions provides granular feedback.
• Robustification against yet-unknown FLIP-specific adversaries.
• Exploring alternative similarity metrics and large-context inference.

Conclusion

FLIP demonstrates that backward inference—reconstructing instructions from responses and measuring similarity—is a practical, stable, and highly performant framework for reward modeling with small-scale LLMs, surpassing LLM-as-a-Judge baselines in accuracy, robustness, and scalability. This generative perspective undermines the centrality of direct scalar judgment and suggests future reward modeling should preferentially exploit generation–validation asymmetries inherent in SLMs.