You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass

Abstract: We present a discriminative multimodal reward model that scores all candidate responses in a single forward pass. Conventional discriminative reward models evaluate each response independently, requiring multiple forward passes, one for each potential response. Our approach concatenates multiple responses with separator tokens and applies cross-entropy over their scalar scores, enabling direct comparative reasoning and efficient $N$-way preference learning. The multi-response design also yields up to $N\times$ wall-clock speedup and FLOPs reduction over conventional single-response scoring. To enable $N$-way reward evaluation beyond existing pairwise benchmarks, we construct two new benchmarks: (1) MR$^2$Bench-Image contains human-annotated rankings over responses from 8 diverse models; (2) MR$^2$Bench-Video is a large-scale video-based reward benchmark derived from 94K crowdsourced pairwise human judgments over video question-answering spanning 19 models, denoised via preference graph ensemble. Both benchmarks provide 4-response evaluation variants sampled from the full rankings. Built on a 4B vision-language backbone with LoRA fine-tuning and a lightweight MLP value head, our model achieves state-of-the-art results on six multimodal reward benchmarks, including MR$^2$Bench-Image, MR$^2$Bench-Video, and four other existing benchmarks. Our model outperforms existing larger generative and discriminative reward models. We further demonstrate that our reward model, when used in reinforcement learning with GRPO, produces improved policy models that maintain performance across standard multimodal benchmarks while substantially improving open-ended generation quality, outperforming a single-response discriminative reward model (RM) baseline by a large margin in both training stability and open-ended generation quality.

• The paper introduces a single-pass multi-response reward model that scores multiple candidates concurrently by processing a concatenated prompt and candidate sequence.
• It achieves an average of 71.2% accuracy across benchmarks and reduces inference latency and FLOPs by up to 4x compared to single-response models.
• The approach enhances RLHF through Group Relative Policy Optimization, improving stability and performance in multimodal generation tasks.

Multi-Response Reward Modeling for Vision-LLMs: A Single-Pass Discriminative Approach

Motivation and Problem Setting

Preference-based alignment of large vision-LLMs (VLMs) relies critically on scalable, high-quality reward models (RMs) for both evaluation and reinforcement learning from human feedback (RLHF). Traditional discriminative RMs operate by assigning scores to each candidate response independently, requiring one forward pass per response, while generative RMs leverage the VLM's generation pipeline to produce a preference ranking or rationale, incurring substantially higher inference costs, especially as the number of candidates $N$ increases. Standard multimodal reward benchmarks focus on pairwise comparisons, leaving a gap in evaluation for multi-response ($N > 2$) settings, which are essential for best-of-$N$ selection and group-based policy optimization.

Model Design and Single-Pass Scoring

The paper introduces a discriminative multimodal reward model supporting efficient multi-response scoring in a single forward pass. At inference, it concatenates the given prompt and all $N$ candidate responses, separated by a dedicated <|resp_sep|> token, and processes the concatenated sequence in one pass through a 4B VLM backbone. A multi-response value head, located at each response's final token, produces a vector of $N$ scalar scores, which are then used for both training and inference.

This architecture is contrasted against (i) single-response discriminative RMs, which compute scores independently for each (prompt, response) pair, and (ii) generative RMs, which require autoregressive generation per comparison or pair.

Figure 1: Architectural comparison of single-response discriminative, generative, and the proposed multi-response discriminative reward models for scoring $N$ candidates.

The model is trained using a cross-entropy (CE) loss across the $N$ scores, optimizing for the ground-truth top-ranked candidate. The last-token hidden state is used for each response, providing a summary representation via the causal attention mechanism.

Multi-Response Benchmark Infrastructure

Recognizing the limitations of pairwise-centric evaluation, the authors construct two new $N$-way ranking benchmarks for robust assessment:

• MR$^2$Bench-Image: 240 real-user prompt-image pairs with 8 VLM-generated responses per sample, ranked by human annotators to produce complete orderings. 4-response subsets are used for evaluation.
• MR$^2$Bench-Video: 495 open-ended video question-answering prompts with 19 candidate responses each, annotated through ~94K pairwise human preferences, denoised using Preference Graph Ensemble and Denoising (PGED) to construct consistent $N > 2$0-way rankings.

These benchmarks expand coverage to groupwise response selection, safety, VQA, and visual reasoning in both image and video domains.

Empirical Performance and Ablation

Reward Model Quality

The 4B Molmo2-based multi-response RM achieves an average of 71.2% accuracy across six standard multimodal benchmarks, outperforming significantly larger generative and discriminative reward model baselines, including state-of-the-art open-source and proprietary VLM judges.

Ablations on value head activation, loss function, and response pooling confirm that SiLU-activated MLP heads with last-token pooling and cross-entropy loss robustly outperform alternatives.

Efficiency Evaluation

The single-pass, multi-response design yields substantial reduction in end-to-end inference cost. On $N > 2$1 settings, up to 3.9x wall-clock per-sample latency and 4.0x FLOPs reduction are observed versus the single-response scoring baseline on powerful hardware.

Figure 2: Inference efficiency gains from single-pass multi-response scoring, achieving significant latency and FLOPs reduction as $N > 2$2 increases.

Further, efficiency scales proportionally with $N > 2$3—the computational cost of the multi-response model remains nearly constant with increasing $N > 2$4, while baseline costs scale linearly. This outcome is significant in best-of-$N > 2$5 scenarios where rapid inference is a decisive constraint.

Policy Optimization and Learning Dynamics

To validate practical utility, the model is deployed as a reward function in Group Relative Policy Optimization (GRPO) for RLHF-style fine-tuning. The multi-response RM provides a much more stable and steadily increasing reward signal during policy optimization compared to the single-response baseline, ensuring robust convergence and regularization against reward hacking.

Figure 3: Validation reward trajectories in GRPO: only the multi-response RM delivers consistent, monotonic improvements, supporting stable RL optimization.

This design translates to stronger open-ended generation performance, as evidenced by substantial gains on open-ended image and video benchmarks (e.g., +5.6% WildVision win rate, LLaVA-Bench +4.6, MMHal +0.27), while fully preserving accuracy on 24 closed-form recognition and reasoning tasks.

Comparison with Open-Source Reward Models

A comprehensive comparison against open-source discriminative RMs (e.g., Skywork-VL-Reward, IXC-2.5-Reward), generative RMs (e.g., R1-Reward, MM-RLHF-Reward, LLaVA-Critic), and off-the-shelf VLMs highlights the multi-response RM’s dominance, despite using only 4B parameters (versus 7B–78B for prior works). On challenging video benchmarks, including MR$N > 2$6Bench-Video, it achieves best-of-4 accuracy of 50.7%, a nontrivial outcome owing to the complexity of human-aligned video preference tasks.

Efficiency Across Hardware and Input Modality

Efficiency profiling demonstrates that the single-pass architecture consistently yields lower FLOPs and latency relative to all baselines across both image and video modalities, regardless of the underlying VLM backbone. It is worth noting that the computational savings scale particularly well when the shared context (especially visual tokens in video) dominates overall sequence length, further extending the regime where multi-response design is advantageous.

Implications, Limitations, and Future Directions

This work enables scalable best-of-$N > 2$7 preference modeling and RLHF for VLMs, facilitating more practical deployment of multimodal instruction-tuned agents in open-ended applications. The single-pass approach lowers the barrier for training and deploying competitive RMs on commodity infrastructure—a key step in democratizing access to RLHF for vision-LLMs.

Practically, the approach is immediately relevant for large-scale selection, reranking, and as a reward function in RL pipelines. Theoretically, it opens up new avenues for research on listwise preference learning, comparative representation learning, and the study of attention-based reasoning across candidate pools.

Figure 4: Per-sample FLOPs comparison: the multi-response approach delivers 2–17x lower computational cost than open-source baselines for both image and video benchmarks.

Outstanding challenges include extending model and training protocol robustness to much larger $N > 2$8 in practical deployments, and the lack of natural language rationales, which are inherent in generative judge approaches but absent in discriminative models.

Conclusion

The paper establishes that single-pass, multi-response reward modeling is both more efficient and more accurate than traditional approaches for multimodal language agents trained with human feedback. The newly introduced MR$N > 2$9 benchmarks fill a critical gap in evaluation infrastructure, and the architecture’s efficiency and learning stability support its immediate adoption for advanced vision-language alignment. The work provides both empirical and methodological foundations for future progress in scalable, groupwise preference modeling for multimodal systems.

(2604.10966)