From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

Abstract: LLMs have achieved remarkable progress in text generation but still struggle with generative writing tasks. In terms of evaluation, existing benchmarks evaluate writing reward models coarsely and fail to measure performance from the perspective of specific requirements. In terms of training, existing training methods either use LLM-as-a-judge approaches or train coarse-grained reward models, lacking fine-grained requirement-adherence reward modeling. To address these issues, we propose a fine-grained evaluation pipeline WEval for writing reward models and a fine-grained reinforcement learning training framework WRL. The evaluation data of WEval covers multiple task categories and requirement types, enabling systematic evaluation of writing reward models by measuring the correlation between the rankings of the reward model and gold rankings. WRL constructs positive and negative samples by selectively dropping instruction requirements, allowing for more precise reward model training. Experiments show that our models achieve substantial improvements across various writing benchmarks and exhibit strong generalization. The code and data are publicly available at \href{https://github.com/Rainier-rq1/From_Coarse_to_Fine}{https://github.com/Rainier-rq1/From\_Coarse\_to\_Fine}.

• The paper introduces a dual-framework combining WEval and WRL for fine-grained evaluation and reinforcement learning in writing-centric tasks.
• It employs atomic requirement dropout and Bradley-Terry loss to optimize reward models for precise instruction adherence.
• Empirical results show significant improvements in writing quality, instruction satisfaction, and generalization across diverse domains.

Fine-Grained Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

Overview and Motivation

LLMs have achieved notable advances in generic text generation, yet generative writing tasks—particularly those requiring creative, structured, or precise outputs—remain high-friction due to multifaceted instruction requirements. Prevailing benchmarks and training paradigms primarily focus on coarse-grained evaluation (e.g., knowledge, safety, reasoning) and reward modeling along general attributes (e.g., fluency, coherence). These approaches fail to assess granular adherence to specific writing requirements, thereby impeding the development and evaluation of robust reward models and reinforcement learning strategies tailored to writing-centric tasks.

To address these deficiencies, this work introduces a dual-framework: WEval, a fine-grained evaluation pipeline predicated on atomic requirement dropout and requirement-level ranking correlation metrics, and WRL, a reinforcement learning methodology harnessing Bradley-Terry-based reward model optimization for precise requirement-wise reward signals. This holistic approach enables systematic benchmarking and effective RL training for LLMs in writing-intensive scenarios.

Figure 1: Comparison with prior writing-task evaluation and training paradigms, highlighting the transition from coarse-grained to fine-grained requirement-level evaluation and modeling.

Framework Design: WEval and WRL

WEval constructs partial orders over writing responses by systematically dropping atomic instruction requirements, thereby forming golden rankings based on decremental adherence. Reward models are evaluated by calculating correlation metrics between their predicted rankings and the golden rankings, allowing precise quantification of requirement-level reward modeling capabilities. WRL leverages this construction to perform fine-grained reward model training: for each writing prompt and response, positive and negative samples are synthesized by requirement dropout, and the Bradley-Terry loss is employed to optimize the reward model for instruction satisfaction at the atomic level. This reward model provides dense, requirement-sensitive signals for policy model RL training, implemented via Group Relative Policy Optimization (GRPO).

Figure 2: Schematic of the WEval evaluation pipeline and WRL training architecture, emphasizing requirement dropout and fine-grained reward signal flow.

Experimental Protocol and Evaluation

The empirical evaluation encompasses proprietary, open-source, and writing-enhanced LLMs across diverse benchmarks: WritingBench, LongWriter, Arena-Write for in-domain assessment; and DeepResearch Bench-RACE, FINDER_DEFT for out-of-domain generalizability. The WEval dataset stratifies writing tasks by content, style, format, and length requirements, indexing atomic adherence and response ordering for robust correlation metrics. The GRPO RL training regime is applied to base models, utilizing WRL-trained reward models as feedback mechanisms.

Figure 3: Arena-Write win rates for WRL-trained models, demonstrating performance uplift against strong baselines.

Figure 4: DeepResearch-related benchmark results for FINDER_DEFT and Bench-RACE, evidencing cross-domain generalization of WRL-trained models.

Numerical Results and Analysis

WRL consistently enhances writing performance across base models:

• Qwen2.5-7B-Instruct achieves a +7.4 score improvement on WritingBench, outperforming specialized writing models with comparable scale.
• Distill-Qwen-14B-WRL and Qwen3-8B-WRL yield +6.0 and +1.3 gains, respectively, surpassing proprietary GPT-4o and o1-Preview.
• WRL delivers uniform improvements across academic, business, political, artistic, educational, and marketing domains, as well as across style, format, and length requirements.

On LongWriter, WRL-trained models exhibit superior scores in relevance, accuracy, coherence, clarity, breadth and depth, and reading experience, confirming fine-grained optimization efficacy.

Reward model evaluation on WEval reveals distinctly superior correlation (94.6), instruction-level (97.3), and prompt-level (78.0) metrics, closely matching human annotator performance and outperforming LLM-as-a-Judge and other reward model baselines.

Ablation studies indicate that replacing reward models with WRL-optimized variants yields the highest downstream RL performance and aligns closely with WEval results, underscoring evaluation-train match.

Case studies illustrate that requirement-level evaluation and reward modeling provide a substantially more granular and accurate assessment compared to coarse-grained scoring paradigms: responses violating explicit constraints (e.g., length, rhetorical device, formatting) are correctly penalized, which is missed by fluency/coherence-based evaluators.

Figure 5: Case study demonstrating the superiority of fine-grained requirement-level evaluation for nuanced writing tasks.

Practical and Theoretical Implications

The proposed framework fundamentally enables:

• Requirement-aware benchmarking for reward models in complex writing tasks.
• RL optimization attuned to specific, user-defined constraints, transcending simplistic attribute-based reward modeling.
• Enhanced generalizability to out-of-domain tasks (e.g., structured research reports) without recourse to costly LLM-as-a-Judge or multi-stage pipelines.
• Empirical validation of reward model utility and evaluation-train consistency for RLVR paradigms.
• Robustness of evaluation and training across different teacher models and base architectures, emphasizing model-agnostic extensibility.

This approach sets a new standard for instruction-following evaluation and training in generative writing settings, establishing a foundation for future work in hierarchical reward modeling, multi-constraint instruction following, and transfer learning for writing-centric RL.

Conclusion

By operationalizing atomic requirement dropout in both evaluation and reward model training, this paper delivers a fine-grained benchmarking and RL framework for writing-centric generation tasks. The experimental results unequivocally demonstrate substantial gains in writing quality, instruction adherence, and reward modeling reliability. The dual-framework is broadly applicable and highly predictive of downstream RL performance, offering a scalable and principled methodology for advancing LLMs in generative writing and complex instruction-following contexts (2604.27453).