WritingBench: Evaluating LLM Generative Writing

• WritingBench is a benchmark and evaluation framework that tests generative writing abilities in LLMs through diverse, domain-specific tasks.
• Its multidomain structure spans academic, finance, law, art, education, and marketing, offering fine-grained insights into real-world writing performance.
• The framework employs dynamic, query-specific rubrics and a criteria-aware critic model to provide rigorous, explainable assessments that drive model improvements.

WritingBench is a comprehensive benchmark and evaluation framework specifically designed for the assessment and advancement of generative writing abilities in LLMs. It establishes an extensible, multi-domain testbed for both academic and practical research into writing quality, style, coherence, and domain adaptation in LLM-generated text. Distinguished from earlier benchmarks by its breadth, task diversity, and dynamic evaluation strategy, WritingBench has accelerated progress in the rigorous, context-aware evaluation and training of foundation models for writing applications.

1. Multidomain Benchmark Structure

WritingBench comprises six primary writing domains and one hundred secondary subdomains, representing the full spectrum of professional and creative text generation tasks encountered in real-world scenarios. This hierarchical categorization allows systematic and fine-grained evaluation of LLMs across the writing landscape.

Primary Domains:

1. Academic & Engineering: Outlines, abstracts, technical reports, patents, scientific documentation.
2. Finance & Business: Contracts, market analyses, investment reports, business communications.
3. Politics & Law: Legal opinions, policy documents, governmental analysis, court judgments.
4. Literature & Art: Creative writing, fiction, poetry, scripts, reviews, character and plot design.
5. Education: Lesson plans, textbooks, assignments, instructional feedback, curriculum design.
6. Advertising & Marketing: Slogans, brand stories, marketing copy, social media campaigns, product descriptions.

Secondary subdomains are exhaustively specified (e.g., "Market Analysis," "Book Review," "Character Design"), each annotated with explicit requirements for style, format, and length.

Example Subdomain	Domain	Requirements Examples
Paper Outline	Academic & Engineering	Structure, clarity, brevity
Legal Opinion	Politics & Law	Authority, citation norms, objectivity
Character Design	Literature & Art	Creativity, depth, consistency
Social Media Content	Advertising & Marketing	Conciseness, engagement, style

The annotation framework enables the construction of prompts such as “Write a technical summary in IEEE format, 500 words” or “Draft a policy speech in a persuasive tone,” ensuring model evaluation aligns with real professional and creative requirements.

2. Query-Dependent Evaluation Framework

A core innovation of WritingBench is its instance-specific, query-dependent evaluation approach. Instead of static rubrics, evaluation criteria are dynamically generated for each test query:

• For a given query $q$, the framework prompts an LLM (or a human expert) to create five strict assessment criteria $C_q = \{c_1, c_2, \ldots, c_5\}$, each with:
  • A concise title,
  • An extended description,
  • A detailed scoring rubric (integer scale, typically 1–10, with level descriptions).

These criteria encompass style (tone, reader fit), format (section, structure), content fulfiLLMent, length, and other requirements tied closely to the prompt. This design permits tailored assessment for specialized and creative writing tasks, capturing nuances a global rubric would miss.

For each LLM-generated response $r$, the evaluation proceeds as: $$\text{Score}(q,r) = \frac{1}{|C_q|} \sum_{i=1}^{|C_q|} s_i$$ where $s_i$ is the integer result for criterion $c_i$.

This framework provides fairness and granularity, allowing the same LLM to be rigorously tested on tasks with very different, instance-specific requirements.

3. Criteria-Aware Critic Model

To facilitate scalable and objective evaluation, WritingBench introduces a trained criteria-aware critic model:

• The critic model $\mathcal{M}_c$ is based on Qwen-2.5-7B-Instruct and trained on 50,000 LLM-generated samples paired with human ratings and dynamically generated criteria.
• For triplets $(q, r, C_i)$, the model produces a score in $[1,10]$ and a text-based justification, reflecting both the content and the grading rubric: $$\mathcal{M}_c: (q, r, C_i) \mapsto [1,10] \times \mathcal{J}$$
• Training uses human-annotated rubrics, maximizing model alignment with expert judgment.

This critic enables both automated batch scoring and explainable evaluation, reporting, for example, that a response missed a required section or failed to use the stipulated writing style. Empirically, the dynamic, query-specific evaluation using the critic achieves an 83% agreement rate with human preference, exceeding static-criteria baselines.

4. Data Curation and Model Improvement

WritingBench’s evaluation framework is directly leveraged for high-quality data curation and model fine-tuning:

• Large-scale SFT data (e.g., 24,000 responses) are filtered through the critic, with the top 50% (highest-scoring by criteria) retained for further training.
• LLMs fine-tuned on this selectively filtered dataset—such as Qwen-2.5-7B and Llama-3.1-8B—close the gap with much larger or proprietary models in WritingBench and other relevant benchmarks, indicating that criteria-aware curation delivers significant improvement.

Model	WritingBench Score	Benchmark2 Score
Qwen-2.5-7B-Instruct	7.43	4.39
Llama-3.1-8B-Instruct	6.35	3.12
Qwen-2.5-7B-filtered (post-curation)	8.49	4.70
Llama-3.1-8B-filtered	8.49	4.65
DeepSeek-R1 (strong SOTA)	8.55	4.79

The approach is domain-agnostic and extensible; new writing types and requirements can be incorporated with minimal effort due to the query-dependent rubric generation protocol.

5. Open-Source Tools and Modular Framework

WritingBench is published with a full open-source suite:

• 1,239 queries spanning all domains and requirements, with reference materials and fine-grained annotations.
• Criteria/rubric generation utilities (prompt templates and scripts),
• Scoring and justification tools,
• The trained critic model (weights, code),
• Enhanced/fine-tuned model checkpoints,
• Modular pipelines for benchmarking, data curation, and model improvement.

The framework and resources are available at https://github.com/X-PLUG/WritingBench.

These tools enable researchers to:

• Benchmark novel models or techniques on realistic, broad-spectrum writing challenges,
• Generate new domain- or project-specific benchmarks with dynamic rubrics,
• Systematically curate training data for improved downstream generative writing performance.

6. Impact and Research Significance

WritingBench represents the most comprehensive, fine-grained benchmark for advanced generative writing assessment to date. Key impacts include:

• Systematic comparison of LLMs on task coverage, style, and detailed writing requirements, rather than generic text generation.
• Enabling consistent, transparent, and customizable evaluation across all important professional and creative writing domains.
• Facilitating high-quality data filtering for training/fine-tuning, notably raising small and medium LLMs to state-of-the-art performance.
• Providing infrastructure for the rapid extension to new domains, user requirements, or emerging writing norms.

A plausible implication is that WritingBench establishes a new standard for writing evaluation in AI, supporting progress in general-purpose, creative, and requirement-driven LLM writing applications.

7. Outlook and Limitations

While WritingBench substantially raises the rigor and coverage of writing evaluation, certain domains—particularly highly creative or structurally complex writing (e.g., fiction, advanced poetic forms)—remain challenging for LLMs, even under criteria-specific evaluation. The approach's dependence on quality of criterion generation and critic model calibration suggests benefits from ongoing human-in-the-loop oversight, expanding rubrics, and further enhancing the explainability and reproducibility of scoring results.

Continual benchmark development, community curation, and integration with dynamic training paradigms are likely to extend WritingBench's role as a central research infrastructure for generative writing systems.