Document Reward Model

• Document Reward Model is a neural network that assesses document professionalism by decoupling structure and style from text content.
• It leverages rendered document images and a pairwise Bradley-Terry loss to measure layout coherence, formatting, and visual design.
• The model’s multi-modal design and DocPair dataset enable robust evaluation to guide generation workflows and improve document presentation.

A document reward model is a learned function—typically a neural network or hybrid architecture—that produces a scalar or vectorial signal evaluating the quality, relevance, or professionalism of a document according to task-specific criteria. Unlike generic or purely content-based reward functions, document reward models are explicitly engineered to address document-level attributes such as structure, format, style, or domain-compliance, and are used to guide or evaluate agentic workflows in document generation, information extraction, or summarization. They often leverage large annotation corpora, integrate multi-modal features, and provide signals that go beyond semantic correctness, filling a crucial gap for high-stakes document-centric applications.

1. Objectives and Novelty of DocReward

DocReward is an evaluation model focused on the structural and stylistic professionalism of documents, decoupling these aspects from semantic or textual quality. Most existing LLMs and reward models such as GPT-4o and GPT-5 prioritize fluency, informativeness, or semantic adequacy, largely neglecting “visual” factors: layout coherence, formatting, whitespace usage, grid alignment of tables, typographic consistency, heading hierarchies, and proper use of bullet/numbered lists.

The innovation of DocReward lies in its design and training specifically for textual-quality-agnostic document assessment. Using rendered document images as input, DocReward can evaluate visual structure and formatting independent of semantic content. This enables guidance of document generation agents toward outputs that not only “read well” but are also well-structured and visually professional—characteristics critical for user engagement and credibility in technical, business, or educational contexts.

2. Dataset Construction and Design

Central to DocReward is the DocPair dataset, comprising 117,000 paired documents spanning 32 domains and 267 document types. Each pair consists of two documents with identical textual content but differing in structure and style—either a high-professionalism, human-authored document versus a synthetic variant, or two synths compared relative to a human reference. This ensures that training targets only structural and stylistic quality rather than semantic or factual differences.

All training, validation, and test pairs are chosen to enforce the constraint

$D_{text,i} = D_{text,j} \quad \forall i, j$

within each comparison group, ensuring style and structure are isolated evaluation factors.

For evaluation, a curated test set of 473 document pairs is bundled for human annotation by well-educated professionals, providing a ranking π* of documents within each bundle based solely on structure and style.

3. Model Architecture and Training Objectives

DocReward is a multi-modal model that accepts rendered images of complete document pages, thus directly attending to spatial arrangement, visual cues, tables, indentation, and other style features. The neural architecture processes these images and outputs a scalar reward for the entire document.

Its training regime employs a pairwise Bradley-Terry loss:

$$L(\theta) = -\log \sigma(R_\theta(D_{img}^w) - R_\theta(D_{img}^l))$$

where $R_\theta$ denotes the model’s score for a rendered document, $D_{img}^w$ and $D_{img}^l$ are images of the preferred and less preferred documents in a training pair, and $\sigma(x) = 1/(1 + \exp(-x))$. This objective penalizes any instance where the less professional document (as judged by human annotators) is assigned a higher score.

At evaluation time, ranking consistency is measured by comparing the model’s argsort of bundle scores to π*, using a domain similarity metric:

$$\max_{\theta} Sim(\pi^*, \operatorname{Argsort}(R_\theta(D_{img,1}), \dots, R_\theta(D_{img,N})))$$

subject to textual identity.

4. Empirical Results and Comparative Evaluation

DocReward demonstrates a substantial advantage in both intrinsic and extrinsic settings:

• On human-annotated test bundles, DocReward achieves an absolute improvement of 30.6 percentage points in accuracy over GPT-4o and 19.4 over GPT-5. These gains are consistent in both pairwise and pointwise scoring settings, confirming that DocReward’s signal aligns more closely with human perceptions of professionalism in structure and style.
• In extrinsic agentic document generation, DocReward selects final outputs with a 60.8% human win rate versus 37.7% for GPT-5’s internal selection criterion. This margin demonstrates that DocReward’s structuring signal causes generation systems to output documents that professionals deem more visually and organizationally competent.

5. Design Implications and Practical Significance

DocReward’s architecture and data regime enable several critical capabilities:

• The explicit exclusion of textual content variation ensures that training generalizes across topics, domains, and semantic variations, making the reward model robust as a generic document structure/style evaluator.
• By operating on rendered document images, DocReward can leverage layout-level patterns (grids, section boundaries, presenters, infographics) unattainable via token-based approaches.
• DocReward’s signal is modular and can be integrated into reinforcement learning pipelines or agentic workflows, serving as a reward shaping mechanism for iterative document generation, reranking, and post-processing.

This paradigm is particularly valuable in industrial, academic, and public sector domains where document presentation quality is closely tied to trust, user comprehension, and compliance.

6. Limitations and Prospective Directions

While DocReward achieves strong performance within the structural and stylistic domain, several future directions are indicated:

• Expansion of the DocPair dataset to include more document types, style archetypes, and international formatting standards could enhance generalization.
• Incorporating finer-grained feedback (e.g., penalizing color clashes, inconsistent iconography, non-standard templates) could extend the model’s utility to broader design attributes beyond structure and style.
• Hybrid frameworks that combine DocReward’s structural/style assessment with content quality models (for example, for multi-objective guided generation) may enable more holistic document evaluation pipelines.

Further research may also explore real-time integration with interactive agentic workflows, where DocReward’s detailed signal can be used as an optimization criterion in document editing, formatting suggestions, or automated correction tools.

7. Technical Summary and Formulation Reference

Key mathematical components from the DocReward framework:

Formula	Description
$$L(\theta) = -\log \sigma(R_\theta(D_{img}^w) - R_\theta(D_{img}^l))$$	Bradley-Terry pairwise loss for reward model training
$$\max_{\theta} Sim(\pi^*, \operatorname{Argsort}(R_\theta(D_{img,1}), ..., R_\theta(D_{img,N})))$$	Optimization problem ensuring model ranking matches human judgment $\pi^*$, with argsort over bundle scores
Constraint: $D_{text,i} = D_{text,j}$	Ensures semantic equivalence of compared documents

This technical framework ensures that DocReward produces reward signals targeting professional document structure and visual style, substantially outperforming language-only baselines and LLMs in both isolated and workflow-integrated settings.