PRBench: Auto Scholarly Promotion Benchmark

• PRBench is a multimodal benchmark pairing academic papers with tailored social media posts, enabling objective assessment of promotional strategies.
• It employs controlled annotations and metrics—fidelity, engagement, and alignment—to evaluate the accuracy and appeal of automated promotional outputs.
• The PRAgent framework uses multi-agent coordination for content extraction, synthesis, and platform adaptation, significantly boosting research visibility.

PRBench refers to a standardized benchmark introduced for the evaluation of automated academic promotion systems, particularly in the context of transforming scholarly research papers into platform-tailored promotional content. Originating alongside the AutoPR task, PRBench provides a multimodal dataset and a rigorous evaluation suite to measure the ability of intelligent agents to generate accurate, engaging, and alignment-optimized posts for social dissemination of research outputs (Chen et al., 10 Oct 2025).

1. Definition and Scope

PRBench is a multimodal benchmark that links 512 peer-reviewed research articles to high-quality, human-crafted promotional posts. The benchmark serves as a reference task for assessing the performance of academic promotion agents by offering controlled, richly annotated data that includes both the textual and visual content from papers and their corresponding promotional posts prepared for social media. The core objective is to objectively evaluate how systems automate the process of transforming complex academic content into concise, impactful, and platform-adapted communications.

Formally, the generative objective in AutoPR is framed as:

$$\hat{P} = \operatorname{argmax}_{P} \Pr(P | D, T_p, T_A)$$

where $D$ is the research document, $T_p$ is the target platform, and $T_A$ is the audience. The task is governed by a multi-objective function:

$$\max_{\hat{P}} \left\{\alpha_1 S_{\text{Fidelity}}(P|D) + \alpha_2 S_{\text{Align}}(P|T_p) + \alpha_3 S_{\text{Engage}}(P|T_A)\right\}$$

with tunable weights $\alpha_i$ for balancing fidelity, alignment, and engagement.

2. Dataset Composition and Structure

The PRBench dataset comprises pairs of research papers and their corresponding multimodal social media posts. Each datum includes:

• Raw PDF or source file of the peer-reviewed paper.
• Extracted textual content (abstract, sections, etc.) and figures.
• Social-media-ready promotional post, including formatted text and images.

This structure facilitates both textual and visual learning as well as cross-modal evaluation of generated outputs for multiple platforms (e.g., Twitter, RedNote). The controlled pairing allows for robust annotation of key information points and for systematic evaluation across a variety of academic fields.

3. Evaluation Axes and Metrics

PRBench operationalizes three primary axes for the assessment of automatic promotion outputs:

Axis	Sub-Metrics	Evaluation Criteria
Fidelity	Author & Title Accuracy, Factual Checklist Score	Faithful representation of content
Engagement	Hook Strength, Logical and Visual Attractiveness, CTA Score	Audience appeal and attention
Alignment	Contextual Relevance, Visual–Text Integration, Hashtag Strategy	Platform-specific optimizations

Fidelity is quantified using metrics like the Factual Checklist Score,

$$S_{\text{Checklist}}(P|D) = \frac{\sum_{i=1}^{n} w_i \cdot v(P|c_i, D)}{\sum_{i=1}^{n} w_i}$$

where $v(P|c_i, D)$ is a LLM judge output assessing whether $c_i$ (a fact with weight $w_i$) is correctly represented in the post.

Engagement and Alignment are measured through both intrinsic metrics and pairwise preference judgments, evaluating narrative structure, integration of visuals, use of audience-appropriate language, and conformity to platform norms.

4. PRAgent Framework and System Design

PRBench was data-driven in the development and evaluation of the PRAgent framework, a three-stage multi-agent system for AutoPR:

1. Content Extraction and Preparation: Raw PDF is parsed and hierarchically summarized; visual content is extracted via layout analysis (e.g., DocLayout-YOLO).
2. Collaborative Content Synthesis: Specialized agents (Logical Draft, Visual Analysis, Textual Enriching, and Visual–Text Interleaved Combination) iteratively draft and enrich the promotion, ensuring coverage of research questions, contributions, and visual interpretation.
3. Platform-Specific Adaptation: An Orchestration Agent adapts the draft into a post optimized for the target platform (adjusting for thread structure, tone, hashtags/mentions, etc.).

Formally, the summarization step is represented as:

$$D_T^{\text{sum}} = \text{Summarize}(\text{Parse}(D_T^{\text{raw}}))$$

Real-world experimental gains are notable: PRAgent, when benchmarked on PRBench, achieved a 604% increase in total watch time, a 438% escalation in likes, and at least a 2.9x boost in engagement compared to direct LLM prompting.

5. Impact on Scholarly Communication

PRBench and the associated PRAgent framework offer a paradigm for scalable, objective assessment and automation of academic dissemination. The ability to robustly evaluate fidelity, engagement, and alignment addresses key bottlenecks in the manual promotion process, leading to:

• Reduced human workload in research dissemination.
• Improved visibility and reach for scholarly outputs, with real-world increases in metrics reflecting audience interaction.
• An objective protocol to evaluate and iterate on academic PR systems, catalyzing continuous progress in automated communication agents.

A noted challenge remains the “fidelity bottleneck”: even advanced LLMs occasionally omit subtle scientific nuances. Ensuring that factual and conceptual accuracy is on par with that of human experts is an explicit, ongoing research goal.

6. Future Developments and Research Directions

Areas identified for further advancement include:

• Enhanced Contextual Fidelity: Improving the capture and transmission of nuanced scientific results and caveats.
• Dynamic Engagement Strategies: Developing richer narrative forms and more adaptive engagement hooks, moving beyond formulaic structures.
• Improved Agent Coordination: Introducing iterative, feedback-driven collaboration among synthesis agents.
• Advanced Platform Modeling: Adapting to evolving platform conventions and experimenting with new modalities, particularly as social media features change.
• Generalization and Multilingual Capabilities: Assessing and ensuring performance across domains, formats, and possibly languages, with an emphasis on maintaining high scores across the PRBench axes.

These directions aim to ensure that automated promotion systems remain robust, reliable, and impactful as scholarly communication continues to evolve.

7. Significance and Outlook

By establishing a rigorously defined, multimodal, and multidimensional benchmark, PRBench supplies the scholarly community with a tractable and measurable research problem for intelligent academic promotion. The evidence of substantial engagement improvements, coupled with objective, transparent evaluation axes, positions PRBench as both a catalyst and an arbiter for future developments in this field. As platforms for scholarly discourse diversify and expand, the continued refinement and adoption of PRBench and its associated frameworks are likely to play a central role in shaping the next generation of automated scientific communication (Chen et al., 10 Oct 2025).