PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf

Abstract: Expert writing feedback from experienced researchers is critical for early-career scholars to improve their manuscripts, yet high-quality feedback often remains scarce because reviewing research papers is labor-intensive. Emerging AI-powered writing assistants largely focus on grammar fixes or simulating peer review with final scores, yet they fall short of providing concrete, actionable suggestions that help students improve their papers during drafting. We present PaperMentor, a human-centered writing assistant system that delivers actionable suggestions as Overleaf-native inline comments while leaving the actual writing entirely to human authors. PaperMentor integrates an expert skill library carefully curated from established researchers' writing advice with 12 specialized agents covering different aspects of paper writing, such as formatting compliance, phrasing accuracy, and terminology consistency. In a user study (n=14), 90.6% of the generated comments were rated actionable and 67.5% were rated valid, significantly outperforming a GPT-5.2 baseline uswithout the skill library. We release PaperMentor as open source for public use. Our code is publicly available under the AGPL-3.0 license at https://github.com/jiarui-liu/overleaf

• The paper introduces a human-centered, multi-agent system that delivers detailed, section-specific feedback for AI research papers using a curated expert skill library.
• The system employs 12 specialized review agents that analyze LaTeX projects and offer actionable suggestions, demonstrating superior validity and actionability over baseline LLM prompting.
• Empirical results show improvements of 6.5 percentage points in validity and 4.1 points in actionability, validating the efficacy of the system's structured feedback approach.

PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf

Introduction

PaperMentor addresses a well-documented gap in academic writing support for early-career AI researchers: the lack of actionable, high-quality feedback typically provided through close mentorship or expert reviewer comments. Existing AI-driven tools are largely confined to grammar checking or coarse peer-review simulation, offering little in the way of concrete textual and structural guidance at the drafting stage. By operationalizing a curated library of expert skills through specialized review agents and a tightly integrated Overleaf interface, PaperMentor systematically generates actionable, section-anchored comments. This design provides a scalable approach to delivering individualized, expert-driven feedback while preserving authorial control, positioning PaperMentor as a new, extensible infrastructure for AI scientific writing assistance (2606.08857).

System Architecture

PaperMentor consists of a three-phase pipeline: input processing, multi-agent review, and comment aggregation. This architecture leverages 12 specialized agents, each targeting distinct review domains—section-level (e.g., Methods, Results), global (e.g., writing style, LaTeX formatting), and dynamically configured (e.g., paper type, venue-specific)—all grounded in a skill library distilled from published guides, faculty recommendations, and annotated peer reviews.

Figure 1: The system's three-phase pipeline: input processing (merging and annotating LaTeX projects), specialized agent analysis guided by expert skill files, and aggregation of actionable comments mapped back to the Overleaf source.

Each agent receives not just the relevant source text but also tailored expertise from the skill library (over 40 curated files, 16,000 words total), which encompasses conventions and best practices for different paper types, sections, venue requirements, and rhetorical strategies. This modular expertise assignment enables both high coverage and precision for feedback, with section-specific agents operating on granular slices of the document, while global agents maintain holistic oversight.

Human-Centered Interaction and Integration

PaperMentor's feedback is delivered as Overleaf-native inline comments, rendered seamlessly via the ShareJS operational protocol. Authors do not relinquish control over their drafts; instead, they receive targeted feedback analogous to that provided by experienced mentors. The frontend, built atop the Overleaf Community Edition, allows users to specify the backbone LLM (e.g., GPT-5.2), target venue, and optionally upload a role model paper to further condition agent behavior.

Figure 2: The PaperMentor panel integrated into Overleaf: users configure the review and receive file-by-file, severity-annotated feedback within their established writing workflow.

Comments produced by PaperMentor are comprehensive and context-anchored, covering a spectrum from critical omissions (e.g., vague contribution statements) to detailed technical corrections (e.g., LaTeX cross-referencing syntax). Each suggestion is mapped to a specific file location, includes a severity label, and is deduplicated and prioritized in aggregation.

Evaluation and Empirical Results

PaperMentor was evaluated in a user study involving 14 AI researchers and 80 papers (ICLR 2026 submissions and representative student drafts). Comments generated by PaperMentor and by direct LLM prompting (baseline) were rated by blinded annotators along three axes: validity, actionability, and conciseness. PaperMentor substantially outperformed the baseline: validity increased by 6.5 percentage points, actionability by 4.1 points, both with $p < 0.001$; conciseness was marginally lower, reflecting more detailed feedback due to the structured skill library.

Figure 3: Distribution of comments across review domains and human annotation scores for validity, actionability, and conciseness, demonstrating high effectiveness and generalization across all paper sections.

Approximately 90.6% of comments were rated actionable and 67.5% valid, compared to 86.5% and 61.0% for direct prompting—demonstrating that skill-guided, expert-informed agent decomposition translates directly into higher-quality, actionable feedback. The feedback distribution also matched author and reviewer attention: methods, results, and core sections attracted the bulk of commentary, with skill-driven prioritization for critical areas.

Theoretical and Practical Implications

PaperMentor's architecture highlights several advances in AI-driven academic writing support:

• Multi-Agent Decomposition: Decomposing the drafting review task across skill-informed, specialized agents increases both coverage and domain precision while maintaining modularity for extensibility.
• Preservation of Authorial Agency: By localizing suggestions to comments (rather than direct rewriting), the system prioritizes expertise-driven mentoring and avoids over-automation, a design validated by prior HCI and NLP studies [lee2024design, dhillon2024shaping].
• Skill Library as Infrastructure: The decoupling of expert writing guidance from agent logic allows for systematic, scalable evolution of best practices in scientific communication, facilitating both maintenance and community-driven enhancement.
• Section and Venue Awareness: Conditioning feedback on paper type, venue, and role models enables rapid adaptation to evolving standards across AI subfields and conferences.

In terms of theoretical implications, PaperMentor demonstrates the utility of explicit expert knowledge over in-context prompting for structure-sensitive tasks and the importance of feedback granularity and anchoring for actionable suggestions.

Limitations and Future Directions

PaperMentor operates on LaTeX source, with current limitations in capturing PDF-rendered issues such as visualizations and numerical table alignment. The agent specialization, while yielding focused commentary, can result in validity errors when document-wide context is fragmented; future implementations should address this through shared content representations or dynamic grounding. The evaluation corpus, while diverse, is limited in size; future benchmarks should incorporate direct comparison with senior researcher-generated feedback. Additionally, continued expansion and diversification of the skill library—especially toward inclusivity of rhetorical variations and venues—is necessary to prevent homogenization of writing style and scope.

Conclusion

PaperMentor establishes that the integration of a human-centered, multi-agent system, grounded in a curated expert skill library, significantly improves the quality and actionability of writing feedback for AI research papers. The system sets a new standard for AI-supported academic writing, moving from generic rewrites to context-sensitive mentorship. Its architecture and skill library model provide a template for future developments in intelligent, author-centered writing assistance, with promising directions for document-level grounding, inclusivity, and large-scale, community-driven knowledge curation.