Accepted with Minor Revisions: Value of AI-Assisted Scientific Writing (2511.12529v1)

Abstract: LLMs have seen expanding application across domains, yet their effectiveness as assistive tools for scientific writing -- an endeavor requiring precision, multimodal synthesis, and domain expertise -- remains insufficiently understood. We examine the potential of LLMs to support domain experts in scientific writing, with a focus on abstract composition. We design an incentivized randomized controlled trial with a hypothetical conference setup where participants with relevant expertise are split into an author and reviewer pool. Inspired by methods in behavioral science, our novel incentive structure encourages authors to edit the provided abstracts to an acceptable quality for a peer-reviewed submission. Our 2x2 between-subject design expands into two dimensions: the implicit source of the provided abstract and the disclosure of it. We find authors make most edits when editing human-written abstracts compared to AI-generated abstracts without source attribution, often guided by higher perceived readability in AI generation. Upon disclosure of source information, the volume of edits converges in both source treatments. Reviewer decisions remain unaffected by the source of the abstract, but bear a significant correlation with the number of edits made. Careful stylistic edits, especially in the case of AI-generated abstracts, in the presence of source information, improve the chance of acceptance. We find that AI-generated abstracts hold potential to reach comparable levels of acceptability to human-written ones with minimal revision, and that perceptions of AI authorship, rather than objective quality, drive much of the observed editing behavior. Our findings reverberate the significance of source disclosure in collaborative scientific writing.

• The paper examines a large-scale, incentivized RCT that manipulates abstract authorship and source disclosure to assess AI assistance in scientific writing.
• It demonstrates that AI-generated abstracts typically require fewer edits unless modulated by the editor's expertise, highlighting differences in perceived readability.
• Results suggest that transparent attribution modestly shifts editorial behavior without affecting peer review acceptance, validating AI as a viable co-author.

Assessing the Value and Dynamics of AI-Assisted Scientific Writing

Experimental Design and Methodology

This paper presents a large-scale, incentivized RCT designed to critically evaluate the behavioral and qualitative impact of AI writing assistance in scientific abstract composition. The authors construct a double factorial ($2\times2$) experiment, manipulating both the source of each abstract (human-authored vs. GPT-4o-generated) and whether authors are informed of that source. Participants, all with computer science backgrounds and varying levels of academic expertise (from undergraduates to PhDs), are tasked with editing provided abstracts based on research excerpts to produce a peer-review-ready product. Reviewer panels, blinded to original content and editing provenance, determine acceptance through majority decision.

The experimental workflow is comprehensively illustrated, emphasizing controlled exposure to four conditions (AI-noInfo, Human-noInfo, AI-withInfo, Human-withInfo), careful interface design to preclude external text manipulation or additional AI assistance, and collection of detailed keystroke-level editing data.

Figure 1: The randomized controlled workflow covering assignments, editing, reviewing, and verdict aggregation across varied information conditions.

Methodological innovations include:

• Firepad-based web editing interfaces: Ensuring all edits are recorded granularly and copy-paste into external tools is inaccessible.

  Figure 2: Interface and graphical author instructions for direct in-system editing, eliminating off-platform modifications.

  

  Figure 3: Visual prohibitions on external editing pathways, ensuring protocol compliance and analytic integrity.

• Sophisticated reviewer assignment via semantic alignment, ensuring robust topical fit and reviewer independence.
• Incentive-aligned economic design to elicit effortful, high-fidelity editing and reviewing behavior.

Patterns of Editorial Effort and Influence of Source Attribution

Unattributed AI-generated abstracts, on average, received significantly fewer edits than human-authored ones (63-character reduction; $p=0.0293$), attributed to higher perceived readability. However, education level modulates this: while less-experienced authors treat fluent AI outputs as near-publishable, PhD-level editors are more discerning and edit AI text more thoroughly.

Figure 4: Editing intensity is generally lower for AI-generated abstracts; editing patterns flip with increasing author expertise, and perceived readability directly drives editorial confidence.

Authors' self-assessed ability to distinguish AI-generated text correlates with more intensive editing in the AI-noInfo condition, but this effect disappears when controlling for confidence and readability, indicating that editorial effort is largely a function of subjective readability and the author’s epistemic certainty rather than source suspicion alone. Reviewer acceptance rates are statistically equivalent across human and AI conditions when the source is hidden, reinforcing that editorial transformation acts as an equalizer under blind peer review.

Disclosure Effects and Social-Cognitive Mechanisms

The disclosure of source information markedly modulates editorial effort and confidence. Revealing human authorship decreases the extent of editing (statistically significant, $p=0.045$), suggesting deference or social restraint. Disclosure for AI-generated abstracts yields a non-significant reduction in edits ($p=0.176$); the magnitude of this effect, however, is contingent on AI adoption attitudes. Authors inclined to use AI make fewer edits when its use is disclosed, while AI-averse authors’ editing remains unaffected.

Figure 5: Source disclosure reduces editing for human-authored abstracts; individual AI attitudes moderate editorial effort for AI-generated abstracts; disclosure-induced confidence shifts are marginal.

Both confidence and acceptance outcomes are minimally influenced by disclosure, except for a marginal increase in acceptance for AI-generated abstracts in the withInfo condition ($\Delta=0.068,p=0.081$), driven primarily by the quantum of editorial work rather than source itself.

Stylistic Analyses of Edits

A comprehensive computational linguistic analysis demonstrates that editorial behavior towards AI-generated and human-authored abstracts diverges in systematic, quantifiable ways. Without attribution, edits to AI-generated text improve topic-continuity and reduce nominalization—enhancing readability and clarity—while edits to human text inadvertently make it heavier and less cohesive.

Figure 6: Delta distributions in stylistic metrics reveal that AI-generated abstracts receive edits that systematically improve cohesion and information density, whereas human-authored text is often made syntactically more complex.

Under explicit disclosure, AI abstracts are revised to be more concise and information-dense, with reductions in passive constructions and subject-onset delays. Human-authored text, when attributed, is edited to lengthen openings and delay thematic subjects, reflecting a shift towards a more traditional or deferential style. Notably, mere disclosure of source alone does not independently shift writing style; it is the interaction between disclosure and the underlying text structure that governs these changes.

Qualitative Insights Into Authorial Strategy

Post-experiment interviews corroborate quantitative findings and illuminate nuanced authorial rationales:

• Readability optimization: AI abstracts are actively simplified for clarity; human drafts are structurally reorganized.
• Emphasis and rhetorical restructuring: Disclosure prompts authors to foreground contributions and reshape narratives, particularly for human-authored texts.
• Calibration of authority and accountability: Attribution alters critical stance, inducing either trust (for human texts) or scrutiny (for AI).
• Intrinsic responsibility: All authors, regardless of treatment, perceive a core obligation to uphold scientific integrity.

Practical, Theoretical, and Future Consequences

The empirical results situate LLM-driven scientific writing as not only plausible but, after minimal revision, on par with human-produced prose in peer-reviewed contexts. Expert review displays no reliable source bias, and acceptance is largely determined by the carefulness and intensity of human edits rather than initial provenance.

Key implications include:

• Editorial workload allocation: AI assistance can reduce the time required to reach publishable quality, particularly for less-experienced writers, provided oversight persists.
• Epistemics of trust and algorithm aversion: The observed editorial scrutiny post-disclosure is emblematic of algorithm aversion, but this behavior translates to actionable improvements and does not propagate into downstream reviewer rejection.
• Attribution policy and AI transparency: Clear attribution does not harm peer review outcomes but subtly reshapes editing strategies, suggesting that transparency can be adopted without negative impact.

From a theoretical perspective, the findings emphasize the centrality of cognitive biases—specifically, automation bias and algorithm aversion—in shaping human-AI collaborative dynamics. These phenomena are observable at the behavioral rather than simply self-reported attitudinal level.

Future research directions logically include:

• Domain transferability: Extending controlled evaluations to non-CS scientific domains where discourse conventions may differ.
• Collaborative and longitudinal studies: Assessing team-based co-writing with AI and tracking how reliance and skill development evolve over repeated exposure.
• Full-manuscript synthesis and multimodal integration: Moving beyond abstracts to entire papers, incorporating figures, tables, and supplementary material.

Conclusion

This paper offers robust evidence that LLMs are viable co-authors for scientific abstract writing, contingent on post-hoc domain-expert intervention. Editorial effort, not original authorship, most substantially predicts acceptance under blinded peer review. Attribution effects manifest in measurable, though context-dependent, editorial strategies rather than reviewer gatekeeping. Practically, careful AI integration—paired with transparent disclosure and conscientious human editing—can enhance both efficiency and consistency in scientific communication. Theoretically, these results advance our understanding of human-AI collaborative behaviors, highlighting the necessity of empirical, incentivized methodologies for ongoing assessment of AI’s role in professional authorship.