sciwrite-lint: Verification Infrastructure for the Age of Science Vibe-Writing

Abstract: Science currently offers two options for quality assurance, both inadequate. Journal gatekeeping claims to verify both integrity and contribution, but actually measures prestige: peer review is slow, biased, and misses fabricated citations even at top venues. Open science provides no quality assurance at all: the only filter between AI-generated text and the public record is the author's integrity. AI-assisted writing makes both worse by producing more papers faster than either system can absorb. We propose a third option: measure the paper itself. sciwrite-lint (pip install sciwrite-lint) is an open-source linter for scientific manuscripts that runs entirely on the researcher's machine (free public databases, a single consumer GPU, and open-weights models) with no manuscripts sent to external services. The pipeline verifies that references exist, checks retraction status, compares metadata against canonical records, downloads and parses cited papers, verifies that they support the claims made about them, and follows one level further to check cited papers' own bibliographies. Each reference receives a per-reference reliability score aggregating all verification signals. We evaluate the pipeline on 30 unseen papers from arXiv and bioRxiv with error injection and LLM-adjudicated false positive analysis. As an experimental extension, we propose SciLint Score, combining integrity verification with a contribution component that operationalizes five frameworks from philosophy of science (Popper, Lakatos, Kitcher, Laudan, Mayo) into computable structural properties of scientific arguments. The integrity component is the core of the tool and is evaluated in this paper; the contribution component is released as experimental code for community development.

• The paper introduces sciwrite-lint, a fully local verification pipeline that enhances manuscript integrity by validating references and argumentative structure through deterministic checks.
• The methodology employs multi-signal matching, semantic consistency analysis, and deep citation graph traversal, achieving 98.5% recall in error injection tests.
• The SciLint Score operationalizes integrity and contribution via computable philosophical frameworks, offering a robust alternative to traditional peer review.

Authoritative Analysis of "sciwrite-lint: Verification Infrastructure for the Age of Science Vibe-Writing" (2604.08501)

Motivation and Problem Statement

The paper addresses fundamental inadequacies in the scientific manuscript verification ecosystem. Existing paradigms—journal gatekeeping and open science—fail to directly assess evidence validity and argumentative contribution. Peer review is empirically slow, biased, and ineffective at detecting errors (e.g., fabricated citations pass undetected at top venues), while open repositories lack any filtering mechanism beyond author integrity. The rise of AI-assisted writing exacerbates these failures, introducing a prolific stream of manuscripts with systematic verification gaps, particularly when relying on local, smaller generative models which are demonstrably prone to reference hallucination.

sciwrite-lint: Verification Pipeline and Architecture

The proposed solution is sciwrite-lint, an open-source, fully local verification pipeline that performs deterministic integrity checks and semantic validation using open-weight LLMs and public databases. Notably, the manuscript is never sent to external services, preserving privacy and compliance with contemporary conference reviewer policies. The pipeline performs:

• Figure extraction and semantic consistency analysis via vision-LLMs
• Reference existence and metadata verification against canonical sources (DOI, arXiv, PMID, ISBN, LCCN)
• Retraction cross-referencing with Retraction Watch
• Full-text download, parsing, and claim retrieval for cited papers, including secondary-level bibliography checks
• Aggregate reliability scoring per reference using convergent signals from API, LLM, and bibliography verification

Figure 1: LIGO (2016) exemplifies evidentiary chain integrity—a key benchmark in pipeline calibration.

The pipeline achieves reproducibility and auditability on a single consumer GPU, with batching and sequential scheduling that saturate available VRAM efficiently. System architecture supports scalable citation graph traversal; deeper verification becomes practical as consumer hardware advances.

SciLint Score: Beyond Integrity

A significant extension is the SciLint Score, a metric operationalizing both integrity and contribution. Integrity is defined as the product of internal manuscript consistency, claim-support scores for references, purpose weights, and reliability signals. The contribution component draws from five philosophical frameworks—Popper, Lakatos, Kitcher, Laudan, Mayo—mapping core scientific values to computable argumentative properties (specificity/falsifiability, progressiveness, unification, problem-solving, test severity). Novelty is omitted in favor of progressiveness due to retrospective ascertainment limitations.

Radar plot diagnostics illustrate profile differences across high- and low-quality papers, with automated calibration against ordinal expectation constraints (e.g., LIGO 2016 ranks above LK-99 2023).

Empirical Results and Evaluation

Robustness and efficacy are demonstrated via comprehensive evaluations:

• Error injection achieves 98.5% recall (dangling references, cross-references) with zero false positives
• Full pipeline runs on 30 unseen arXiv/bioRxiv papers, producing aggregate findings per paper and per-reference reliability scores
• False positive adjudication highlights that deterministic identifier extraction and matching engine improvements are critical for correcting reference verification on PDF-only submissions
• SciLint Score calibration against 20 open-access papers produces expected ordinal rankings, with Nobel Prize works consistently scoring in the top quartile and retracted fraudulent papers in the bottom
• Bold claim penalty mechanism (for vacuous progressiveness without substantive problem-solving) is shown to function correctly

Technical Contributions

Highlighted innovations include:

• Fully automated, manuscript-local, pip-installable bibliography verification
• Multi-signal matching engine for resilient reference identification absent structured identifiers
• Deep citation graph traversal for claim and bibliography verification
• Graduated citation purpose classification (eight roles with weighted argumentative contribution)
• Empirically optimized semantic checks with eval-driven prompt engineering
• Open-weights reproducibility and configurable pipeline diagnostic outputs

The Verification Prior Hypothesis is posited: smaller LLMs may be intrinsically better for manuscript verification due to weaker prior conformity and reduced sycophancy, supported by scaling-law evidence and comparative model benchmarking.

Implications and Future Directions

The framework reshapes quality assurance into direct assessment of evidence and argument rather than indirect prestige proxies. Practical applications span manuscript pre-submission, journal pre-reviewer filtering, automated ingestion for AI RAG systems, and feedback for authors. The open-source infrastructure invites community-led optimization of scoring weights, model selection, and disciplinary calibration.

Algorithmic limitations (e.g., figure vector coverage, axis-specific claim classification) are identified for future refinement. The five-axis contribution scoring constitutes an experimental foundation for systemic improvement in manuscript assessment; further scaling of model fine-tuning, calibration sets, and domain-specific evaluation are logical next steps.

The paradigm is well-positioned to address evidence chain fragility and AI-generated knowledge propagation, offering rigorous gatekeeping without centralized authority.

Conclusion

sciwrite-lint delivers a structured, reproducible solution for manuscript verification, enabling integrity and contribution scoring grounded in computable philosophical principles. The tool provides actionable diagnostics and reliability signals, directly addressing both practical and theoretical weaknesses in current scientific quality assurance. Its open-source nature ensures transparency and auditability, vital qualities for any accountability infrastructure adopted at scale.