SciLint Score Metric

• SciLint Score is a paper-centric metric that quantifies a manuscript’s integrity and contribution by evaluating internal consistency, referencing quality, and claim support.
• It employs a local verification pipeline that combines deterministic checks, LLM-powered figure analysis, and bibliographic cross-checks without sending manuscripts to external services.
• The score’s multiplicative structure penalizes unsupported claims, integrating both robust integrity measures and an experimental contribution evaluation based on established scientific frameworks.

SciLint Score is a paper-centric metric proposed to measure, in a single number, whether a manuscript’s evidence chain is real and whether its claims matter. It is defined on top of the sciwrite-lint pipeline, an open-source linter for scientific manuscripts that runs entirely on the researcher’s machine using free public databases, open-weights models, and a single consumer GPU, with manuscripts not sent to external services. The proposal is explicitly motivated by the claim that journal gatekeeping is slow and prestige-biased, while open science offers no quality assurance, and that AI-assisted writing exacerbates both conditions by producing more papers faster than either system can absorb (Samsonau, 9 Apr 2026).

1. Motivation and intended measure

SciLint Score is introduced as a response to what the paper characterizes as two inadequate quality-assurance regimes. Journal gatekeeping is described as claiming to verify both integrity and contribution while in practice measuring prestige; peer review is described as slow, biased, and capable of missing fabricated citations even at top venues. Open science is described as providing no quality assurance, leaving the author’s integrity as the only filter between AI-generated text and the public record. The proposed alternative is to measure the paper itself rather than the venue or dissemination channel (Samsonau, 9 Apr 2026).

The metric has two stated aims. The first is integrity, defined as whether the manuscript’s internal reporting is consistent and whether the references it relies on, including their upstream bibliographies, are real, correctly cited, and supportive of the claims made about them. The second is contribution, implemented as an experimental component that operationalizes five frameworks from philosophy of science—Popper, Lakatos, Kitcher, Laudan, and Mayo—into computable structural properties. In the paper’s framing, integrity asks whether the evidence is real, whereas contribution asks whether the claims matter (Samsonau, 9 Apr 2026).

The contribution component is explicitly marked as experimental, while the integrity component is identified as the core of the tool and the component evaluated in the paper. This division is important for interpretation: a low or high overall score may reflect mature integrity checks combined with a contribution model that is still intended for community development rather than definitive assessment (Samsonau, 9 Apr 2026).

2. Formal structure of the score

SciLint Score for a paper $p$ with references $r_1,\dots,r_n$ is presented as a multiplicative composition of three factors: internal consistency, referencing quality, and contribution. The operational form reconstructed in the paper is

$$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$

Here $I(p) \in [0,1]$ is the internal consistency score, $V(p,r_i) \in [0,1]$ is the claim-support score for the use of reference $r_i$ in paper $p$, $R(r_i) \in [0,1]$ is the per-reference reliability score, $w_i$ is a citation-purpose weight, and $\sum_{a \in \mathcal{A}} \beta_a C_a(p)$ is the contribution factor, with default equal weights $r_1,\dots,r_n$0 across the five contribution axes (Samsonau, 9 Apr 2026).

The intended interpretation is multiplicative rather than additive. A manuscript can therefore be strongly penalized by deficiencies in any one of the three factors. This suggests that the framework treats unsupported or unreliable evidence chains as non-compensable by argument structure alone, and conversely that structurally ambitious claims do not suffice when internal reporting or referencing quality is weak.

The paper also defines variants. In the integrity-only default, $r_1,\dots,r_n$1 is set to $r_1,\dots,r_n$2 unless the contribution mode is invoked. In standalone contribution calibration, $r_1,\dots,r_n$3 is set to $r_1,\dots,r_n$4 to isolate evaluation of the contribution component. In referencing-quality-only diagnostics, the internal-consistency and contribution factors are omitted to focus on the middle term (Samsonau, 9 Apr 2026).

3. Integrity verification pipeline

The underlying sciwrite-lint system executes a structured pipeline over a manuscript in LaTeX or PDF form. Deterministic text checks include dangling citations, dangling cross-references, and unreferenced figures. A local vision-LLM then produces figure descriptions, after which eleven full-paper checks are run using vLLM on Qwen3 8B with automatic prefix caching. The listed checks include numbers-vs-tables, percentages-sum, sample-size-consistency, arithmetic-consistency, causal-language-audit, abstract-body-alignment, statistical-reporting, figure-caption/content consistency, axis-labels, and figure-data-vs-table (Samsonau, 9 Apr 2026).

Reference verification proceeds through external metadata services while keeping manuscript text local. The paper lists CrossRef, OpenAlex, Semantic Scholar, Open Library, and the Library of Congress for existence and metadata lookup, and Retraction Watch for retraction status. Cited papers are downloaded and parsed with GROBID into section-structured markdown, and Snowflake Arctic Embed M v2.0 embeddings are used for semantic retrieval. The same local LLM checks used on the target manuscript are then run on the cited paper’s content. Claim verification is performed by KNN retrieval of relevant sections followed by local LLM classification of support (Samsonau, 9 Apr 2026).

A distinctive feature is one-level citation chasing. For each cited paper, the system verifies the existence and metadata of that paper’s own bibliography via OpenAlex and Semantic Scholar, and computes hallucination rates together with metadata mismatch and retraction counts. This extends verification beyond direct references to the immediate upstream evidentiary layer (Samsonau, 9 Apr 2026).

Internal consistency is scored as the fraction of non-error findings among checked manuscript-level checks:

$r_1,\dots,r_n$5

Skipped and indeterminate checks are excluded from $r_1,\dots,r_n$6, and warnings do not reduce $r_1,\dots,r_n$7. Consequently, $r_1,\dots,r_n$8 if no errors are found, and decreases proportionally with the observed error rate (Samsonau, 9 Apr 2026).

4. Per-reference reliability and referencing quality

Each cited reference receives a per-reference reliability score $r_1,\dots,r_n$9 that aggregates metadata and tier-of-verification signals, local consistency checks on the cited paper, and one-level bibliography verification. The paper specifies tiered base scores: T1, meaning API-verified with full text, scores $$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$0; T2, API-verified without full text, scores $$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$1; T3, not found in APIs, scores $$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$2; and retracted references score $$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$3. An Expression of Concern applies a multiplier of $$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$4 to the base. Metadata mismatches, cross-ID mismatches, and non-formal document type each incur specified penalties; consistency warnings and errors in the cited paper also incur penalties; bibliography hallucination rate is subtracted proportionally, with additional capped deductions for bibliography mismatches and bibliography retractions (Samsonau, 9 Apr 2026).

The operational form given for the reliability score is

$$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$5

Scores are clamped to $$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$6. Documents above a size threshold of approximately 50 pages are excluded from consistency checks and receive a neutral consistency score of $$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$7, so that books are not penalized purely for scale (Samsonau, 9 Apr 2026).

Referencing quality is then computed as a weighted mean of $$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$8 across references. The weights are citation-purpose weights produced by the linter’s cite-purpose check.

Citation purpose	Weight
Evidence	1.0
Contrast	0.9
Method	0.8
Definition	0.7
Example	0.6
Attribution	0.5
Tool	0.4
Context	0.2

The paper emphasizes that evidence failures weigh more heavily than context failures: an evidence citation with $$S(p) \;=\; I(p) \;\times\; \frac{\sum_{i=1}^{n} w_i \, V\!\bigl(p, r_i\bigr) \, R\!\bigl(r_i\bigr)}{\sum_{i=1}^{n} w_i} \;\times\; \sum_{a \in \mathcal{A}} \beta_a \, C_a(p).$$9 is a significant integrity problem, whereas a context citation with $I(p) \in [0,1]$0 barely registers. Conversely, a manuscript whose citations all classify as context has low referencing quality even if all references exist. This weighting scheme is therefore not a mere confidence aggregation; it is intended to encode argumentative function (Samsonau, 9 Apr 2026).

5. Experimental contribution model

The contribution component is framed as an experimental extension grounded in five frameworks from philosophy of science. Each axis is defined on $I(p) \in [0,1]$1 and contributes to a convex combination with default equal weights (Samsonau, 9 Apr 2026).

Popper (empirical content) is defined as the fraction of claims that are specific and falsifiable, with the local LLM classifying claim type, specificity, and testability:

$I(p) \in [0,1]$2

Lakatos (progressiveness) is defined as the ratio of novel predictions to the sum of novel predictions and ad-hoc accommodations:

$I(p) \in [0,1]$3

Kitcher (explanatory unification) is operationalized through citation-graph bridging across communities by clustering the citation graph and rewarding edges that connect distinct clusters:

$I(p) \in [0,1]$4

Laudan (problem-solving effectiveness) is defined as the balance between problems claimed solved and limitations acknowledged:

$I(p) \in [0,1]$5

Mayo (test severity) is defined by the presence of ablations, strong baselines, and alternative explanations addressed. The paper gives both a product-like expression in prose and a normalized-sum form described as the simplest released code:

$I(p) \in [0,1]$6

The composite contribution factor is

$I(p) \in [0,1]$7

with default $I(p) \in [0,1]$8 for each of the five axes. A dampening penalty applies when a paper is progressive in the Lakatos sense but shows very low Laudan-style problem-solving effectiveness. Specifically, if $I(p) \in [0,1]$9 and $V(p,r_i) \in [0,1]$0, the contribution factor is multiplied by a factor in $V(p,r_i) \in [0,1]$1, with the paper providing the linear instantiation

$V(p,r_i) \in [0,1]$2

The stated rationale is to dampen bold but ineffective claims; the paper gives LK-99 as the motivating case for this penalty (Samsonau, 9 Apr 2026).

The paper also states clear epistemic limits for this component. Claim testability and specificity classification are described as error-prone, and the underlying philosophical axes are said to have known critiques, including Duhem–Quine, Lakatos’s degeneracy, and domain-dependent unification. The contribution scores are therefore presented as heuristics rather than definitive measures (Samsonau, 9 Apr 2026).

6. Implementation, evaluation, and limitations

The implementation is local by design. Installation is via pip install sciwrite-lint, and example commands include sciwrite-lint check path/to/manuscript.pdf, sciwrite-lint check path/to/manuscript.tex --contrib, sciwrite-lint eval-calibration, and sciwrite-lint eval-real-world-matching. The paper specifies local LLMs with open weights, deterministic API lookups over HTTPS transmitting only citation metadata, and a single consumer GPU with 16GB or more VRAM recommended. The model stack comprises Qwen3-VL 2B or 8B for figure description, Snowflake Arctic Embed M v2.0 for embeddings, and Qwen3 8B served via vLLM with FP8 KV cache, APC, and chunked prefill. Sequential scheduling is described as VL, then embeddings, then vLLM (Samsonau, 9 Apr 2026).

The matching engine is described as multi-signal scoring for cases where identifiers are missing. It performs loose search per API and combines title similarity, author overlap with variant expansion, a quadratic year penalty that tolerates $V(p,r_i) \in [0,1]$3 year but strongly penalizes large mismatches, and a venue tiebreak. The best candidate is accepted above $V(p,r_i) \in [0,1]$4; otherwise the reference is classified as T3. Configuration is exposed through .sciwrite-lint.toml, including enabling or disabling checks, thresholds, citation-purpose weights, dampening, and per-discipline $V(p,r_i) \in [0,1]$5 values (Samsonau, 9 Apr 2026).

Evaluation of the integrity-oriented pipeline is reported on 30 unseen papers, comprising 13 from arXiv and 17 from bioRxiv. In error-injection tests, 68 synthetic errors were injected and 67 were detected: dangling citations were detected at $V(p,r_i) \in [0,1]$6, dangling references at $V(p,r_i) \in [0,1]$7, yielding aggregate recall of $V(p,r_i) \in [0,1]$8 with zero false positives for those deterministic checks. Across the full pipeline, 27 of 30 papers completed, preliminary SciLint Scores ranged from $V(p,r_i) \in [0,1]$9 to $r_i$0 with mean $r_i$1, and the system produced 1,755 findings, averaging 65 per paper. The findings were dominated by reference-exists checks and reference-accuracy checks, especially for PDFs lacking DOIs (Samsonau, 9 Apr 2026).

The paper also reports a false-positive analysis conducted before the improved matching engine. The top 20 findings per paper, 379 in total, were adjudicated by Claude Sonnet and categorized as 14% true positive, 60% false positive, and 26% uncertain. The elevated false-positive rate was concentrated in reference-database checks on PDF papers without DOIs under naive title search. Two mitigations were then implemented: DOI extraction via GROBID TEI when available for deterministic verification, and replacement of naive matching with the composite matching engine (Samsonau, 9 Apr 2026).

Standalone contribution calibration was performed on 20 open-access papers of at most 25 pages across 7 domains using 38 ordinal constraints; 30 of the 38 constraints passed, or 79%. Nobel Prize papers ranked in the top quartile and retracted papers in the bottom quartile. The bold-claims penalty flagged LK-99. The paper also notes under-detection on some axes, including RCT severity and unification, the latter often returning zero because of clustering limitations (Samsonau, 9 Apr 2026).

Several limitations are specified. First-run latency on a paper with roughly 50 references can reach approximately 30 minutes, dominated by claim verification, though cached runs complete in minutes. Figure analysis is sensitive to representation: raster and TikZ figures are described by the VL model, but pure vector graphics in PDFs without LaTeX sources may be missed. For injected figure errors, the 8B VL model detected 100% of 62 cases, whereas the 2B model detected 85%; downstream caption-versus-content and text-versus-figure checks are said to require further false-positive calibration. Bibliographic coverage is affected by a reduction in open abstract coverage after Elsevier’s request to remove abstracts from OpenAlex, from 82% to 22.5%, although the core verification uses titles, authors, and DOIs rather than abstracts. Finally, the paper emphasizes that SciLint Score does not claim to measure novelty, importance to society, or overall scientific merit; it measures verifiable structural properties of evidence chains and argument structure, and is intended to inform rather than replace expert judgment (Samsonau, 9 Apr 2026).