---
title: 'PaperJury: Automated Manuscript Governance'
url: https://www.emergentmind.com/topics/paperjury
type: topic
---

# PaperJury: Automated Manuscript Governance

PaperJury is a closed-loop **review-verdict-revise-verify** system for the **pre-submission hardening** of already-written LaTeX computer science papers. It is designed for a setting that differs from drafting assistance because it requires **adversarial whole-paper review**, **explicit no-fix outcomes**, and **bounded artifact-safe revision**. Its central architectural claim is a **deterministic-versus-semantic split**: deterministic orchestration manages decomposition, a frozen claim spine, a durable ledger, routing, stopping, and exact-once patch application, while semantic agents are limited to bounded review, judgment, and repair [2606.16322].

## 1. Task definition and governing thesis

PaperJury addresses manuscripts that are locally fluent but globally unsound. The target failures are manuscript-level problems such as unsupported claims, cross-section contradictions, overstatement of results, and positioning that breaks once the whole paper is considered. The motivating example is a cross-section inconsistency in which one table claims best benchmark performance while a later breakdown table shows a baseline with a higher number. The system therefore treats manuscript improvement as **pre-submission hardening** rather than generic writing assistance [2606.16322].

The paper distinguishes this task from adjacent systems. Drafting assistants help write or expand a paper; generic writing assistants improve wording; critique generators stop at comment generation; and judge-centered loops let an LLM critique, judge, revise, and decide when it is done. PaperJury argues that these alternatives lack some combination of **durable issue identity across rounds**, **deterministic routing from critique to adjudication**, and **manuscript control** able to reject invalid concerns or defer author-dependent ones. This motivates the thesis that the “load-bearing safety and completion logic” should reside in deterministic orchestration rather than model discretion [2606.16322].

The system is organized around a **precision-recall-cost trilemma**. It aims to find real problems, avoid hallucinated complaints, and do so without the token and runtime expansion associated with exhaustive critique generation. A plausible implication is that PaperJury is best understood as a control architecture for manuscript governance rather than as a stronger reviewer prompt.

## 2. Deterministic orchestration and persistent manuscript state

The deterministic layer begins with manuscript decomposition. For an input LaTeX manuscript $x$, PaperJury defines a deterministic decomposition $D(x)$ into sections, passages, anchors, and cross-reference targets. This yields stable addressable units before any semantic processing occurs. Review comments and edits are subsequently tied to these anchors rather than to fragile raw spans [2606.16322].

The second state-bearing object is the **frozen claim spine** $S$, extracted before revision and described as the manuscript’s “protected semantic backbone.” The function of $S$ is constraint rather than summary: revisions may improve support, wording, and local organization, but should not silently rewrite central claims. Together, $D(x)$ and $S$ establish the system’s basic invariants [2606.16322].

PaperJury then performs **bounded holistic review** using a small number of whole-paper reviewers. The reviewer count is $N$, clamped to $[2,4]$, with default $N=3$. Each reviewer reads the full manuscript once and emits evidence-anchored weaknesses plus coverage signals. Review is “holistic” because it spans the entire manuscript, but “bounded” because first-pass critique generation is deliberately limited. The paper contrasts this with exhaustive fan-out baselines that review each unit under multiple lenses; PaperJury instead triggers targeted re-read only when deterministic anti-skim checks indicate weak coverage or poor grounding [2606.16322].

All discovered concerns are merged into a **durable ledger** $L$. The ledger stores issue identity, evidence and provenance, verdicts, patch history, and application history. Near-duplicate complaints are consolidated, so the same underlying problem persists as one tracked object across rounds. The paper formalizes the broader issue set as $I$, with contestability labels $c_i$ used for downstream routing. This durable-ledger design is one of PaperJury’s main differences from iterative critique loops that recreate ephemeral issue lists at each pass [2606.16322].

## 3. Due-process adjudication and terminal verdicts

After merge, PaperJury applies **deterministic routing** based on contestability. Mechanical and minor-substantive issues go through a cheaper “polish path,” while contestable substantive-major issues are escalated to a **due-process trial**. Routing is not left to model discretion. The paper treats this as a central safety property because expensive adjudication is invoked only when a mistaken first-pass decision would be consequential [2606.16322].

For a routed issue $i$, the system runs a trial procedure $T(i)$ and computes a terminal verdict in deterministic code:
$$
v_i = \mathrm{Verdict}(T(i), q, m).
$$
Here $q$ is a quorum threshold and $m$ is a majority threshold. The available terminal outcomes are exactly three:

- **invalid-drop**
- **valid-fixable**
- **author-required**

This three-way verdict space is integral to the system. PaperJury rejects a binary “fix or ignore” scheme because issue validity and machine editability are distinct questions. Some concerns are invalid and should be dropped. Some are valid but should not be auto-edited because they require new experiments, claim-level intent decisions, or author judgment. Only valid-fixable concerns proceed to automated repair [2606.16322].

The trial itself combines two perspectives: a **whole-paper defense** that argues against the criticism using global manuscript context, and a **decorrelated local-context jury** that examines localized evidence with reduced dependence on the original reviewer framing. This is described as a due-process mechanism rather than a free-form model debate. A plausible implication is that PaperJury treats review as an adversarially contestable process, not merely as criticism accumulation.

The direct audit of verdict classes shows that this three-way design is operational rather than decorative. In the evaluation, **invalid-drop** achieved $\mathrm{Acc_v}=0.872$, **valid-fixable** achieved $\mathrm{Acc_v}=0.913$, and **author-required** achieved $\mathrm{Acc_v}=0.860$ [2606.16322].

## 4. Bounded revision, guard chains, and exact-once patching

Only issues with $v_i=\text{valid-fixable}$ are allowed to generate candidate patches $P_i$. Each patch induces an **anchor-bounded diff** $a(P_i)$, meaning the change is explicitly tied to the deterministic anchor structure created during decomposition. This is meant to limit edit blast radius and prevent local repairs from mutating unrelated claims or neighboring sections [2606.16322].

Every patch is then passed through a **risk-proportional guard chain**. The paper formalizes the aggregate result as
$$
G(P_i) \in \{\text{pass}, \text{fail}\}.
$$
Risk is tracked with a discrete category $\rho_i$, with labels such as **LOW** and **RISKY**. LOW-risk patches take a lighter semantic path; RISKY patches trigger stronger audits. The guard chain includes anchor-bounded diff checks, cross-reference checks, semantic meaning or edit audits when triggered, compile checks, and application journaling [2606.16322].

The key gating rule for revision is:
$$
A = \{P_i \mid v_i = \text{valid-fixable} \land G(P_i)=\text{pass}\}.
$$
Thus, only patches attached to valid-fixable issues and passing all guards are applied. This is the system’s principal safety barrier against model-generated overreach [2606.16322].

Patch application itself is **exact-once** and journaled. The deterministic orchestrator records application history and supports revert semantics if later checks fail. This is designed to eliminate duplicate edits, stale state, and divergence between ledger state and manuscript state. The same deterministic philosophy governs termination. With $U_r$ denoting genuinely new issues in round $r$ and $C_r$ denoting issues closed in round $r$, the unattended loop stops when the deterministic predicate
$$
\tau(L, U_r, C_r, r) = \text{true}
$$
holds, subject to a five-round cap $K$ [2606.16322].

The paper’s evaluation treats **clean re-review** as a further control measure: after edits, the current manuscript state is re-read afresh rather than continuing from prior model outputs. This reduces contamination from earlier critique framing and is intended to surface new issues introduced by edits.

## 5. Empirical evaluation and operating characteristics

PaperJury is evaluated in a **two-arm expert-review evaluation** on **12 held-out LaTeX computer science papers**: four each from **Vision**, **NLP**, and **ML**. Arm 1 measures issue discovery quality against expert issue panels using panel-relative precision $P_{\mathrm{panel}}$, audit-corrected precision $P_{\mathrm{verified}}$, recall $R$, and macro-averaged $F1$. Arm 2 audits terminal verdicts, routing, and applied edits, yielding verdict accuracy $\mathrm{Acc_v}$, routing accuracy $\mathrm{Acc_r}$, edit-safety violation rate $\mathrm{ESVR}$, round count $K$, and wall-clock runtime $W$ [2606.16322].

The main baselines are **Forward-Only Rewriter**, **LLM Critic Only**, **LLM-as-Judge Review-Revise Loop**, and **Naive Unbounded Per-(Unit×Lens) Generator**. PaperJury’s headline results are:

- $P_{\mathrm{panel}} = 0.684$
- $P_{\mathrm{verified}} = 0.847$ with audited $n=98$ and 95% CI $[0.763, 0.905]$
- $R = 0.637$
- $F1 = 0.656$
- $\mathrm{Acc_v} = 0.887$
- $\mathrm{Acc_r} = 0.913$
- $\mathrm{ESVR} = 0.025$
- $K = 3.08 \pm 0.67$ rounds, with cap hits on $0/12$ papers
- $W = 2.47$ hours per paper

The paper also reports an average cost of **6.76 million tokens per paper** [2606.16322].

A concise comparison is given below.

| System | Key quality metrics | Safety / cost |
|---|---|---|
| **PaperJury** | $F1=0.656$, $\mathrm{Acc_v}=0.887$, $\mathrm{Acc_r}=0.913$ | $\mathrm{ESVR}=0.025$, $W=2.47$ h |
| **LLM Critic Only** | $F1=0.446$ | No verdict/edit audit path |
| **LLM-as-Judge Loop** | $F1=0.519$, $\mathrm{Acc_v}=0.681$ | $\mathrm{ESVR}=0.110$, $K=3.33\pm1.07$, cap hits $2/12$ |
| **Naive Unbounded Generator** | $R=0.721$, $F1=0.459$ | $W=8.37$ h, 31.4M tokens/paper |
| **Forward-only rewriter** | No issue list or verdicts | $\mathrm{ESVR}=0.240$, $W=0.31$ h |

These results support the paper’s central claim that deterministic orchestration improves the balance among issue quality, adjudication quality, edit safety, and completion behavior relative to judge-centered loops and rewrite-only baselines [2606.16322].

The ablation study is especially diagnostic. Removing **bounded review** reduces $F1$ by $0.077$ and increases runtime to **4.81 hours**. Removing **routing** drops verdict accuracy by **0.075** and increases runtime to **3.49 hours**. Removing the **trial** lowers $\mathrm{Acc_v}$ by **0.153** with little change in $F1$, isolating trial as the main verdict-quality component. Removing the **claim spine** raises $\mathrm{ESVR}$ from **0.029** to **0.112**. Removing the **guard chain** raises $\mathrm{ESVR}$ to **0.181** while reducing runtime to **1.94 hours**. This decomposition closely matches the paper’s stated thesis: bounded review mainly affects discovery quality and efficiency, trial mainly affects verdict reliability, and claim-spine plus guard-chain controls mainly affect edit safety [2606.16322].

The per-domain slices remain relatively stable: Vision, NLP, and ML all achieve $F1$ between **0.646** and **0.671**, $\mathrm{Acc_v}$ between **0.878** and **0.902**, and $\mathrm{ESVR}$ between **0.021** and **0.028**. The paper treats these as diagnostic rather than powered comparisons because there are only four papers per family [2606.16322].

## 6. Scope, limitations, and relation to adjacent “jury” systems

PaperJury is explicitly bounded to **LaTeX computer science manuscripts** and to the task of **pre-submission hardening**. It is not presented as a general writing assistant, a paper drafting system, or a replacement for author judgment. The presence of the **author-required** verdict class is itself evidence of that boundary: some upheld issues require new experiments, claim-level intent decisions, or author intervention and therefore remain outside automated repair [2606.16322].

The system’s main practical costs are operational complexity and inference budget. It requires deterministic decomposition, a durable ledger, routing logic, trial machinery, patch guards, compile verification, journaling, and revert semantics. It is also substantially more expensive than a forward-only rewriter, even though it is far cheaper than a naive exhaustive critique generator. The evaluation scope is also limited: 12 papers across three CS subfields, with only partial detail in the paper text on expert annotation protocol and inter-rater reliability [2606.16322].

Within the broader literature, PaperJury belongs to a growing family of “jury”-oriented evaluation systems, but its object and control philosophy are distinct. **SE-Jury** is an **LLM-as-Ensemble-Judge** metric for evaluating generated software artifacts through multiple prompt strategies and dynamic team selection [2505.20854]. **GrandJury** is a collaborative human-evaluation protocol built around dynamic rubrics, time-decayed aggregation, and traceability for settings without absolute ground truth [2508.02926]. **“Vibe Coding on Trial”** studies **unanimous LLM juries** for safety-first text-to-SQL acceptance, emphasizing false-accept control through unanimous committees [2602.18492]. PaperJury differs from these systems because it is centered on a **deterministic orchestration layer** that governs manuscript state, adjudication, stopping, and exact-once patch application, while semantic agents are intentionally bounded [2606.16322].

This suggests a broader classification. SE-Jury and unanimous LLM juries treat “jury” primarily as an evaluation ensemble. GrandJury treats it as a transparent, pluralistic governance protocol for human judgment. PaperJury treats it as **due-process review architecture** for bounded revision of a structured scientific artifact. In that sense, its most distinctive contribution is not a better reviewer model but a theory of where system authority should reside: semantic agents may read, argue, and draft, but deterministic code holds the ledger, the verdict thresholds, the stopping predicate, and the eraser.

Source: https://www.emergentmind.com/topics/paperjury