Towards an Interactive Evidence-RAG Peer-Review Workspace for the Journal of Digital History

Abstract: This preliminary paper presents an interactive Evidence-RAG workspace for editorial assessment of AI-assisted peer review in the Journal of Digital History. The workflow makes model recommendations easier to inspect by linking reviewer comments, paper evidence, retrieval traces, and reproducibility checks. The system does not replace editors or reviewers. It treats LLMs as auditable assistants whose outputs must be checked by human scholars. We describe the current pipeline: paper conversion, semantic chunking, vector indexing, retrieval-augmented evidence assessment, and a lightweight editorial interface. This is a preliminary version of a full paper associated with the accepted presentation "Towards an Interactive Evidence-RAG Peer-Review Workspace for the Journal of Digital History" at the conference AI through History, History through AI, C2DH, University of Luxembourg, 15-16 June 2026. The work was submitted to the conference on 26 February 2026. We also report a first editor-annotation analysis of 80 saved decisions for the Claude-Qwen audit configuration. For Claude-Qwen, strict editor-confirmed accuracy is 70.0%, the correct-or-mostly-correct rate is 86.2%. The useful-output rate includes all responses judged correct, mostly correct, or partially correct, since these outputs can still assist with editorial review even if the model's assessment is not fully accurate. This rate was 90.0%. The full version may include additional experiments, extended evaluation, and a more complete release of materials.