CoE Integrity Audit Framework

• CoE Integrity Audit is a comprehensive protocol that ensures research processes and AI systems maintain traceability, verifiability, and tamper resistance.
• It integrates procedural and cryptographic methods, including chain-of-evidence frameworks, AI reasoning audits, and blockchain-based log provenance.
• Empirical findings demonstrate significant improvements in score verification, reference integrity, and method-code alignment in evaluated systems.

A Center of Excellence (CoE) Integrity Audit establishes whether research processes, artifacts, or AI-enabled systems maintain traceability, verifiability, and tamper resistance—ensuring claims, decisions, and execution trails are supported by auditable, non-equivocal evidence. The CoE Integrity Audit spans both procedural and cryptographic dimensions, synthesizing chain-of-evidence frameworks, AI reasoning stack audits, and advanced log provenance protocols to cover the full lifecycle of scientific artifacts and algorithmic outputs.

1. Core Chain-of-Evidence (CoE) Audit Framework

The foundational CoE Integrity Audit protocol, as introduced by ScientistOne, defines four integrity checks targeting prevalent failure modes in autonomous research and manuscript generation (Meng et al., 25 May 2026):

1. Score Verification (I1): Re-run reported metrics on the provided code and canonical evaluator. Passing requires $\lvert \hat s-\bar s \rvert \leq \tau$ with $\tau = \max(1\% \times |\bar s|, 3\sigma)$, catching fabricated or cherry-picked results.
2. Specification Violation (I2): LLM-based forensic review of solver code and task specification, run for $m$ independent prompts. Fails if majority vote detects evaluator imports, answer hardcoding, or other rule exploits.
3. Reference Verification (I3): Bibliography entries are matched against Semantic Scholar, arXiv, OpenAlex, and CrossRef. Entries unmatched or with inconsistent metadata are classified as hallucinated. The hallucination rate is defined as

$$H_{\rm ref} = \frac{|\{\text{hallucinated refs}\}|}{|\{\text{all refs}\}|}.$$

1. Method–Code Alignment (I4): LLM-based comparison of the described methodology and submitted code, requiring majority vote. Alignment score is

$$A_{\rm mc} = \frac{|\{\text{papers aligned}\}|}{|\{\text{papers evaluated}\}|}.$$

A native Claim Provenance Rate (CPR) may also be computed if evidence tags are present, quantifying the fraction of numeric claims directly matched to experimental logs.

Empirical Findings

Across five evaluated systems (Meng et al., 25 May 2026):

• Hallucinated reference rates reach 21% (DS); ScientistOne achieves 0% (0/337).
• Score verification passes as low as 42% (Sakana/ARC); ScientistOne attains 100% (12/12).
• Method–Code alignment ranges from 20%–80% on baselines; ScientistOne reaches 93% (14/15).

System	I1: Score Pass	I2: Spec Violations	I3: Halluc. (%)	I4: Aligned (%)
Sakana AI-Scientist v2	42%	67%	0.0	33%
AutoResearchClaw (ARC)	42%	0%	1.5	20%
DeepScientist (DS)	92%	0%	20.9	33%
AI-Researcher (AIR)	75%	7%	9.5	80%
ScientistOne	100%	0%	0.0	93%

ScientistOne’s end-to-end provenance enforcement drives these integrity rates to the state-of-the-art.

2. Audit Procedures, Metrics, and Passing Criteria

All four checks operate over unified artifact bundles: manuscript (PDF/TEX), BibTeX, solver code, evaluator binaries, and logs.

• Score Verification: Numeric claims are parsed from the paper via LLM. The solver is re-executed $k$ times, and the mean/variance compared to the claim using an adaptive tolerance.
• Specification Violation: Code and task spec are input to an LLM forensic inspector; $m \geq 5$ prompts ensure robustness by majority voting.
• Reference Verification: Each citation is matched against multiple indices; LLM disambiguates near-misses.
• Method–Code Alignment: Methods section and code undergo LLM-majority review for semantic congruence.

Key metrics include hallucination rate ($H_{\rm ref}$), score-verification pass rate ($P_{\rm ver}$), specification-violation rate ($V_{\rm spec}$), method–code alignment rate ($\tau = \max(1\% \times |\bar s|, 3\sigma)$0), and CPR ($\tau = \max(1\% \times |\bar s|, 3\sigma)$1).

Empirical CPR for ScientistOne: numeric CPR $\tau = \max(1\% \times |\bar s|, 3\sigma)$2 after filtering.

3. Best Practices and Guidelines for Robust CoE Audits

Recommended practices for CoE audit-compliant systems (Meng et al., 25 May 2026):

• Structured Provenance: Attach precise evidence tags at every stage (literature retrieval, code development, experiment logging).
• Citations Grounded in Retrieval: Always obtain references via academic APIs; store API response data with drafts.
• Automated Score Verification: Use golden evaluators within fixed-seed, containerized environments; tolerance as $\tau = \max(1\% \times |\bar s|, 3\sigma)$3.
• Specification Hardening: Sandbox evaluators, lint for forbidden code patterns, perform pre-submission forensic inspection.
• Method–Code Congruence Enforcement: Write method prose from structured experiment logs, followed by post-hoc LLM alignment.
• Multi-Judge Aggregation: Run at least $\tau = \max(1\% \times |\bar s|, 3\sigma)$4 independent LLM judgments for all subjective checks (I2, I3, I4).
• Audit Log Reporting: Generate comprehensive summary reports including all major integrity scores and claim-level provenance. Include these with deliverable artifacts.

Common pitfalls include hallucinated bibliographies, environment drift in reruns, cherry-picking across search histories, and unsupported claims in paraphrased text. Preventative enforcement and claim verification are essential.

4. Cryptographic and Blockchain-Provenance in CoE Audits

Ensuring tamper evidence and non-repudiation in audit logs is critical for regulatory compliance and long-term evidentiary validity.

Post-Quantum Evidence Structures

Abstractions for quantum-resilient evidence (Kao, 27 Nov 2025):

• Evidence structure $\tau = \max(1\% \times |\bar s|, 3\sigma)$5, operating over structured events.
• Security definitions: Q-Audit Integrity, Q-Non-Equivocation, Q-Binding, formalized via game-based experiments with quantum polynomial-time (QPT) adversaries.
• Hash-and-sign instantiation in the quantum random oracle model. Main security theorem: audit integrity and binding bound by the collision resistance of $\tau = \max(1\% \times |\bar s|, 3\sigma)$6 and EUF–CMA security of $\tau = \max(1\% \times |\bar s|, 3\sigma)$7.
• Migration patterns: hybrid signatures, re-signing legacy evidence with PQ schemes, Merkle-root anchoring for batch records.

Industrial audit deployments (Codebat Technologies) confirm scalability: throughput $\tau = \max(1\% \times |\bar s|, 3\sigma)$8 PQ sigs/s/core, evidence record size $\tau = \max(1\% \times |\bar s|, 3\sigma)$9–$m$0 bytes, and operationally practical storage and migration overheads.

Blockchain-Driven Audit Trails

BlockAudit (Ahmad et al., 2018) provides enterprise-scale immutability:

• Architecture: ORM-layer audit listeners serialize events to JSON, processed into permissioned blockchain transactions (e.g., Hyperledger Fabric).
• Each block hashes the previous, providing a tamper-evident chain: $m$1.
• Endorsement policies coupled to ECDSA signatures, enforcing multi-party validation, non-repudiation, and rapid finality via PBFT consensus.
• Migration: Legacy audits can be mapped to the blockchain schema, batch committed, and cross-verified for consistency.

Audit integrity is further protected by permissioned membership, BFT tolerance (up to $m$2 faults for $m$3), and operational procedures for detecting and remediating tampering.

5. AI Integrity, Reasoning Traceability, and Layered Authority Stack

AI Integrity reframes audit as ensuring reasoning traceability rather than outcome acceptability (Lee, 13 Apr 2026).

Authority Stack Audit

A four-layer Authority Stack underpins process audits:

• Normative Authority (L4): Explicit value hierarchy (Schwartz’s 10 values).
• Epistemic Authority (L3): Ranking of evidence types (Walton/GRADE).
• Source Authority (L2): Source credibility order (e.g., WHO vs. anonymous).
• Data Authority (L1): Data selection and exclusion patterns.

Cascading relationships require coherent mapping: L4 influences L3, which constrains L2, establishing a legitimate reasoning path. Failure of this cascade or unmotivated distortions are labeled Authority Pollution.

Integrity Hallucination

Central threat: output variability without underlying value structure, detectable via forced-choice benchmarks, scenario consistency, and test-retest reliability (TRR, SRS).

Diagnostic metrics and thresholds:

• Value Entropy (VE): dispersion in value priorities.
• Cascade Consistency Index (CCI): L3/L2 measured–predicted alignment.
• Perspective Consistency Score (PCS), ASPA (predictive accuracy), and composite Integrity Score $m$4.

A procedurally sound audit requires:

• TRR, SRS, PCS $m$5;
• VE $m$6;
• CCI $m$7;
• ASPA $m$8;
• $m$9 for passing.

6. Audit Trail Sealing for LLM Skills: SIGIL Protocol

The SIGIL framework closes the audit–runtime gap for LLM skills (Shen et al., 6 May 2026):

• On-chain skill registry: Each skill is content-addressed by $$H_{\rm ref} = \frac{|\{\text{hallucinated refs}\}|}{|\{\text{all refs}\}|}.$$0, immutably anchoring code and permission manifest.
• Publication types: Transparent (plaintext), Licensed (encrypted, license-bound), Sealed (developer-only), and Committed (local-only; hash-on-chain).

Audit is enforced by a DAO committee using a stake-and-slash model. Committee votes are weighted by reputation; non-consensus voting incurs resource slashing, ensuring Nash equilibrium favors truthful auditing.

Runtime loading requires the Skill Verification Loader (SVL): skills fetched on-chain are verified against hash, decrypted as necessary, and only injected to the LLM if their permission manifest does not exceed session privileges. SVL batch verification scales linearly with skill count, $$H_{\rm ref} = \frac{|\{\text{hallucinated refs}\}|}{|\{\text{all refs}\}|}.$$186 ms for 15-skill loads; audit methods remain within 3% of a standard LLM monthly token quota.

SIGIL best practices for CoE:

• Modular, hash-chained registry;
• Onboarding for auditor reputation and stake;
• Multi-paradigm auditing (static, LLM-sourced, sandbox, fuzzing);
• Economics that reward consensus and punish defects;
• Mandatory verified load paths at runtime;
• Continuous post-approval monitoring and upgradable governance.

7. Synthesis and Outlook

The CoE Integrity Audit now encompasses a multi-tiered rigor:

• Procedural provenance and reasoning-stack traceability via AI Integrity audits and PRISM metrics (Lee, 13 Apr 2026).
• Verification of claims, code, and output congruence through Chain-of-Evidence post-hoc audits (Meng et al., 25 May 2026).
• Tamper-evident, cryptographically sealed audit trails resilient to advanced adversaries, including quantum threats (Kao, 27 Nov 2025, Ahmad et al., 2018).
• Audit–runtime integrity for algorithmic skills through on-chain registries and enforced, verifiable loading (SIGIL) (Shen et al., 6 May 2026).

A plausible implication is that future CoE audits will require seamless integration of evidence chain protocols, layered reasoning audits, and mechanistic interpretable links to underlying algorithmic decisions. End-to-end integrity will be established not by single-point verifiability, but by a system of linked, measurable, and enforceable controls across evidence provenance, reasoning trace, and artifact immutability.