PoQ-Judge: Decentralized Quality Evaluation

• PoQ-Judge is a decentralized framework that employs incentivized evaluator nodes and distributed statistical adjudication to assess generative AI quality.
• It leverages lightweight, reference-free quality functions and robust consensus methods (e.g., median, trimmed mean) to ensure reliable model output verification.
• The protocol integrates cost-aware reward mechanisms and extends to quantum verification, enabling secure and scalable decentralized AI inference networks.

Proof-of-Quality Judge (PoQ-Judge) is a decentralized, reference-free framework for evaluating, verifying, and incentivizing the quality of generative model outputs—especially LLMs—in settings where traditional cryptographic computation proofs are infeasible or inefficient. The PoQ-Judge paradigm replaces direct validation of inference, as in ZKML or OPML frameworks, with distributed statistical adjudication over output quality by incentivized evaluator nodes (“Judges”). It is foundational to scalable, economically robust decentralized LLM inference networks and is being actively extended for quantum device verification in the memory-bounded regime.

1. Formal Foundations and Protocol Structure

PoQ-Judge prescribes that, for each user query $q\in\mathcal Q$, a single inference node $F$ computes an output $r=F(q)$, which is then assessed by a committee of Judges $J=\{J_1,…,J_k\}$. Each Judge independently applies a certified, lightweight quality function $M(q,r)$ (for NLP, typically a cross-encoder or dedicated judge model) to obtain a score $s_i\in[L, U]$, e.g., $[0,10]$ or $[–1,1]$ mapped to $[0,10]$.

The PoQ-Judge protocol consists of:

• Phase 1 (Commit): Each Judge computes $s_i = M(q, r)$, encrypts $F$0 using a local ephemeral key, and publishes the ciphertext.
• Phase 2 (Reveal): After all encrypted scores are posted, public keys are released for decryption, ensuring that no Judge can copy the scores of others or engage in “lazy” behavior.
• Consensus: Aggregate the scores to obtain the consensus value $F$1.
• Decision: If $F$2 (with $F$3 a deployer-selected threshold), the output is accepted and rewards are distributed; otherwise, the output is rejected and no reward is granted (Zhang et al., 2024).

This protocol is generic: judges can be run on- or off-chain; the selection of $F$4 and $F$5 can be deterministic, randomized, or energy-balance-based; and the framework accommodates both cross-encoder and reference-free evaluation models (Tian et al., 20 Apr 2026, Tian et al., 18 Dec 2025).

2. Quality Functions, Model Architectures, and Reference-Free Evaluation

PoQ-Judge operationalizes “quality” through specific judge models $F$6 which predict the semantic or task fidelity of model outputs using learned neural architectures without requiring ground-truth references. Three canonical architectures are employed:

• TextCNN Judge: A shallow convolutional model with 10M parameters, employing multiple kernel sizes and max-over-time pooling. Offers $F$71 ms GPU latency, sub-ms CPU latency, and robust alignment with QA ground-truth proxies (Pearson $F$8).
• MiniLM Cross-Encoder Judge: A distilled Transformer with 22M parameters; leverages full self-attention over $F$9 input. Achieves 13 ms latency and a Pearson $r=F(q)$0.
• DeBERTa Cross-Encoder Judge: 184M parameters, based on DeBERTa-v3-base; delivers the strongest performance with $r=F(q)$1 on held-out test data (Tian et al., 20 Apr 2026).

Training is pipeline-based: pre-training on GPT-4-labeled UltraFeedback, fine-tuning on in-domain GPT-4o-mini-labeled QA and summarization data (Tian et al., 20 Apr 2026). This enables direct reference-free scoring, outperforming reference-based metrics and closing the “deployment gap” for decentralized protocols.

3. Consensus Rules, Reward Mechanisms, and Incentive Engineering

PoQ-Judge employs various consensus functions for aggregation:

• Simple mean, median, trimmed mean (trim ratio $r=F(q)$2), and adaptive trust-weighted mean, where each evaluator’s score is weighted based on historical consistency (Tian et al., 29 Jan 2026).
• Adversary-resilient consensus rules (median/trimmed/weighted mean) maximally align consensus with ground-truth in the presence of noisy or malicious evaluators.

Inference rewards are designed to maximize quality per cost: $r=F(q)$3 where $r=F(q)$4 is the consensus output quality, $r=F(q)$5 is normalized latency or cost, $r=F(q)$6, $r=F(q)$7 configure the quality/cost trade-off (Tian et al., 18 Dec 2025). Evaluator (Judge) rewards depend on consistency: $r=F(q)$8 where closeness is $r=F(q)$9 and $J=\{J_1,…,J_k\}$0 is the cost of evaluation.

In the original PoQ-Judge protocol (Zhang et al., 2024), Judge rewards use a softmax over $J=\{J_1,…,J_k\}$1 for outlier-penalization; theoretical results establish monotonicity and incentive compatibility: rational inference nodes are incentivized toward maximal quality; lazy or guessing judges are economically disfavored.

4. Robustness, Adversarial Models, and Security Guarantees

PoQ-Judge explicitly models adversaries controlling up to a fraction $J=\{J_1,…,J_k\}$2 of judge slots, with perturbation strategies including random noise, boosting (reward inflation), sabotage (reward suppression), and strategic intermittent manipulation (Tian et al., 29 Jan 2026). The consensus mechanism’s robust aggregation (median, trimmed mean) dramatically reduces the impact of such attacks; e.g., under sabotage ($J=\{J_1,…,J_k\}$3), mean-based consensus sees a $J=\{J_1,…,J_k\}$4 reward drop, but median/trim operators halve this. Trust-weight adaptation, using $J=\{J_1,…,J_k\}$5, further shrinks manipulator impact by downweighting chronic deviators.

Evaluators’ sample size $J=\{J_1,…,J_k\}$6 is a key parameter: increasing $J=\{J_1,…,J_k\}$7 increases attack tolerance at the cost of lower per-judge rewards and higher variance. Practical deployment guidance is to use $J=\{J_1,…,J_k\}$8 in moderate-risk, open-participation environments (Tian et al., 29 Jan 2026).

5. Cost-Aware Design and Quality–Cost Trade-Offs

PoQ-Judge integrates explicit, latency-normalized costs for both inference and evaluation, reflected in the reward equations above (Tian et al., 18 Dec 2025). Cost normalization ensures that higher-quality and/or lower-cost nodes are consistently favored in reward allocation. Evaluator model selection is critical for maximizing overall system efficiency: in empirical studies, STS-DistilRoBERTa bi-encoders outperform cross-encoders both in correlation with ground truth ($J=\{J_1,…,J_k\}$9) and in speed, enabling high-throughput batch evaluation.

Cost-aware incentives dominate reward allocation. For instance, Llama-3.2-3B and Gemma-2-2B, leading in both F1 and latency, receive consistently higher average rewards. The reward function can be reparameterized as $M(q,r)$0 to interpolate between pure quality and pure efficiency objectives (Tian et al., 18 Dec 2025).

6. Extensions: Quantum PoQ-Judge and Memory-Bounded Verification

In the quantum setting, PoQ-Judge refers to a classical–quantum interactive protocol for unconditionally verifying “quantumness” of devices under memory bounds (Malavolta et al., 29 May 2025). Here, the classical Judge interacts with a quantum prover, accepting only if the device produces outcomes unattainable by classical protocols within a bounded memory model.

Two protocols exist:

• Quadratic memory-gap PoQ: Achieves soundness against any adversary with $M(q,r)$1 bits memory versus an honest prover using $M(q,r)$2 qubits. Relying on parity learning lower bounds [Raz18], the protocol is efficiently verifiable by a fully classical Judge.
• Exponential memory-gap PoQ: Via streaming and interactive hashing, honest memory is polylogarithmic in adversarial memory; soundness holds for classical or quantum adversaries with $M(q,r)$3 bits/qubits.

Both protocols are efficiently verifiable, but the quadratic-gap protocol is more practical in near-term quantum hardware due to low quantum memory requirements and gate complexity (Malavolta et al., 29 May 2025).

7. Practical Considerations, Limitations, and Deployment Guidance

PoQ-Judge deployments should:

• Prioritize bi-encoder evaluators (e.g., STS-DistilRoBERTa) except where other signals are needed for diversity or NLI coverage.
• Select robust aggregation rules (median or trimmed mean with $M(q,r)$4) for adversarial resilience.
• Use explicit on-chain/off-chain reporting and normalized latency for cost accounting, with random audits to deter misreporting.
• Consider integrating trust-weighting and online calibration (using infrequent high-fidelity anchors) to automatically tune dimension weights and suppress unreliable evaluators (Tian et al., 18 Dec 2025, Tian et al., 20 Apr 2026).
• Typically set $M(q,r)$5 evaluators per round for robustness/economy trade-off.

The major limitation is the dependence of judge accuracy on the quality of the ground-truth proxy (e.g., token-F1 for summarization is weak). Current research focuses on improving these proxies and on robustifying lightweight judge architectures for broader generalization in open, heterogeneous networks (Tian et al., 20 Apr 2026).

---

References:

• “Proof of Quality: A Costless Paradigm for Trustless Generative AI Model Inference on Blockchains” (Zhang et al., 2024)
• “PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference” (Tian et al., 20 Apr 2026)
• “Adaptive and Robust Cost-Aware Proof of Quality for Decentralized LLM Inference Networks” (Tian et al., 29 Jan 2026)
• “How to Verify that a Small Device is Quantum, Unconditionally” (Malavolta et al., 29 May 2025)
• “Design and Evaluation of Cost-Aware PoQ for Decentralized LLM Inference” (Tian et al., 18 Dec 2025)