PerceptionRubrics Evaluation Framework

• PerceptionRubrics is a rubric-based multimodal evaluation framework that replaces holistic matching with instance-specific atomic auditing.
• It employs a dual-stream rubric architecture with ‘Must-Right’ and ‘Easy-Wrong’ criteria to enforce mandatory perceptual facts and diagnose common errors.
• Empirical results reveal a reliability gap in model perception, aligning evaluation closely with human judgment through non-compensatory gating.

PerceptionRubrics is a rubric-based multimodal evaluation framework that calibrates image-caption assessment to human perception by replacing holistic semantic matching with instance-specific atomic auditing. It was introduced to expose the gap between saturated benchmark scores and real-world brittleness in multimodal LLMs (MLLMs): models can score highly on forgiving similarity metrics while still missing digits, miscounting objects, flipping spatial relations, or hallucinating localized content that humans treat as reliability failures. The framework pairs 1,038 information-dense images with 10,718 atomic rubrics, organizes those rubrics into Must-Right and Easy-Wrong streams, and scores responses with a hard gate that zeroes out any sample missing mandatory visual facts (Wei et al., 26 Jun 2026).

1. Problem formulation and evaluative stance

PerceptionRubrics addresses a specific failure mode of contemporary multimodal benchmarking: high aggregate scores can coexist with brittle perceptual behavior. The motivating claim is not merely that models remain imperfect, but that many existing benchmarks are both saturated and forgiving. They often rely on information-poor images or closed-ended tasks, and they typically reward broad semantic overlap through linear averaging or global similarity metrics. Under those conditions, localized but fatal errors can be diluted away, even when those errors would invalidate a caption in practical use (Wei et al., 26 Jun 2026).

The framework therefore shifts evaluation from holistic semantic matching to atomic auditing. In PerceptionRubrics, a caption is not primarily judged by how similar it sounds to a reference caption; it is judged by whether it satisfies a set of small, verifiable, instance-specific factual checks derived from the image’s true content. This is a concrete instantiation of a broader rubric formalism in which rubrics are treated as explicit, structured, decomposable, and verifiable criteria sets for assessing model outputs (Chen et al., 7 Jun 2026).

This evaluative stance is closely aligned with other work that treats perception quality as multidimensional rather than reducible to final-answer accuracy. For example, “Perception in Reflection” operationalizes perceptual quality through criteria such as authenticity, correctness, detailness, coherence, and completeness, and uses them in both dataset construction and evaluation (Wei et al., 9 Apr 2025). PerceptionRubrics makes a narrower but stricter move: it centers visual factual fidelity, then enforces non-compensatory penalties when mandatory perceptual facts are missing (Wei et al., 26 Jun 2026).

2. Corpus construction and Circular Peer-Review consensus

The PerceptionRubrics corpus contains 1,038 information-dense images spanning seven domains: Natural Scenes, Document OCR, Digital UI / UX, Structured Data, STEM Expert, Logic Puzzle, and Creative Cultural (Wei et al., 26 Jun 2026). The images are intentionally dense and diverse so that models cannot rely on shallow priors.

Each image is paired with a single golden caption. These captions are unusually long and detailed, with an average length of 770.42 words, a median of 569 words, and a long tail up to 3,461 words (Wei et al., 26 Jun 2026). They are not authored directly by humans from scratch. Instead, the paper introduces a Circular Peer-Review consensus pipeline:

1. Three strong MLLMs generate independent captions.
2. They iteratively critique, rank, and rewrite one another’s outputs in a circular review process.
3. The process runs for at most $N \le 2$ iterations.
4. Samples without unanimous agreement are discarded.
5. Humans perform final lightweight verification rather than full authoring (Wei et al., 26 Jun 2026).

From these golden captions, the system distills 10,718 atomic rubrics: 4,053 Must-Right rubrics and 6,665 Easy-Wrong rubrics. The average density is 10.33 rubrics per image, comprising 3.90 Must-Right items and 6.42 Easy-Wrong items (Wei et al., 26 Jun 2026). The rubrics are explicitly instance-specific rather than task-generic.

Element	Value
Images	1,038
Domains	7
Golden captions	1,038
Atomic rubrics	10,718
Must-Right rubrics	4,053
Easy-Wrong rubrics	6,665
Average rubrics per image	10.33

This construction pipeline matters because it ties rubric generation to a high-fidelity textual anchor while avoiding full manual authoring. A plausible implication is that the framework treats caption production as an intermediate consensus mechanism and rubric distillation as the true evaluative endpoint.

3. Dual-stream rubric architecture and Gated Scoring

PerceptionRubrics uses a dual-stream rubric architecture. Must-Right rubrics encode essential facts that define the minimum acceptable perceptual understanding of an image. They are extracted a priori from the image and golden caption, focus on clearly visible essential elements, and are adapted by domain; for example, OCR-heavy cases emphasize textual precision, whereas natural scenes emphasize objects and relations (Wei et al., 26 Jun 2026).

Easy-Wrong rubrics encode fine-grained pitfalls that models frequently get wrong, including hallucinations, omissions, subtle misreadings, incorrect relations, OCR mistakes, and wrong counts. They are constructed a posteriori: the authors build a response pool from multiple baseline MLLMs, compare those outputs to the golden caption, and convert common error patterns into rubrics (Wei et al., 26 Jun 2026). The two streams therefore play different roles. Must-Right asks whether the model captured the core facts; Easy-Wrong asks whether it avoided realistic, model-typical failure modes.

The scoring mechanism is intentionally non-linear. Let the Must-Right set be

$\mathcal{R}_{m} = \{r_{m,1}, \dots, r_{m,j}\}.$

The gate status is

$$G = \prod_{i=1}^{j} \mathbb{I}(r_{m,i} = \text{True}),$$

so $G \in \{0,1\}$ and any failed Must-Right item sets the gate to zero. Let the Easy-Wrong set be

$\mathcal{R}_{e} = \{r_{e,1}, \dots, r_{e,k}\}.$

The final score is

$$S = G \cdot \frac{1}{k} \sum_{i=1}^{k} \mathbb{I}(r_{e,i} = \text{True}).$$

If the gate fails, the score is $0$; if the gate passes, the score becomes the average Easy-Wrong accuracy (Wei et al., 26 Jun 2026).

This design is the framework’s central technical claim. Mandatory perceptual facts are treated as hard prerequisites rather than dimensions that can be compensated for elsewhere. The paper also defines two derived quantities: Atomic Accuracy, the mean accuracy over individual rubrics, and Must-Right Pass Rate, the average of the binary gate $G$ (Wei et al., 26 Jun 2026).

Comparable essential-versus-additional separations now appear in several later rubric systems. rDPO constructs instance-specific checklist rubrics with essential and additional criteria for visual preference optimization (Yu et al., 14 Apr 2026), and $\text{RLR}^3$ uses instance-specific rubrics with Essential and Additional criteria plus hierarchical aggregation in online RL (Yu et al., 28 May 2026). PerceptionRubrics is distinguished by making the essential stream strictly conjunctive at evaluation time.

4. Empirical findings and diagnostic behavior

The evaluation covers 25 models across open-source and proprietary families, including Qwen2.5-VL, Qwen3-VL, Step3-VL-10B, Kimi-K2.5/K2.6, MiniMax-M3, MiMo-V2.5, Qwen3.5-397B, GPT-4o, GPT-5.4, GPT-5.5, Gemini-3-Pro/Flash/3.1-Pro/3.5-Flash, Seed-1.6/1.8/2.0-Pro/Lite, and GLM-5V-Turbo. The main judge is GPT-OSS-120B, and comparisons are made against DOCCI, DetailCaps, and human preference rankings from Vision Arena (Wei et al., 26 Jun 2026).

The paper’s first major result is the Reliability Gap. Models often perform well on Atomic Accuracy yet fail much more often on the conjunctive Must-Right gate. The reported examples are explicit:

• Qwen3.5-397B-A17B: Must-Right Item 93.64, Gate Pass 78.90, Easy-Wrong Item 78.01
• Seed-2.0-Lite: Must-Right Item 95.59, Gate Pass 82.85, Easy-Wrong Item 84.69
• GPT-4o: Must-Right Item 70.01, Gate Pass 32.27, Easy-Wrong Item 36.00 (Wei et al., 26 Jun 2026)

These numbers show that local factual correctness does not guarantee globally reliable perception under strict conjunction. The framework interprets this as evidence that models can recognize fragmented elements correctly while failing all-at-once perceptual fidelity.

The second major result is Open-Closed Stratification. The best open-source model, Qwen3.5-397B-A17B, reaches 61.61%, whereas proprietary frontier models reach roughly 68.79–70.07%, summarized as an approximately 8% perception deficit for open-source systems (Wei et al., 26 Jun 2026). This is notable because the paper argues that the open-source/proprietary gap has narrowed more on reasoning tasks than on perception-heavy ones.

The overall leaderboard remains discriminative in a regime where many prior benchmarks saturate. Seed-2.0-Lite is reported as the top overall model at 70.07%, Gemini-3.5-Flash is close behind at 69.88%, and GPT-4o-2024-05-13 performs surprisingly poorly at 12.59% (Wei et al., 26 Jun 2026). Domain breakdown is also sharply uneven: natural scenes are easiest, whereas GUI is hardest; for example, Qwen2.5-VL-7B scores 5.13% on GUI (Wei et al., 26 Jun 2026). This domain asymmetry is part of the paper’s claim that dense, structured interfaces remain a major unresolved perception challenge.

The third major result is Human-Aligned Rigor. On the five overlapping models with Vision Arena, PerceptionRubrics achieves Pearson correlation 0.916 and Spearman correlation 1.000 with human preference rankings (Wei et al., 26 Jun 2026). The paper attributes this to the metric’s nonlinearity: humans often treat certain perceptual failures as binary disqualifications, and the gating mechanism reproduces that structure more faithfully than linear similarity metrics.

The paper also reports several robustness analyses. Final score has negligible correlation with output length; for Gemini-3.1-Pro, $r=-0.079$ with $\mathcal{R}_{m} = \{r_{m,1}, \dots, r_{m,j}\}.$0, and for Kimi-K2.6, $\mathcal{R}_{m} = \{r_{m,1}, \dots, r_{m,j}\}.$1 with $\mathcal{R}_{m} = \{r_{m,1}, \dots, r_{m,j}\}.$2 (Wei et al., 26 Jun 2026). Scores using GPT-OSS-120B and GPT-5.5 are stable, ranking order remains identical, and GPT-OSS-120B is about 6.0% stricter (Wei et al., 26 Jun 2026). When rubric coverage is subsampled at 20%, 40%, 60%, and 80%, score variance decreases monotonically as coverage increases (Wei et al., 26 Jun 2026). Human error analysis highlights recurring failure modes such as material/boundary confusion, subtle spatial ambiguity, and hallucinations in blurry or shadowed regions (Wei et al., 26 Jun 2026).

5. Rubric lineage, competing methodologies, and common misconceptions

PerceptionRubrics belongs to a longer rubric tradition in which rubrics are used to articulate expectations, standardize judgment, and expose otherwise hidden reasoning criteria. Earlier educational work defined a rubric as “a scoring tool that articulates the expectations for a given task in describing levels of quality,” and decomposed it into task description, evaluation criteria, and awarding scores (Smith et al., 2016). Other physics-assessment work explicitly separated a mastery-style grading rubric for reliable scoring from a distinct difficulties rubric for diagnosing student reasoning, showing that scoring and diagnosis can be separated rather than forced into a single instrument (Doughty et al., 2014).

This lineage also reveals a recurring implementation problem: a rubric does not automatically change the evaluator’s philosophy. In a study of physics graduate teaching assistants, many TAs resisted reducing scores for missing explanation when the final answer was correct, inferred understanding from brief solutions, and did not substantially shift toward a cognitive-apprenticeship grading model even after a rubric-based professional-development intervention (Yerushalmi et al., 2017). A plausible implication for PerceptionRubrics is that strict rubric design and strict rubric execution are distinct problems.

Modern rubric research has generalized this logic beyond education. A recent survey characterizes rubrics as a unifying framework across evaluation, reinforcement learning, and safety alignment, and organizes them into evaluative, training, and intrinsic levels (Chen et al., 7 Jun 2026). Within that broader landscape, two controversies are especially relevant.

The first concerns rubric formation. RubricBench isolates rubric generation from rubric execution and shows a substantial gap between human-annotated and model-generated rubrics. In one highlighted result, OpenRubric + DeepSeek-v3.2 reaches 84.9% preference accuracy with human rubrics but 57.8% with self-generated rubrics, and the paper summarizes an average gap of about 27% absolute accuracy (Zhang et al., 2 Mar 2026). This directly cautions against assuming that model-authored rubrics are already aligned with human standards.

The second concerns rubrics versus pairwise comparative judgment. JudgmentBench compares rubric-based scoring and comparative judgment on the same legal outputs from the same experts. Comparative judgments recover intended quality ordering much better than rubrics, with mean Spearman rank correlation 0.908 versus 0.150, while requiring less than half the annotation time; median completion times are 1.92 minutes for comparative judgment and 4.74 minutes for rubrics (Yang et al., 24 May 2026). The same paper is explicit that rubrics remain valuable for diagnostic explanation, per-criterion auditing, and transparency (Yang et al., 24 May 2026). These findings do not negate PerceptionRubrics, but they delimit its strongest use case: decomposed, auditable perceptual fidelity rather than necessarily the best universal summary signal in every expertise domain.

6. From benchmark to training primitive

PerceptionRubrics is an evaluation framework, but related work shows that its underlying design principles—instance specificity, criterion decomposition, essential-versus-additional structure, and non-holistic scoring—are increasingly being reused as training-time supervision.

SibylSense treats rubric construction as an inference-time memory optimization problem for open-ended generation. A frozen rubric generator is steered by a tunable memory bank of validated rubric items, and adversarial probing is used to expose previously uncaptured quality dimensions as policies improve (Xu et al., 24 Feb 2026). In effect, discriminative rubrics become adaptive perceptual primitives rather than static checklists.

In multimodal preference optimization, rDPO uses instance-specific checklist rubrics with essential and additional criteria, scores each criterion with discrete credit $\mathcal{R}_{m} = \{r_{m,1}, \dots, r_{m,j}\}.$3, and constructs on-policy preference pairs only when rubric-grounded quality gaps are clear. On public downstream benchmarks, rubric-based filtering raises the macro average to 82.69, whereas outcome-based filtering drops it to 75.82 from 81.14 (Yu et al., 14 Apr 2026). In online RL, $\mathcal{R}_{m} = \{r_{m,1}, \dots, r_{m,j}\}.$4 extends RLVR from task-level verification to criterion-level verification, routing verifiable rubric items through deterministic tools and fuzzy criteria through an LLM judge, while using hierarchical aggregation to prioritize essential criteria. Across 15 benchmarks, its best model reaches a 4.7-point improvement over the base model (Yu et al., 28 May 2026).

Process-supervision variants continue the same pattern. AutoRubric-R1V constructs problem-specific reasoning rubrics by aggregating common checkpoints across successful multimodal trajectories, then combines rubric reward with answer reward during GRPO; its average score across six multimodal reasoning benchmarks is 54.81 versus 47.29 for the base Qwen2.5-VL-7B (Jia et al., 16 Oct 2025). RubricRL applies prompt-adaptive structured rubrics to text-to-image RL, generating a top-10 criterion set per prompt and averaging binary per-criterion judgments into the reward; on GenEval, RubricRL reaches 0.8468 overall for Phi3-3.8B and 0.6014 for Qwen2.5-0.5B, both best in their reported tables (Feng et al., 25 Nov 2025). Outside multimodal generation, question-specific rubrics for code evaluation also outperform question-agnostic ones, especially on logically diverse data structures and algorithms tasks (Pathak et al., 31 Mar 2025).

Taken together, these systems indicate that PerceptionRubrics is not an isolated benchmark artifact. It exemplifies a broader shift from scalar or holistic quality judgments toward explicit criterion-level supervision. The recurring lesson across this literature is consistent: when the target quality is multidimensional, localized, or partially verifiable, decomposed rubrics expose failures that aggregate metrics hide and provide training signals that coarse rewards cannot express.