EyePCR: Benchmark for Ophthalmic Surgery Analysis

• EyePCR is a comprehensive benchmark that evaluates ophthalmic surgery analysis by testing multimodal LLMs on perception, comprehension, and reasoning.
• It combines structured annotations from over 210k VQA pairs, 1,048 fine-grained attributes, and a semantic graph of 25k+ triplets to simulate surgical cognition.
• EyePCR-MLLM, a domain-adapted variant, achieved superior accuracy over open-source and commercial systems in discerning subtle ophthalmic surgical cues.

EyePCR is a comprehensive benchmark for ophthalmic surgery analysis designed to evaluate multimodal LLMs on surgeon-like cognitive competencies in ophthalmic surgery. It addresses a gap left by surgical video datasets that largely test low-level perception and shallow VQAs, and instead formalizes a three-stage PCR hierarchy—Perception $\rightarrow$ Comprehension $\rightarrow$ Reasoning—grounded in real videos and structured surgical knowledge. The benchmark contains more than 210k VQAs, 1,048 fine-grained attributes across seven semantic views, a semantic memory graph with 25,567 triplets, and four clinically grounded reasoning tasks; it also introduces EyePCR-MLLM, a domain-adapted variant of Qwen2.5-VL-7B (Wang et al., 19 Sep 2025).

1. Conceptual framework

EyePCR is organized around a staged account of surgical cognition. Perception evaluates whether a model can extract rich, fine-grained, multi-view attributes from ophthalmic surgical video segments. Comprehension maps these attributes and aligned subtitles into structured scene graphs and a temporal semantic memory graph, enabling knowledge-grounded understanding. Reasoning tests four clinically grounded dimensions: procedural flow understanding, surgical intent inference, intraoperative decision simulation, and anomaly/risk awareness assessment (Wang et al., 19 Sep 2025).

This organization is clinically motivated. The benchmark is intended to simulate how surgeons perceive visual cues and combine them with domain knowledge to make decisions, rather than treating surgical video understanding as only tool recognition or phase labeling. A central design choice is that all tasks are grounded in structured clinical semantics, so that evaluation remains interpretable and clinically meaningful.

A frequent misconception is that “multi-view” refers to multi-camera feeds. In EyePCR, “multi-view” refers to semantic views, not multi-camera feeds. The seven views are Ocular Anatomy, Pathological Diagnosis, Surgical Operation, Device & Instrumentation, Intraoperative Anatomy, Medicine, and Monitoring. Across these views, the attribute hierarchy is annotated at three levels: L1 with 76 coarse domains, L2 with 240 subtypes, and L3 with 732 fine-grained details, totaling 1,048 attributes.

2. Corpus, annotation design, and benchmark scope

EyePCR is large-scale at the level of videos, temporally coherent segments, VQAs, and structured knowledge. The dataset composition reported for the benchmark is summarized below (Wang et al., 19 Sep 2025).

Component	Reported scope	Notes
Videos	2,968 collected; 1,544 retained	High-quality full-length ophthalmic surgery videos after expert screening
Segments	82,893	Temporally segmented and annotated
VQAs	213,883	64,425 MCQs and 149,458 OEQs
Attributes	1,048	L1: 76, L2: 240, L3: 732
Knowledge graph	25,567 triplets	16,034 entities and 3,507 relations

The videos were collected from OphNet, LensID, Cataract-1k, curated clinical repositories from Wenzhou Medical University Eye Hospital and Wills Eye Hospital, Eyetube, and AAO. Personally identifiable information was removed, and only publicly sourced videos were used. Videos were screened for completeness and clarity; segments preserve temporal coherence and are aligned with subtitles. Spatial resolution was kept high at $\ge 720\text{p}$. Frame rate and sampling specifics are not reported.

The dataset description reports 213,883 VQA pairs overall, comprising 64,425 multiple-choice questions and 149,458 open-ended questions. It further itemizes 102,827 Perception VQAs, reported as 64,426 MCQs and 38,401 OEQs across seven semantic views and three levels; 24,693 Comprehension OEQs grounded in scene-graph triplets; and 86,363 Reasoning OEQs spanning the four dimensions. A patient-level split is used to prevent identity leakage, yielding 77,133 segments for training and 5,760 for testing. The paper does not report a separate validation split.

The annotation schema aggregates attributes across all seven views for each segment, producing a multifaceted perceptual representation that is later used to construct segment-level scene graphs. Attributes exhibit long-tail distributions typical of surgical scenes. Inter-annotator agreement is not reported.

3. Knowledge structures and task construction

A defining feature of EyePCR is that comprehension and reasoning are explicitly grounded in graph structure rather than only in free-text supervision. At the segment level, EyePCR uses Structured Scene Graphs that encode entity–action–target events such as $$[\text{Forceps}] \rightarrow \text{grasp} \rightarrow [\text{Capsular Flap}]$$. Segment graphs are then temporally aggregated into a Semantic Memory Graph that preserves procedural context and cross-step dependencies, enabling multi-hop reasoning (Wang et al., 19 Sep 2025).

The semantic memory graph is formalized as

$G = \{ (h_i, r_i, t_i) \}_{i=1}^{M},$

where $h_i \in E$ are entities, $r_i \in R$ are relations or actions, $t_i \in E$, and $M = 25{,}567$. Entity types include anatomy, pathology, device/tool, medication, and procedural step. Relations represent actions and interactions such as injects into, grasps, incises, sutures, positions, removes, and relates_to. The graph contains 25,567 triplets, 16,034 entities, and 3,507 relations; 11,462 logical paths are extracted, including 9,499 two-hop paths and 1,963 paths of at least three hops.

These structures support question generation and constrain plausible answers. The benchmark states that subgraphs

$$P = (v_1 \rightarrow v_2 \rightarrow \dots \rightarrow v_k)$$

encode clinically meaningful sequences such as incision $\rightarrow$0 viscoelastic injection $\rightarrow$1 capsulorhexis and are used to ground question templates, constrain plausible answers, and mitigate hallucinations. Comprehension queries extract and verbalize triplets or path-consistent statements, while evaluation compares generated text to ground-truth OEQs via lexical metrics and rubric scoring.

Representative templates make the design concrete. A Perception MCQ asks, “Which ocular structure is being manipulated in this segment?” with options {Cornea, Conjunctiva, Sclera, Iris}. A Perception OEQ asks, “Describe the baseline anatomical condition of the cornea prior to any surgical intervention.” A Comprehension OEQ asks, “Describe the path of the anchoring suture,” with an expected answer grounded by $\rightarrow$2 and $\rightarrow$3. Reasoning OEQs include prompts such as “Which surgical phase does this action belong to?”, “Why did the surgeon choose to perform an iridectomy?”, “What would happen if the perfusion is skipped?”, and “What potential complications may arise from this action involving the iris?”

4. EyePCR-MLLM and evaluation methodology

EyePCR-MLLM is a domain-adapted variant of Qwen2.5-VL-7B trained on EyePCR to improve surgical perception and reasoning. The reported setup uses a Qwen2.5-VL-7B multimodal encoder–decoder adapted to video clips and paired text. Fine-tuning is instruction-style and uses both MCQs and OEQs, leveraging structured annotations and graph-grounded questions. At inference time, the paper reports standard multimodal prompting; no explicit chain-of-thought or knowledge-graph retrieval is reported at inference time, and the graphs primarily ground dataset construction (Wang et al., 19 Sep 2025).

A natural multi-task formulation for training on EyePCR is stated as

$\rightarrow$4

with $\rightarrow$5 given as cross-entropy over options,

$\rightarrow$6

The paper presents this as useful for reproduction and future work, while also noting that it does not detail the full training pipeline.

The evaluation protocol combines categorical and generative metrics. For MCQs, accuracy is

$\rightarrow$7

For OEQs, lexical metrics include BLEU-1–4 and ROUGE-1/2/L, with BLEU-4 and ROUGE-L reported per stage and dimension. In addition, a GPT-4.1-based 5-point Likert rubric assesses factual correctness, specificity, and completeness over the entire OEQ test set.

Human evaluation is integrated into the benchmark. A total of 66 ophthalmology professionals—47 trainees and 19 surgeons—answered a balanced set of MCQs and OEQs spanning the PCR hierarchy. Compared models include GPT-4.1, GPT-o3, Claude 3.7 Sonnet, Gemini 2.0 and 2.5 in Flash and Pro variants, and open-source systems including Qwen2.5-VL-7B, Qwen2.5-VL-32B, InternVL-14B, LLaVA-Video-7B, and VideoLLaMA3-7B. Prompts, input resolution, hyperparameters, and chain-of-thought or retrieval specifics are not detailed.

5. Empirical results and cognitive implications

On Perception MCQs, EyePCR-MLLM achieves the highest accuracy among the compared models. The human reference accuracies are 0.9090 for an ophthalmic surgeon and 0.6567 for an ophthalmology trainee. EyePCR-MLLM reaches an average accuracy of 0.7412, outperforming open-source baselines including Qwen2.5-VL-7B at 0.3866, Qwen2.5-VL-32B at 0.4036, and InternVL-14B at 0.4398, and also exceeding GPT-4.1 at 0.7069 and Gemini 2.5 Pro at 0.7138. Per-view accuracies for EyePCR-MLLM are reported as 0.8991 on V1 Ocular Anatomy, 0.8768 on V2 Pathology, 0.6872 on V3 Operation, 0.6776 on V4 Device/Instrumentation, 0.8339 on V5 Intraoperative Anatomy, 0.5452 on V6 Medicine, and 0.6683 on V7 Monitoring (Wang et al., 19 Sep 2025).

For Comprehension and Reasoning OEQs, commercial models remain strongest on some metrics. Gemini 2.5 Pro is reported as strongest on lexical overlap with average BLEU-4 of 0.0731 and ROUGE-L of 0.3293, while GPT-o3 has the highest rubric score at 3.935. EyePCR-MLLM nevertheless consistently leads open-source models on OEQs, with average BLEU-4 of 0.0678, ROUGE-L of 0.2784, and rubric score 3.006, narrowing the gap with commercial systems. Human experts show an inverse pattern: an ophthalmic surgeon has BLEU-4 of 0.0116 and ROUGE-L of 0.1454 but the highest rubric score, 4.158.

That disparity is central to the benchmark’s interpretation. EyePCR explicitly shows that lexical overlap can under-score clinically adequate answers: human experts have low lexical overlap yet the best rubric scores. This suggests that rubric scoring better captures clinical adequacy than n-gram metrics in this setting. A related finding is that models that rely on surface recognition often miss intent and phase dependencies even when they can identify tools or structures.

The benchmark’s cognitive analysis emphasizes several recurring failure modes. Models are challenged by microanatomy and subtle dynamics such as anterior chamber depth and iris stability. Commercial and open-source MLLMs lag in V6 Medicine and V7 Monitoring, where reasoning ties pharmacology and postoperative checks to ongoing anatomical states. Models also falter under ambiguous visual contexts, such as distinguishing partial iris prolapse from pupil capture, unless trained on fine-grained attributes. The large gains of EyePCR-MLLM over its base model are presented as evidence of the importance of domain adaptation.

6. Reliability, comparative position, and related uses of the name

EyePCR is intended as a research evaluation resource rather than a clinical decision system. The benchmark states that there remains a sizable gap between EyePCR-MLLM and ophthalmic surgeons, especially on subtle cues and complex reasoning. Annotation quality is supported by multi-level, multi-view labels and graph triplets curated from videos and subtitles, but inter-annotator agreement is not reported. The long-tail distribution of attributes may induce bias toward common patterns, and EyePCR’s scale helps but does not eliminate the issue. The benchmark also notes that outputs should not be used for autonomous clinical decision-making (Wang et al., 19 Sep 2025).

Several methodological limitations are explicit. Lexical metrics under-score expert answers; rubric scoring addresses semantics but depends on LLM graders. Frame-rate and synchronization details are not reported. The benchmark emphasizes ophthalmology, so generalization to other surgical domains would require careful adaptation. Future directions named in the paper include explicit knowledge-graph retrieval at inference, temporal transformers for long-range dependencies, robust semantic evaluation, and broader clinical settings.

Within the surgical-video literature cited in the benchmark, EyePCR is presented as singular in combining fine-grained multi-view attributes beyond phases and tools, knowledge graph integration with 25k+ triplets and multi-hop paths, clinically grounded reasoning tasks across four dimensions, and large-scale ophthalmic VQAs tied to structured semantics. The comparative context names datasets including Cholecseg8k, SurVLP, OphNet, Cataract-1k, AVOS, MedVidQA, TEMSET, GenSurg+, HecVL, OphVL, GP-VLS, SSG-VQA, and Surg-QA. Availability details remain limited: the paper does not specify release details, licensing, or a project page, but states that the dataset includes segment annotations, VQA pairs, and knowledge graph structures and directs access and reproducibility requests to the corresponding authors.

The name “EyePCR” also appears in two distinct contexts elsewhere in the supplied literature. In the WB/PCR image-forensics paper “Forgery Blind Inspection for Detecting Manipulations of Gel Electrophoresis Images,” practical guidance is described for “EyePCR-style pipelines” built around the FBI$\rightarrow$8 method for integrity analysis of WB and PCR images (Shao et al., 2020). In “DNA-to-go: A portable smartphone-enabled PCR assay platform,” the phrase “The ‘EyePCR’ concept is naturally aligned with this platform” is used for a smartphone-enabled, portable PCR system and its possible ocular adaptations, including tear fluid or conjunctival swabs (Priye et al., 2016). In those papers, the term functions as a workflow label or conceptual framing, whereas in the ophthalmic-surgery literature it is the formal name of the benchmark itself.