---
title: 'CataractSurg-80K: Preoperative Planning Benchmark'
url: https://www.emergentmind.com/topics/cataractsurg-80k
type: topic
---

# CataractSurg-80K: Preoperative Planning Benchmark

to=arxiv_search  彩神争霸代理_search  天天中彩票一等奖 code: {"query":"CataractSurg-80K knowledge-driven benchmarking structured reasoning ophthalmic surgery planning", "max_results": 5, "sort_by": "relevance"}
to=arxiv_search ՞ւազ_search code: {"query":"cataract surgery planning benchmark structured clinical reasoning ophthalmic reports LLM", "max_results": 10, "sort_by": "relevance"}
CataractSurg-80K is a benchmark for **cataract surgery planning** rather than a surgical video-analysis dataset. It is presented as the **first large-scale benchmark for cataract surgery planning with structured clinical reasoning**, designed to model the preoperative workflow in which heterogeneous ophthalmic examinations are integrated into patient assessment, intraocular lens (IOL) selection, surgical risk evaluation, and recommendation generation. Its defining pipeline maps structured information derived from **OCT**, **Pentacam corneal topography**, **fundus reports**, and **master biometric reports** into a **diagnostic-style question**, an expert-style **Complex\_CoT** reasoning trace, and a **structured surgical recommendation** [2508.20014].

## 1. Scope and clinical task definition

CataractSurg-80K addresses **preoperative cataract surgery planning** as a structured reasoning problem. The benchmark is motivated by the observation that cataract planning is not a single-label prediction task: clinicians must reconcile multiple ophthalmic reports, interpret parameters such as **axial length (AL)**, **anterior chamber depth (ACD)**, **K1**, **K2**, **lens thickness (LT)**, and **central corneal thickness (CCT)**, and convert them into actionable decisions about IOL selection, surgical feasibility, perioperative risk, and postoperative management [2508.20014].

The benchmark therefore organizes outputs into two task formulations. The first is **Question $\rightarrow$ CoT**, in which a model generates an expert reasoning chain from a diagnostic question. The second is **Question $\rightarrow$ Response**, in which a model produces the final surgical advice. The intended recommendation content includes a summary of findings, personalized IOL selection including **type and power**, surgical technique suggestions, risk alerts, and postoperative management. This structure places CataractSurg-80K in the category of **decision-oriented medical AI**, rather than generic medical question answering or image classification [2508.20014].

A recurrent misconception is to interpret CataractSurg-80K as a cataract surgery video benchmark because of its name. The published description instead frames it as a **report-to-recommendation** benchmark grounded in ophthalmology electronic medical records and structured clinical reasoning. The benchmark focuses on preoperative planning, not on tool tracking, phase recognition, or scene segmentation [2508.20014].

## 2. Data provenance, construction workflow, and release model

The benchmark was curated from **ophthalmology electronic medical records at Peking University Third Hospital**. Its construction pipeline begins with raw hospital reports, routes them through a **knowledge-driven Multi-Agent System (MAS)**, merges the agent outputs into a structured patient description, and then derives benchmark fields for question generation, reasoning supervision, and recommendation synthesis [2508.20014].

The released resource is not the raw clinical record collection. The authors state that **raw patient EMRs cannot be released** because of privacy and ethics constraints. Instead, the public release consists of **strictly desensitized and structured downstream data**. The open-sourced subsets are listed as `cataract_base_zh`, `cataract_base_en`, `inference01_zh`, `inference01_en`, `inference02_zh`, and `inference02_en`, and all subsets follow the **Alpaca SFT format**. The accompanying codebase is released through the **CataractCareMAS** repository [2508.20014].

The paper reports a **benchmark subset of 777 cases** for evaluation. This subset includes **all common types of cataracts** and **a small number of normal (non-cataract) cases** as negative controls. Reported demographic statistics are a **mean age of 69.4 years**, **median age 70.0 years**, and **age SD 10.5 years**. The surgical method distribution is described only coarsely: the majority underwent **phacoemulsification**, and a small number involved **capsular tension ring (CTR) implantation**. The benchmark also analyzes completeness for six biometric parameters—AL, ACD, K1, K2, LT, and CCT—while noting that some values are missing because of incomplete documentation or extraction issues [2508.20014].

An important unresolved point is the benchmark’s name. Although the title uses **“80K”**, the published description does **not clearly explain** whether this refers to total training instances, total instruction pairs, multilingual expansions, or another count. Likewise, the text does not fully report total raw-case counts, exact train/validation/test splits, or the complete size of the full underlying resource beyond the 777-case benchmark subset [2508.20014].

## 3. Example structure and reasoning supervision

Each CataractSurg-80K example is built around a **structured patient description** and contains at least three semantically distinct fields: a **diagnostic-style question**, a **Complex\_CoT** reasoning trace, and a **Response** containing the structured recommendation. The benchmark explicitly separates reasoning supervision from response generation through the pairings $\langle \text{Question}, \text{Complex\_CoT} \rangle$ and $\langle \text{Question}, \text{Response} \rangle$ [2508.20014].

The **question field** is designed to foreground clinically relevant findings without prematurely committing to a diagnosis or treatment plan. The description states that these questions emphasize features such as **axial length**, **lens opacity grade**, **corneal astigmatism**, and **biometric anomalies**. The intention is to make the prompt diagnostic and decision-relevant while preserving the need for reasoning rather than templated answer retrieval [2508.20014].

The **Complex\_CoT** field follows an **eight-step reasoning framework**. The published description does not enumerate all eight headings verbatim, but it states that the framework covers **biometric parameter analysis**, **corneal topography interpretation**, **identification of abnormal findings**, **IOL selection logic**, **surgical risk assessment**, and **patient-specific adaptations**. This design converts the benchmark into a substrate for structured interpretability, not merely answer scoring [2508.20014].

The **Response** field synthesizes the findings and reasoning into a structured plan. Reported components include a key-finding summary, **IOL type**, **IOL power**, **surgical technique suggestion**, **risk warnings**, and a **postoperative management plan**. The benchmark description also indicates associated metadata such as age, sex, case status as abnormal/normal/unspecified, biometric parameter availability, and recommended surgical method category, although a full schema is not printed in the text [2508.20014].

## 4. Knowledge-driven Multi-Agent System

The **Multi-Agent System** is the benchmark’s central intermediate layer. It is designed to reduce the mismatch between general-purpose language models and heterogeneous ophthalmic reporting by decomposing report interpretation into modality-specific specialist agents. The named agents correspond to **OCT**, **corneal topography/Pentacam**, **fundus**, and **master reporting/biometric summary** [2508.20014].

Each agent receives raw report content for its modality and produces a **structured summary** constrained to a **shared output schema**. The system is described as limiting extraction to **clinically verifiable parameters only**, with each extracted fact linked to its **source** and **responsible agent**. The stated purpose is to improve interoperability, traceability, and auditability while reducing hallucination and over-interpretation [2508.20014].

The same MAS logic is used in both **training** and **deployment**. During data construction, raw ophthalmic reports are routed to specialist agents, transformed into structured modality outputs, merged into a unified patient description, and then converted into question, Complex\_CoT, and recommendation fields. During deployment, the same routing-and-aggregation process is used to transform incoming reports into actionable structured summaries before downstream reasoning. This implies that CataractSurg-80K is not merely a static annotation corpus; it is tied to a modular report-normalization framework [2508.20014].

Expert involvement is emphasized in the prompting layer rather than through a formal adjudication table. The prompts were **co-designed with ophthalmologists**, the four report types were chosen by **field experts**, and prompt templates are provided for **master report and fundus interpretation**, **Pentacam and OCT interpretation**, and **reasoning-enhanced dataset construction**. Quality control is described through specialized agents, schema constraints, privacy-aware master report reconstruction, and traceability, but the published text does **not report** inter-rater agreement, double-annotation rates, or manual audit proportions [2508.20014].

## 5. Qwen-CSP, training stages, and evaluation protocol

The benchmark is paired with **Qwen-CSP**, a domain-specialized model for cataract surgery planning. The description contains a minor naming inconsistency: the abstract and introduction characterize it as built on **Qwen-4B**, while the methodology refers to **Qwen3-4B**. In both cases, the operative scale is a **4B-parameter Qwen model** adapted for cataract planning [2508.20014].

Qwen-CSP is trained through a **multi-stage supervised fine-tuning pipeline**. The first stage adapts the base model to cataract-domain knowledge using a separate instruction-QA dataset derived from the **APACRS “Patient Guide for Cataract Surgery”** in Chinese and English. The second stage trains on $\langle \text{Question}, \text{Complex\_CoT} \rangle$ pairs to learn ophthalmologist-style reasoning. The third stage trains on $\langle \text{Question}, \text{Response} \rangle$ pairs to generate final structured recommendations. This progression is presented as successive injection of domain knowledge, reasoning ability, and recommendation ability [2508.20014].

Evaluation covers both text similarity and medically salient content recovery. Reported **text generation metrics** are **BLEU**, **ROUGE-L**, and **BERTScore$_{F1}$**. Reported **medical keypoint metrics** are **k-F1**, **k-Precision**, and **k-Recall**, computed using **SciSpacy entity recognition** and automatic extraction of medical terminology vocabulary. For CoT generation specifically, the evaluation adds **Entity F1**, **Entity Precision**, **Entity Recall**, and **NLI Consistency**, the last of which measures logical entailment between consecutive reasoning steps using a pretrained NLI model [2508.20014].

The comparator set includes **GPT4.1**, **Claude-3.7**, **Gemini-2**, **DeepSeek**, **Qwen-Max**, **Qwen-8B**, and **Qwen-4B**. The ablation variants are **Base**, **Base + B + C**, **Base + A + C**, and **Base + A + B + C**, where **A** denotes domain knowledge adaptation, **B** CoT supervision, and **C** response-generation supervision. Results are reported as **mean ± standard deviation over 777 cases**. The published text does **not provide** optimizer settings, learning rate, batch size, epoch counts, decoding hyperparameters, or a formal human-evaluation rubric [2508.20014].

## 6. Empirical results, unresolved issues, and position within cataract-surgery AI

Qwen-CSP is reported to outperform the compared general-purpose and base-domain models on most automatic metrics. On the **CoT generation** task, its reported scores are **BLEU 0.131 ± 0.029**, **ROUGE-L 0.261 ± 0.034**, **BERTScore$_{F1}$ 0.886 ± 0.015**, **k-F1 0.292 ± 0.043**, **k-Precision 0.317 ± 0.046**, **k-Recall 0.275 ± 0.045**, **Entity F1 0.258 ± 0.083**, **Entity Precision 0.294 ± 0.096**, **Entity Recall 0.270 ± 0.082**, and **NLI Consistency 0.158 ± 0.324**. On the **Response** task, the reported scores are **BLEU 0.170 ± 0.041**, **ROUGE-L 0.330 ± 0.044**, **BERTScore$_{F1}$ 0.896 ± 0.013**, **k-F1 0.367 ± 0.054**, **k-Precision 0.377 ± 0.062**, and **k-Recall 0.361 ± 0.051**. The paper summarizes these gains as substantial improvements over strong baselines in both terminology recovery and recommendation quality [2508.20014].

The ablations identify two distinct effects. Removing **domain knowledge adaptation** degrades both CoT and response quality; for example, CoT BLEU drops from **0.131** to **0.103**, and Response BLEU drops from **0.170** to **0.102**. Removing **CoT supervision** sharply reduces explanation-oriented metrics, with CoT BLEU falling to **0.056**, although this variant remains strong on some response metrics and achieves **NLI Consistency 0.976**. The authors interpret this as evidence that richer reasoning supervision improves informativeness but can also increase logical fluctuation across steps [2508.20014].

That trade-off is the benchmark’s most salient caveat. The full Qwen-CSP model shows a marked reduction in **NLI Consistency** relative to the base model, despite outperforming it on factual and recommendation-oriented metrics. This suggests that domain specialization and more detailed clinical reasoning do not automatically yield more internally coherent chains. The paper identifies this as an open problem and suggests future mechanisms such as stepwise NLI discrimination during training and medical knowledge-graph constraints [2508.20014].

Case studies are used to illustrate the function of heterogeneous report integration. In a diabetic cataract case, the presence of an **OCT report** enabled the MAS to identify **diabetic macular edema**, which altered risk warnings and postoperative planning; when OCT was missing, the output became more generic. Another described case combined abnormal corneal morphology, significant astigmatism, dry eye, and glaucoma, and is used to argue that the MAS can aggregate cross-report findings into clinically aligned recommendations [2508.20014].

Within the broader cataract AI literature, CataractSurg-80K occupies a different niche from datasets such as **CaDIS**, which targets semantic segmentation of cataract surgery images, and **Cataract-1K**, which targets scene segmentation, phase recognition, and irregularity detection from surgical video [1906.11586] [2312.06295]. A plausible implication is that CataractSurg-80K functions as a benchmark for **preoperative structured reasoning**, whereas those resources function as benchmarks for **intraoperative visual understanding**. The distinction is methodological as well as clinical: CataractSurg-80K is centered on **heterogeneous report interpretation and surgical planning**, not on microscope-video perception [2508.20014].

Its principal limitations follow directly from that design. The resource uses only **four key report types**; it derives from a **single institution**; it does not release raw EMRs; it does not clearly explain the origin of the **“80K”** name; and the published text does not fully disclose total counts, split sizes, or implementation hyperparameters. These omissions do not alter the benchmark’s stated contribution, but they constrain reproducibility and make exact scale interpretation less straightforward than the title suggests [2508.20014].

Source: https://www.emergentmind.com/topics/cataractsurg-80k