CataractSurg-80K: Preoperative Planning Benchmark
- CataractSurg-80K is a benchmark for preoperative cataract surgery planning that employs structured clinical reasoning to integrate heterogeneous ophthalmic exams.
- It utilizes multimodal data from OCT, corneal topography, fundus, and biometric reports to inform IOL selection, surgical risk assessment, and management strategies.
- The benchmark supports dual task formulations—reasoning chain generation and final surgical advice—underpinning advances in decision-oriented medical AI.
to=arxiv_search 彩神争霸代理_search 天天中彩票一等奖 code: {"query":"CataractSurg-80K knowledge-driven benchmarking structured reasoning ophthalmic surgery planning", "max_results": 5, "sort_by": "relevance"} to=arxiv_search ՞ւազ_search code: {"query":"cataract surgery planning benchmark structured clinical reasoning ophthalmic reports LLM", "max_results": 10, "sort_by": "relevance"} CataractSurg-80K is a benchmark for cataract surgery planning rather than a surgical video-analysis dataset. It is presented as the first large-scale benchmark for cataract surgery planning with structured clinical reasoning, designed to model the preoperative workflow in which heterogeneous ophthalmic examinations are integrated into patient assessment, intraocular lens (IOL) selection, surgical risk evaluation, and recommendation generation. Its defining pipeline maps structured information derived from OCT, Pentacam corneal topography, fundus reports, and master biometric reports into a diagnostic-style question, an expert-style Complex_CoT reasoning trace, and a structured surgical recommendation (Meng et al., 27 Aug 2025).
1. Scope and clinical task definition
CataractSurg-80K addresses preoperative cataract surgery planning as a structured reasoning problem. The benchmark is motivated by the observation that cataract planning is not a single-label prediction task: clinicians must reconcile multiple ophthalmic reports, interpret parameters such as axial length (AL), anterior chamber depth (ACD), K1, K2, lens thickness (LT), and central corneal thickness (CCT), and convert them into actionable decisions about IOL selection, surgical feasibility, perioperative risk, and postoperative management (Meng et al., 27 Aug 2025).
The benchmark therefore organizes outputs into two task formulations. The first is Question CoT, in which a model generates an expert reasoning chain from a diagnostic question. The second is Question Response, in which a model produces the final surgical advice. The intended recommendation content includes a summary of findings, personalized IOL selection including type and power, surgical technique suggestions, risk alerts, and postoperative management. This structure places CataractSurg-80K in the category of decision-oriented medical AI, rather than generic medical question answering or image classification (Meng et al., 27 Aug 2025).
A recurrent misconception is to interpret CataractSurg-80K as a cataract surgery video benchmark because of its name. The published description instead frames it as a report-to-recommendation benchmark grounded in ophthalmology electronic medical records and structured clinical reasoning. The benchmark focuses on preoperative planning, not on tool tracking, phase recognition, or scene segmentation (Meng et al., 27 Aug 2025).
2. Data provenance, construction workflow, and release model
The benchmark was curated from ophthalmology electronic medical records at Peking University Third Hospital. Its construction pipeline begins with raw hospital reports, routes them through a knowledge-driven Multi-Agent System (MAS), merges the agent outputs into a structured patient description, and then derives benchmark fields for question generation, reasoning supervision, and recommendation synthesis (Meng et al., 27 Aug 2025).
The released resource is not the raw clinical record collection. The authors state that raw patient EMRs cannot be released because of privacy and ethics constraints. Instead, the public release consists of strictly desensitized and structured downstream data. The open-sourced subsets are listed as cataract_base_zh, cataract_base_en, inference01_zh, inference01_en, inference02_zh, and inference02_en, and all subsets follow the Alpaca SFT format. The accompanying codebase is released through the CataractCareMAS repository (Meng et al., 27 Aug 2025).
The paper reports a benchmark subset of 777 cases for evaluation. This subset includes all common types of cataracts and a small number of normal (non-cataract) cases as negative controls. Reported demographic statistics are a mean age of 69.4 years, median age 70.0 years, and age SD 10.5 years. The surgical method distribution is described only coarsely: the majority underwent phacoemulsification, and a small number involved capsular tension ring (CTR) implantation. The benchmark also analyzes completeness for six biometric parameters—AL, ACD, K1, K2, LT, and CCT—while noting that some values are missing because of incomplete documentation or extraction issues (Meng et al., 27 Aug 2025).
An important unresolved point is the benchmark’s name. Although the title uses “80K”, the published description does not clearly explain whether this refers to total training instances, total instruction pairs, multilingual expansions, or another count. Likewise, the text does not fully report total raw-case counts, exact train/validation/test splits, or the complete size of the full underlying resource beyond the 777-case benchmark subset (Meng et al., 27 Aug 2025).
3. Example structure and reasoning supervision
Each CataractSurg-80K example is built around a structured patient description and contains at least three semantically distinct fields: a diagnostic-style question, a Complex_CoT reasoning trace, and a Response containing the structured recommendation. The benchmark explicitly separates reasoning supervision from response generation through the pairings and (Meng et al., 27 Aug 2025).
The question field is designed to foreground clinically relevant findings without prematurely committing to a diagnosis or treatment plan. The description states that these questions emphasize features such as axial length, lens opacity grade, corneal astigmatism, and biometric anomalies. The intention is to make the prompt diagnostic and decision-relevant while preserving the need for reasoning rather than templated answer retrieval (Meng et al., 27 Aug 2025).
The Complex_CoT field follows an eight-step reasoning framework. The published description does not enumerate all eight headings verbatim, but it states that the framework covers biometric parameter analysis, corneal topography interpretation, identification of abnormal findings, IOL selection logic, surgical risk assessment, and patient-specific adaptations. This design converts the benchmark into a substrate for structured interpretability, not merely answer scoring (Meng et al., 27 Aug 2025).
The Response field synthesizes the findings and reasoning into a structured plan. Reported components include a key-finding summary, IOL type, IOL power, surgical technique suggestion, risk warnings, and a postoperative management plan. The benchmark description also indicates associated metadata such as age, sex, case status as abnormal/normal/unspecified, biometric parameter availability, and recommended surgical method category, although a full schema is not printed in the text (Meng et al., 27 Aug 2025).
4. Knowledge-driven Multi-Agent System
The Multi-Agent System is the benchmark’s central intermediate layer. It is designed to reduce the mismatch between general-purpose LLMs and heterogeneous ophthalmic reporting by decomposing report interpretation into modality-specific specialist agents. The named agents correspond to OCT, corneal topography/Pentacam, fundus, and master reporting/biometric summary (Meng et al., 27 Aug 2025).
Each agent receives raw report content for its modality and produces a structured summary constrained to a shared output schema. The system is described as limiting extraction to clinically verifiable parameters only, with each extracted fact linked to its source and responsible agent. The stated purpose is to improve interoperability, traceability, and auditability while reducing hallucination and over-interpretation (Meng et al., 27 Aug 2025).
The same MAS logic is used in both training and deployment. During data construction, raw ophthalmic reports are routed to specialist agents, transformed into structured modality outputs, merged into a unified patient description, and then converted into question, Complex_CoT, and recommendation fields. During deployment, the same routing-and-aggregation process is used to transform incoming reports into actionable structured summaries before downstream reasoning. This implies that CataractSurg-80K is not merely a static annotation corpus; it is tied to a modular report-normalization framework (Meng et al., 27 Aug 2025).
Expert involvement is emphasized in the prompting layer rather than through a formal adjudication table. The prompts were co-designed with ophthalmologists, the four report types were chosen by field experts, and prompt templates are provided for master report and fundus interpretation, Pentacam and OCT interpretation, and reasoning-enhanced dataset construction. Quality control is described through specialized agents, schema constraints, privacy-aware master report reconstruction, and traceability, but the published text does not report inter-rater agreement, double-annotation rates, or manual audit proportions (Meng et al., 27 Aug 2025).
5. Qwen-CSP, training stages, and evaluation protocol
The benchmark is paired with Qwen-CSP, a domain-specialized model for cataract surgery planning. The description contains a minor naming inconsistency: the abstract and introduction characterize it as built on Qwen-4B, while the methodology refers to Qwen3-4B. In both cases, the operative scale is a 4B-parameter Qwen model adapted for cataract planning (Meng et al., 27 Aug 2025).
Qwen-CSP is trained through a multi-stage supervised fine-tuning pipeline. The first stage adapts the base model to cataract-domain knowledge using a separate instruction-QA dataset derived from the APACRS “Patient Guide for Cataract Surgery” in Chinese and English. The second stage trains on pairs to learn ophthalmologist-style reasoning. The third stage trains on pairs to generate final structured recommendations. This progression is presented as successive injection of domain knowledge, reasoning ability, and recommendation ability (Meng et al., 27 Aug 2025).
Evaluation covers both text similarity and medically salient content recovery. Reported text generation metrics are BLEU, ROUGE-L, and BERTScore. Reported medical keypoint metrics are k-F1, k-Precision, and k-Recall, computed using SciSpacy entity recognition and automatic extraction of medical terminology vocabulary. For CoT generation specifically, the evaluation adds Entity F1, Entity Precision, Entity Recall, and NLI Consistency, the last of which measures logical entailment between consecutive reasoning steps using a pretrained NLI model (Meng et al., 27 Aug 2025).
The comparator set includes GPT4.1, Claude-3.7, Gemini-2, DeepSeek, Qwen-Max, Qwen-8B, and Qwen-4B. The ablation variants are Base, Base + B + C, Base + A + C, and Base + A + B + C, where A denotes domain knowledge adaptation, B CoT supervision, and C response-generation supervision. Results are reported as mean ± standard deviation over 777 cases. The published text does not provide optimizer settings, learning rate, batch size, epoch counts, decoding hyperparameters, or a formal human-evaluation rubric (Meng et al., 27 Aug 2025).
6. Empirical results, unresolved issues, and position within cataract-surgery AI
Qwen-CSP is reported to outperform the compared general-purpose and base-domain models on most automatic metrics. On the CoT generation task, its reported scores are BLEU 0.131 ± 0.029, ROUGE-L 0.261 ± 0.034, BERTScore 0.886 ± 0.015, k-F1 0.292 ± 0.043, k-Precision 0.317 ± 0.046, k-Recall 0.275 ± 0.045, Entity F1 0.258 ± 0.083, Entity Precision 0.294 ± 0.096, Entity Recall 0.270 ± 0.082, and NLI Consistency 0.158 ± 0.324. On the Response task, the reported scores are BLEU 0.170 ± 0.041, ROUGE-L 0.330 ± 0.044, BERTScore 0.896 ± 0.013, k-F1 0.367 ± 0.054, k-Precision 0.377 ± 0.062, and k-Recall 0.361 ± 0.051. The paper summarizes these gains as substantial improvements over strong baselines in both terminology recovery and recommendation quality (Meng et al., 27 Aug 2025).
The ablations identify two distinct effects. Removing domain knowledge adaptation degrades both CoT and response quality; for example, CoT BLEU drops from 0.131 to 0.103, and Response BLEU drops from 0.170 to 0.102. Removing CoT supervision sharply reduces explanation-oriented metrics, with CoT BLEU falling to 0.056, although this variant remains strong on some response metrics and achieves NLI Consistency 0.976. The authors interpret this as evidence that richer reasoning supervision improves informativeness but can also increase logical fluctuation across steps (Meng et al., 27 Aug 2025).
That trade-off is the benchmark’s most salient caveat. The full Qwen-CSP model shows a marked reduction in NLI Consistency relative to the base model, despite outperforming it on factual and recommendation-oriented metrics. This suggests that domain specialization and more detailed clinical reasoning do not automatically yield more internally coherent chains. The paper identifies this as an open problem and suggests future mechanisms such as stepwise NLI discrimination during training and medical knowledge-graph constraints (Meng et al., 27 Aug 2025).
Case studies are used to illustrate the function of heterogeneous report integration. In a diabetic cataract case, the presence of an OCT report enabled the MAS to identify diabetic macular edema, which altered risk warnings and postoperative planning; when OCT was missing, the output became more generic. Another described case combined abnormal corneal morphology, significant astigmatism, dry eye, and glaucoma, and is used to argue that the MAS can aggregate cross-report findings into clinically aligned recommendations (Meng et al., 27 Aug 2025).
Within the broader cataract AI literature, CataractSurg-80K occupies a different niche from datasets such as CaDIS, which targets semantic segmentation of cataract surgery images, and Cataract-1K, which targets scene segmentation, phase recognition, and irregularity detection from surgical video (Grammatikopoulou et al., 2019, Ghamsarian et al., 2023). A plausible implication is that CataractSurg-80K functions as a benchmark for preoperative structured reasoning, whereas those resources function as benchmarks for intraoperative visual understanding. The distinction is methodological as well as clinical: CataractSurg-80K is centered on heterogeneous report interpretation and surgical planning, not on microscope-video perception (Meng et al., 27 Aug 2025).
Its principal limitations follow directly from that design. The resource uses only four key report types; it derives from a single institution; it does not release raw EMRs; it does not clearly explain the origin of the “80K” name; and the published text does not fully disclose total counts, split sizes, or implementation hyperparameters. These omissions do not alter the benchmark’s stated contribution, but they constrain reproducibility and make exact scale interpretation less straightforward than the title suggests (Meng et al., 27 Aug 2025).