ConnectomeBench: AI Proofreading in Connectomics
- ConnectomeBench is a multimodal benchmark for evaluating AI-driven proofreading of connectome segmentations rendered in 3D.
- It assesses tasks such as segment type identification, split error correction, and merge error detection using expert-annotated MICrONS and FlyWire datasets.
- The benchmark reveals current LLM performance gaps compared to human experts and underscores the benefit of heuristic prompt engineering.
Searching arXiv for ConnectomeBench and closely related benchmark/context papers. ConnectomeBench denotes a multimodal benchmark for connectome proofreading introduced in “ConnectomeBench: Can LLMs Proofread the Connectome?” (Brown et al., 31 Oct 2025). It evaluates whether contemporary multimodal LLMs can perform three proofreading tasks that arise after imaging and machine-learning-assisted segmentation: segment type identification, split error correction, and merge error detection. The benchmark uses expert-annotated data from MICrONS, a cubic millimeter of mouse visual cortex, and FlyWire, the complete Drosophila brain, and frames proofreading as a visual-reasoning problem over rendered 3D neuron segmentations rather than as a text-only task (Brown et al., 31 Oct 2025). The name has also been used for a browser-based neural connectivity visualization system described in “Interactive Web Application for Exploring Matrices of Neural Connectivity” (Caldwell et al., 2017), creating a terminological overlap within the connectomics literature.
1. Motivation and problem formulation
ConnectomeBench was proposed in response to the persistent proofreading burden in connectomics. The benchmark paper states that segmentation algorithms still create split errors, in which one neuron is broken into pieces, and merge errors, in which multiple neurons are incorrectly fused, and it cites the first complete fly connectome as requiring an estimated 33 human years of proofreading (Brown et al., 31 Oct 2025). This positions proofreading not as a marginal cleanup stage but as a major bottleneck in connectome construction.
The benchmark evaluates three tasks. Segment type identification asks a model to classify a 3D segmented object as one of five categories: a single neuron with soma and processes, multiple neurons merged together, neuronal processes without soma, nucleus, or non-neuronal structure. Split error correction asks whether two separated segments should be merged because they are actually one neuron that was split apart. Merge error detection asks whether a segment incorrectly combines multiple neurons and should be split (Brown et al., 31 Oct 2025).
A central conceptual feature is that these are treated as morphology-sensitive visual judgments over 3D structure. The prompts use three rendered views—top, side, and front—so the benchmark targets reasoning about neuronal geometry rather than only local image texture (Brown et al., 31 Oct 2025). This suggests that the benchmark is aimed at a narrower but more operationally meaningful capability than generic image understanding: the ability to reason about proofreading-relevant morphology.
2. Data sources and benchmark construction
The benchmark is built from two large, open-source, human-proofread connectomics datasets: MICrONS and FlyWire (Brown et al., 31 Oct 2025). MICrONS provides a cubic millimeter of mouse visual cortex, while FlyWire provides the complete Drosophila brain. The paper identifies CAVEClient as the key enabling tool because it exposes the edit history of each segmentation, allowing reconstruction of ground-truth examples in which humans corrected merges or splits (Brown et al., 31 Oct 2025).
For each proofread object or proofreading event, the authors render 3D meshes into images from three angles and crop around the relevant region with fixed-size bounding boxes determined by task. Image size is standardized to 1024 × 1024 pixels. The models are not trained on these images; the benchmark is intended to measure baseline multimodal reasoning rather than finetuned connectomics performance (Brown et al., 31 Oct 2025).
For segment type identification, the paper reports the following category distributions. In FlyWire, the counts are 117, 13, 175, 27, and 18 for single soma and processes, multiple somas and processes, neuronal processes without soma, nucleus, and non-neuronal types, respectively. In MICrONS, the counts are 130, 27, 116, 92, and 1 in the same order (Brown et al., 31 Oct 2025). Because MICrONS has only one non-neuronal example, that category is excluded from the balanced-accuracy calculation there.
For the proofreading tasks, the benchmark derives examples directly from edit histories. Positive split-error examples come from human merges that repaired a split, whereas positive merge-error examples come from proofread segments in which humans introduced a split to correct an erroneous merge (Brown et al., 31 Oct 2025). This makes the benchmark historically grounded in actual proofreading operations rather than synthetic corruption alone.
3. Task mechanics and formalization
The split-error task is provided in both binary and multiple-choice forms. In the binary setting, the model returns 1 if a proposed merge is correct and resolves a split error and -1 if there is no split error or the proposed merge is incorrect (Brown et al., 31 Oct 2025). Positive examples are taken from human edit history. Negative examples are generated by finding a local interface point between the true pair—computed by finding nearest mesh vertices across the two segments and averaging their coordinates—and then sampling an incorrect nearby segment under spatial constraints that account for missing imaging slices. The split-error crop is centered on the interface point and uses a 4096 nm × 4096 nm × 4096 nm volume (Brown et al., 31 Oct 2025).
The paper reports the following split-error dataset sizes: FlyWire contains 298 positive and 248 negative examples, while MICrONS contains 494 positive and 473 negative examples (Brown et al., 31 Oct 2025). The multiple-choice version asks the model to select among candidate merge partners or reject all candidates.
Merge-error identification focuses less on the trivial case of multiple somas in one segment—which is partially captured by segment classification—and more on abnormal neurite geometry, such as an axon doubling back or making an unnatural turn at a merge site (Brown et al., 31 Oct 2025). Positive examples are centered at the human proofreading split point. For a positive sample, the bounding box is defined around the smaller resulting segment using
and then
Negative examples reuse the same center and box on the final proofread mesh, assuming the merge error has already been removed (Brown et al., 31 Oct 2025).
For merge errors, the reported dataset sizes are 137 positive and 137 negative in FlyWire and 148 positive and 148 negative in MICrONS (Brown et al., 31 Oct 2025). The benchmark therefore balances the merge-error task explicitly, whereas segment identification is class-imbalanced and is evaluated with balanced accuracy.
4. Evaluation protocol and empirical results
ConnectomeBench evaluates proprietary multimodal models—Claude Sonnet 3.7, Claude Sonnet 4, o4-mini, GPT-4.1, and GPT-4o—as well as open-weight models including InternVL-3 8B, InternVL-3 78B, and NVLM (Brown et al., 31 Oct 2025). The benchmark uses balanced accuracy for segment identification, accuracy for binary and multiple-choice proofreading tasks, majority voting over multiple runs for proprietary models because responses vary, and 95% confidence intervals for most LLM results. Human expert baselines were created by trained graduate and undergraduate raters on subsets of examples. Additional baselines include a ResNet-50 trained separately for each task and a limited NEURD comparison for merge errors (Brown et al., 31 Oct 2025).
The headline findings in the abstract are that current models achieve 52–82% balanced accuracy in segment identification against 20–25% chance, and 75–85% accuracy in binary or multiple-choice split error correction against 50% chance, while generally struggling on merge error identification (Brown et al., 31 Oct 2025).
| Task | Metric | Representative best result |
|---|---|---|
| Segment type identification | Balanced accuracy | Claude Sonnet 3.7 + Description: 0.459 FlyWire, 0.822 MICrONS |
| Split error correction, binary | Accuracy | o4-mini + Heuristics: 0.754 FlyWire, 0.786 MICrONS |
| Merge error identification, binary | Accuracy | o4-mini + Description: 0.628 FlyWire, 0.615 MICrONS |
For segment type identification, chance is approximately 0.2 or 0.25, depending on the number of classes. Reported balanced accuracies include Claude Sonnet 3.7 + Description at 0.459 on FlyWire and 0.822 on MICrONS, GPT-4.1 + Description at 0.529 and 0.655, o4-mini + Description at 0.511 and 0.728, and GPT-4o + Description at 0.396 and 0.588 (Brown et al., 31 Oct 2025). Open-weight models perform worse but remain above chance in some settings: InternVL-3 reaches 0.320 / 0.493, InternVL-3-8B reaches 0.303 / 0.417, and NVLM reaches 0.234 / 0.258. A finetuned ResNet-50 baseline scores 0.552 on FlyWire and 0.587 on MICrONS (Brown et al., 31 Oct 2025).
For binary split-error correction, the best reported result is o4-mini + Heuristics with 0.754 on FlyWire and 0.786 on MICrONS. Human performance is 0.840 and 0.902, respectively, while ResNet-50 reaches 0.720 ± 0.034 and 0.667 ± 0.038 (Brown et al., 31 Oct 2025). In the multiple-choice split task, o4-mini + Description reaches 0.828 on FlyWire and 0.790 on MICrONS, and o4-mini + Heuristics reaches 0.788 and 0.850. Human performance is 0.900 and 0.920 (Brown et al., 31 Oct 2025).
For merge-error identification, performance is substantially weaker. In the binary setting, o4-mini + Description reaches 0.628 on FlyWire and 0.615 on MICrONS, while human performance is 0.740 and 0.800, and ResNet-50 reaches 0.769 ± 0.035 and 0.798 ± 0.02 (Brown et al., 31 Oct 2025). In the multiple-choice setting, o4-mini + Null reaches 0.740 on FlyWire and 0.689 on MICrONS, while human performance is 0.840 and 0.796 (Brown et al., 31 Oct 2025).
A notable methodological result is that the authors extracted seven explicit heuristics from failure analysis, especially for o4-mini, and added them to the prompt. These heuristics address recurring misconceptions such as assuming the orange segment must be tiny, assuming a large gap means the merge is wrong, forgetting that missing slices can create apparent discontinuities, and failing to reason in 3D (Brown et al., 31 Oct 2025). The resulting improvement indicates that prompt engineering can materially affect proofreading performance.
5. Limitations, failure modes, and interpretive cautions
The benchmark paper explicitly states that ConnectomeBench does not capture all proofreading skills (Brown et al., 31 Oct 2025). It omits or only partially addresses synapse identification, full merge correction workflows, more complex interactive proofreading decisions, and some classes of nonlocal or ambiguous errors. The evaluation environment uses rendered 3D images rather than full interactive proofreading software, so the benchmark measures visual reasoning under controlled prompts rather than complete GUI-based connectome editing.
Several model-specific failure modes are reported. Models often rely on superficial visual heuristics, may be biased toward accepting or rejecting merges too aggressively, sometimes misunderstand that large gaps may reflect missing imaging slices, and generally perform worse in merge-error detection than in segment identification or split-error correction (Brown et al., 31 Oct 2025). Open-weight models lag behind frontier proprietary systems, although some remain above chance on selected tasks.
The benchmark’s main conclusion is deliberately cautious. Current multimodal LLMs can do better than chance on important proofreading subtasks; they are already strong on segment identification and can reach substantial, though incomplete, performance on split error correction, especially in multiple-choice form and with heuristic guidance. They remain below expert-human performance overall and struggle most on merge error detection (Brown et al., 31 Oct 2025). A plausible implication is that ConnectomeBench is better interpreted as a progress measure for AI-assisted proofreading than as evidence that end-to-end autonomous proofreading is already available.
6. Position within the connectomics benchmark ecosystem
ConnectomeBench belongs to a broader landscape of benchmark, reference, and infrastructure efforts in connectomics, but it targets a distinct layer of the pipeline. RhoanaNet Pipeline: Dense Automatic Neural Annotation provides an open, reproducible benchmark target for large-scale EM segmentation, with standardized datasets, training and test splits, evaluation metrics centered on , and a full segmentation system rather than a proofreading benchmark (Knowles-Barley et al., 2016). NeuroGraph standardizes graph machine learning on connectome-derived HCP fMRI datasets, including both static and dynamic graphs, more than 15 baselines, and a controlled preprocessing search space (Said et al., 2023). The Budapest Reference Connectome Server v3.0 serves as a parameterizable reference and benchmark resource for healthy human structural connectomes built from 1015-vertex graphs from 477 Human Connectome Project subjects (Szalkai et al., 2016).
| Resource | Primary object | Benchmark or tool role |
|---|---|---|
| ConnectomeBench (Brown et al., 31 Oct 2025) | Proofreading decisions on 3D segmentations | Multimodal benchmark for LLMs |
| RhoanaNet (Knowles-Barley et al., 2016) | EM volume segmentation | Reproducible pipeline benchmark |
| NeuroGraph (Said et al., 2023) | Brain graphs from HCP fMRI | Graph ML benchmark suite |
| Budapest Reference Connectome Server (Szalkai et al., 2016) | Consensus structural connectomes | Reference and benchmark resource |
| Open Connectome Project Data Cluster (Burns et al., 2013) | Spatial imaging and annotation workloads | Infrastructure and workload baseline |
| ConnectomeBench visualization (Caldwell et al., 2017) | Connectivity matrices | Browser-based exploration tool |
At the systems layer, the Open Connectome Project Data Cluster describes scalable infrastructure for connectome-oriented workloads such as large spatial cutouts, annotation writes, metadata queries, and distributed computer-vision pipelines (Burns et al., 2013). At the atlas and population-analysis layer, “Integrated Construction of Multimodal Atlases with Structural Connectomes in the Space of Riemannian Metrics” proposes a geometric framework for structural connectome construction, registration, and atlas building by representing each connectome as a Riemannian metric field, equipping the space with the Ebin metric, and defining atlases as Fréchet means (Campbell et al., 2021). These efforts address segmentation, storage, atlas construction, or graph learning rather than proofreading by multimodal LLMs.
The earlier visualization-oriented use of the name is also distinct. In “Interactive Web Application for Exploring Matrices of Neural Connectivity,” ConnectomeBench refers to a lightweight, open-source, HTML5/JavaScript, client-side, mobile-compatible system for exploring stacked connectivity matrices using a chord diagram with threshold-based pruning, frequency slicing, a linked bar chart, and a histogram (Caldwell et al., 2017). That system addresses exploratory analysis of multivariate neural connectivity matrices, particularly from human ECoG, whereas the 2025 benchmark addresses proofreading decisions over 3D neuron segmentations.
Taken together, these works show that “ConnectomeBench” sits within a heterogeneous methodological field. In one usage it names a browser-native matrix-visualization system (Caldwell et al., 2017); in another, more recent usage, it names a multimodal benchmark for testing whether LLMs can assist with proofreading the connectome (Brown et al., 31 Oct 2025). The latter is explicitly benchmark-centric and operationalizes proofreading as a reproducible AI evaluation problem.