NuInsSeg: H&E Nuclei Instance Segmentation

Updated 8 July 2026

NuInsSeg is a fully manually annotated H&E nuclei segmentation dataset featuring 665 image patches, 30,698 nuclei, and explicit ambiguous area masks.
It encompasses diverse histological variations across 31 organs from human and mouse specimens, enabling robust cross-domain evaluations.
The dataset supports various segmentation tasks including boundary-aware and uncertainty-aware formulations for improved computational pathology.

NuInsSeg is a publicly released, fully manually annotated dataset for nuclei instance segmentation in brightfield Hematoxylin and Eosin (H&E)-stained histological images. It was introduced to address a central bottleneck in computational pathology: supervised deep learning methods for nuclear segmentation require large, accurate instance masks, yet such annotations are expensive and intrinsically difficult in crowded, weakly stained, morphologically variable, or partially indeterminate regions. NuInsSeg contains 665 image patches and 30,698 manually segmented nuclei from 31 human and mouse organs, and its distinguishing feature is the release of dataset-wide ambiguous area masks that explicitly mark regions where precise deterministic annotation is impossible even for experts (Mahbod et al., 2023).

1. Dataset definition and scope

NuInsSeg is positioned as a resource for nuclei instance segmentation, not merely foreground detection. In the dataset paper, “fully annotated” means that, for each image patch, all nuclei instances were manually delineated rather than sparsely labeled or weakly marked. The dataset was designed both for supervised model training and for testing cross-organ generalization under substantial histological variability (Mahbod et al., 2023).

Its core properties are summarized below.

Property	Value
Image patches	665
Manually segmented nuclei	30,698
Patch size	$512 \times 512$ pixels
Source FOV size	$2048 \times 2048$ pixels
Magnification	$40\times$
Species coverage	23 human tissues, 8 mouse tissues
Annotation type	Full manual nucleus instance masks
Additional uncertainty annotation	Ambiguous area masks for the entire dataset

Within the comparison reported by the dataset paper, NuInsSeg has the largest number of image tiles and the widest organ diversity among the fully manually annotated H&E nuclei segmentation datasets listed there, and it is explicitly marked as providing vague masks, unlike most earlier datasets. The paper also contrasts this design with semi-automatically generated datasets such as PanNuke and Lizard, arguing that propagated or model-derived annotations can inherit bias from the generating model rather than reflect direct human delineation (Mahbod et al., 2023).

The dataset is suited to conventional instance segmentation benchmarking, but its structure also supports boundary-aware, distance-based, and uncertainty-aware formulations. A plausible implication is that it can serve simultaneously as a training corpus, a stress test for cross-domain generalization across organs and species, and a benchmark for studying how evaluation changes when annotation uncertainty is made explicit.

2. Acquisition pipeline and biological coverage

NuInsSeg spans 31 organs and tissues across two species. The human tissues are cerebellum, cerebrum, colon, epiglottis, jejunum, kidney, liver, lung, melanoma, muscle, oesophagus, palatine tonsil, pancreas, peritoneum, placenta, salivary gland, spleen, stomach (cardia), stomach (pylorus), testis, tongue, umbilical cord, and urinary bladder. The mouse tissues are bone (femur), fat (subscapularis), heart, kidney, liver, muscle (tibialis anterior muscle), spleen, and thymus (Mahbod et al., 2023).

Human content comprises 472 patches and 23,247 nuclei; mouse content comprises 193 patches and 7,451 nuclei. The overall average is 46.2 nuclei per image, but density varies sharply by tissue. The dataset paper highlights mouse spleen at 236.7 nuclei per image and mouse thymus at 223.7, whereas mouse muscle averages only 5.9 nuclei per image. This heterogeneity is one reason the dataset is relevant for robustness studies rather than only for average-case performance reporting (Mahbod et al., 2023).

The image formation pipeline is unusually explicit. Slides were digitized using a TissueFAXS system built around an Axio Imager Z1 (Zeiss) with a Plan-Neofluar 40×/0.75 air objective, at 8-bit resolution using a Baumer HXG40c color camera. The authors worked from stored $2048 \times 2048$ fields of view rather than whole-slide images as the final learning units; a senior cell biologist selected representative fields, and each selected field was centrally cropped to a $512 \times 512$ patch saved as a lossless PNG (Mahbod et al., 2023).

Preparation protocols also vary across subsets. Some human tissues were formaldehyde-fixed, celloidin-embedded, sectioned at $15\approx20 \mu m$ : jejunum, kidney, liver, oesophagus, palatine tonsil, pancreas, placenta, salivary gland, spleen, and tongue. Other human tissues were formaldehyde-fixed paraffin-embedded and sectioned at $4\approx5 \mu m$ : cerebellum, cerebrum, colon, epiglottis, lung, melanoma, muscle, peritoneum, stomach (cardia), stomach (pylorus), testis, umbilical cord, and urinary bladder. Mouse tissues were FFPE, $4 \mu m$ sections stained with H&E and coverslipped with Entellan. Human specimens came from a teaching collection at the Medical University of Vienna, whereas mouse material came from 8-week-old male C57BL/6J mice (Mahbod et al., 2023).

This combination of species variation, organ breadth, fixation heterogeneity, section-thickness variation, and local density extremes is central to the dataset’s identity. NuInsSeg is not simply multi-organ; it encodes multiple histomorphological and acquisition-induced axes of distribution shift within one release.

3. Annotation protocol and released representations

Annotation was performed in ImageJ/Fiji using the ROI Manager and the freehand tool. The annotator manually traced nucleus borders until all nuclei in a patch were segmented. The primary annotation workforce consisted of three students with a background in cell biology; annotations were then controlled by a senior cell biologist and corrected when necessary. The paper does not report a formal consensus protocol, STAPLE-style fusion, or inter-annotator agreement statistics, so the quality-control model is expert review rather than quantified multi-rater reconciliation (Mahbod et al., 2023).

A central methodological contribution is the explicit introduction of ambiguous or vague area masks. These masks identify regions where accurate and deterministic manual annotation is impossible owing to artifacts or intrinsic difficulty, including folded tissue, out-of-focus scanning, variable stain intensity, dense nuclei clusters, and highly complex morphology. Some images contain no ambiguous regions; others do. The underlying claim is epistemic rather than purely technical: the “ground truth” for nuclear instance segmentation is not always fully objective (Mahbod et al., 2023).

The released data include raw RGB image patches, binary segmentation masks, labeled segmentation masks, ambiguous region masks, and several auxiliary masks. The labeled masks encode instance segmentation by assigning distinct IDs to individual nuclei; the binary masks encode nucleus versus background. Auxiliary masks include border-removed binary masks, distance maps described as “elucidation distance maps of nuclei,” and weighted binary masks in which touching-object borders receive higher weights. ROI zip files are also part of the release, and MATLAB 2020a scripts convert ROI annotations into the various mask forms (Mahbod et al., 2023).

The generation pipeline is described stepwise: tissue preparation and staining, digitization at $40\times$ , representative FOV selection, central cropping to $512 \times 512$ , full manual contouring in ImageJ, ROI zip export, conversion to binary and labeled PNG masks in MATLAB 2020a, tracing of ambiguous regions, generation of auxiliary masks, expert review, and public release. The dataset is hosted on Kaggle, and the mask-generation code is released on GitHub. The paper does not specify a software or data license (Mahbod et al., 2023).

These design choices have direct methodological consequences. A plausible implication is that ambiguous regions can be excluded from loss computation, down-weighted, or used as uncertainty-aware supervision targets; similarly, the auxiliary masks enable training regimes centered on border suppression, distance regression, or boundary weighting. The original paper does not formalize such objectives, but it provides the annotation substrate required to define them.

4. Benchmark protocol and baseline segmentation results

The baseline benchmark in the original paper uses five-fold cross-validation. The full dataset of 665 images was randomly partitioned into five folds with 133 images per fold using Scikit-learn and a fixed random state. The paper does not describe organ-stratified, patient-level, or specimen-level splitting; the stated protocol is a random equal-count image split (Mahbod et al., 2023).

The baseline models are six U-Net-family architectures: shallow U-Net, deep U-Net, attention U-Net, residual attention U-Net, two-stage U-Net, and dual decoder U-Net. The shallow and deep U-Nets are described as close to the original U-Net but with dropout layers between all convolutional layers in encoder and decoder; the shallow model uses four convolutional blocks and the deep model five. Evaluation uses Dice, Aggregate Jaccard Index (AJI), and Panoptic Quality (PQ), but the paper does not give explicit mathematical formulas for these metrics (Mahbod et al., 2023).

The reported five-fold results are as follows.

Model	Parameters	Dice / AJI / PQ
Shallow U-Net	1.9 million	78.8 / 50.5 / 42.7
Deep U-Net	7.7 million	79.7 / 49.4 / 40.4
Attention U-Net	2.3 million	80.5 / 45.7 / 36.4
Residual attention U-Net	2.4 million	81.4 / 46.2 / 36.9
Two-stage U-Net	3.9 million	76.6 / 52.8 / 47.2
Dual decoder U-Net	3.5 million	79.4 / 55.9 / 51.3

Residual attention U-Net achieved the best average Dice at 81.4%, whereas dual decoder U-Net achieved the best instance-oriented scores, with AJI 55.9% and PQ 51.3%. The paper emphasizes that these are baseline rather than optimized results and explicitly notes that performance could likely be improved through ensembling, stain augmentation, or test-time augmentation, none of which were pursued in the study (Mahbod et al., 2023).

The software stack reported for the baseline ecosystem includes TensorFlow 2.6.2, Keras 2.6.0, Scikit-learn 0.23.2, Scikit-image 0.19.1, Albumentations 1.1.0, Pandas 1.3.5, and Matplotlib 3.5.1. However, the manuscript text does not specify exact augmentation operations, loss functions, optimizer settings, learning rates, or stain normalization procedures. That omission is part of the dataset’s interpretive context: NuInsSeg is richly annotated, but the original baseline recipes are not fully parameterized in the paper text (Mahbod et al., 2023).

5. Role in later methodological and evaluation research

NuInsSeg quickly became more than a training set; later work uses it as a testbed for weak supervision, prompt-based segmentation, panoptic reformulations, and evaluation theory. In NucEval, it is the central dataset precisely because it uniquely combines full-image vague area annotations with overlap-preserving and merged-overlap mask representations. Using HoVer-Net, Hover-Next, and CellViT under five-fold cross-validation, NucEval shows that cumulative evaluation modifications for vague regions, score normalization, overlap handling, and border uncertainty can raise PQ from 53.40 to 64.07 for HoVer-Net, from 54.17 to 64.92 for Hover-Next, and from 56.44 to 68.44 for CellViT. NucEval also reports an overlap-format artifact in which even an idealized comparison reaches only PQ 88.20, AJI 89.66, Dice 96.02, DQ 99.35, and SQ 88.87 when overlap is merged into a single label map, demonstrating that annotation representation itself can cap achievable scores (Mahbod et al., 4 May 2026).

ExplainSeg repurposes NuInsSeg into a weakly supervised classification-to-segmentation benchmark. In that study, 600 images are used for training and 65 for validation, with dataset-specific normalization from training-set mean and standard deviation, and a DINO Vision Transformer initialized with deitsmall16 weights. No segmentation masks are used during classifier fine-tuning; masks are derived afterward from Integrated Gradients relevance maps, optionally fused with ViT features, then post-processed through morphology or Normalized Cut and refined with DenseCRF. On NuInsSeg, the best ablation is Fusion + Morphology at 13.4 mIoU and 22.4 Dice, while the main ExplainSeg (XNCut) configuration reaches 13.1 mIoU and 20.6 Dice, outperforming TokenCut, MICRA-Net, and MaskCut but remaining modest in absolute terms (Ma et al., 6 Aug 2025).

PanopMamba uses NuInsSeg as one of two principal nuclei panoptic segmentation benchmarks. In that work, the dataset is randomly split 80/20 into train and test, converted to MS COCO panoptic format, and tissue is treated as background because no tissue panoptic labels are available. The paper treats NuInsSeg as a 31-class nuclei panoptic segmentation benchmark and reports that PanopMamba achieves the best values on every reported metric: PQ 73.68, mPQ+ 69.08, bPQ 73.01, iPQ 79.51, wPQ 80.69, fwPQ 79.96, and $2048 \times 2048$ 0. The same paper remarks that models generally perform better on NuInsSeg than on MoNuSAC2020 because NuInsSeg contains more medium- and large-sized nuclei, which makes them easier to identify and segment (Kang et al., 23 Jan 2026).

Prompt-based pathology benchmarking also uses NuInsSeg as a nuclei-scale stress test. In the SAM3 evaluation, NuInsSeg is treated as a single-class nuclei foreground segmentation problem under three-fold cross-validation. The study reports that specialized medical terminology performs poorly in zero-shot text prompting on NuInsSeg, with 3.00 mIoU and 5.79 Dice, whereas the generic prompt “cell” reaches 68.15 mIoU and 81.06 Dice. Oracle box prompting is highly budget-sensitive: one box yields 6.26 mIoU and 10.33 Dice, 28 boxes yield 69.16 and 80.45, and oracle-all boxes yield 81.19 and 89.54. Under supervised adaptation, SAM3-Adapter reaches Dice 80.14, compared with a pathology-specific reference Dice of 81.4 cited from the NuInsSeg paper (Kong et al., 20 Apr 2026).

Taken together, these later studies show that NuInsSeg functions simultaneously as a dataset, a robustness probe, an uncertainty benchmark, and a platform for comparing classical fully supervised instance segmentation against weakly supervised, prompt-based, and evaluation-aware alternatives.

6. Limitations, interpretive caveats, and name disambiguation

NuInsSeg’s strengths are matched by several explicit limitations. The original paper does not report inter-annotator agreement, does not define a formal ambiguity-aware loss or ambiguity-aware evaluation protocol, does not specify exact baseline training details such as loss functions and augmentation operations, and relies on random folds rather than organ-stratified or specimen-level splits. It also does not specify a license for the data or code release (Mahbod et al., 2023).

Later evaluation work sharpens some of these caveats. NucEval notes that the vague area annotations are subjective and come from a single annotator, and it recommends repeated annotations from multiple experts to better assess reliability. The same work also shows that border-uncertainty parameters can mechanically inflate scores if set too aggressively; its reported choice is a conservative one-pixel uncertainty ring, with larger values producing steadily higher metrics at the cost of discarding more evaluation area (Mahbod et al., 4 May 2026).

A further interpretive issue is semantic reformulation. Several later papers do not evaluate NuInsSeg strictly in the form intended by the original release. ExplainSeg reduces it to binary segmentation derived from image-level classification, SAM3 benchmarking treats it as single-class foreground segmentation with IoU and Dice rather than AJI or PQ, and PanopMamba recasts it as a panoptic benchmark with tissue treated as background. These are not contradictions in the data release; they are downstream task definitions layered on top of the same annotation corpus.

The name also requires disambiguation. NuInsSeg, the computational pathology dataset described here, is unrelated to nuInsSeg, the autonomous-driving extension of nuScenes introduced for multimodal weakly supervised image instance segmentation. The latter contains LiDAR point clouds, RGB images, 3D and 2D boxes, and manually refined instance masks for 10 traffic-scene classes, whereas the former concerns H&E histology and nuclei instance segmentation (Li et al., 2022).

In its original form, NuInsSeg is best understood as a large, fully manually delineated, multi-organ, cross-species H&E nuclei instance segmentation dataset whose most distinctive contribution is the explicit annotation of ambiguity. That combination of scale, biological breadth, manual completeness, auxiliary supervision targets, and dataset-wide vague masks explains why it has become a reference point not only for model benchmarking but also for evaluating what “ground truth” means in nuclear segmentation research.