UniEM-3M: Multimodal EM Dataset & Benchmark

• UniEM-3M is a multimodal electron micrograph dataset featuring over 5,000 high-resolution images and nearly 3M instance segmentation labels for detailed microstructural analysis.
• It incorporates a text-conditioned diffusion model that generates synthetic EM images for data augmentation and privacy-preserving access to complete data distributions.
• The framework introduces UniEM-Net, a flow-based instance segmentation benchmark optimized for handling high-density microstructures and diverse morphological features.

Searching arXiv for the current UniEM-3M paper and related context. UniEM-3M is a large-scale, multimodal electron micrograph dataset and accompanying model suite for automated microstructural analysis, with a primary emphasis on instance segmentation and text-conditioned generation in materials science. It is introduced as the first large-scale and multimodal EM dataset for instance-level understanding, comprising 5,091 high-resolution EM images, approximately 2,985,660 instance segmentation labels, and image-level attribute-disentangled textual descriptions. A subset is publicly released, while a text-to-image diffusion model trained on the full collection is provided as a proxy for the complete data distribution under privacy and licensing constraints (Wang et al., 22 Aug 2025).

1. Motivation and research context

Quantitative microstructural characterization from electron micrographs is fundamental to materials science because microstructure controls properties such as strength, toughness, transport, and degradation. Electron microscopy, primarily SEM and TEM, is routinely used for grain size and shape measurements, particle size and number density distributions, defect and phase topology, agglomeration, and spatial ordering. In practice, however, this analysis remains largely manual, relying on tools such as ImageJ or proprietary software and on expert judgment, which does not scale well across large datasets or across laboratories and material systems (Wang et al., 22 Aug 2025).

The dataset was developed against a background in which prior EM datasets were constrained by scale, annotation granularity, modality coverage, or semantic richness. The data block identifies EMPS as containing 465 EMs with instance masks and mainly simple particle-type microstructures; Aversa et al. (2018) as providing approximately 22,000 SEM images but only image-level labels; and Stuckner et al. (2022) as offering more than 100,000 EMs for pretraining without instance-level annotations. Other segmentation datasets are described as containing 26–237 images with instance masks, often restricted to a single material or a small number of scenes. MMSci is noted as a large EM collection with captions, but with publication-post-processed imagery and noisy or incomplete captions for quantitative analysis. MicroVQA and MicroBench include microscopy data, but focus on biological or medical images rather than materials microstructures (Wang et al., 22 Aug 2025).

Within this context, UniEM-3M is explicitly designed to be large-scale and multimodal, instance-level, attribute-disentangled, and cross-domain. This suggests that its central research function is not merely dataset enlargement, but the construction of a benchmark in which segmentation, multimodal conditioning, and synthetic data generation are all tied to the same microstructural representation.

2. Dataset composition and annotation schema

UniEM-3M consists of 5,091 high-resolution EM images, with resolutions ranging from below $500 \times 500$ to above $8000 \times 6000$ pixels, and approximately 2,985,660 instance segmentation labels. The dataset split is 4,128 training images and 963 test images. The dominant modality is SEM, representing more than 90% of the images; additional modalities include TEM and a small fraction labeled as other microscopy, such as STEM and OM. The data are described as spanning different instruments and operating conditions from academic and industrial sources (Wang et al., 22 Aug 2025).

An instance is defined operationally as an individual microstructural object visible in the electron micrograph, typically including particles, grains, agglomerates, fibrous structures, porous regions with discrete pores or inclusion-like features, and complex morphologies such as plates, rods, and nanowires. The annotation target is geometry-driven segmentation rather than semantic class segmentation. The labels therefore correspond primarily to foreground microstructures rather than to a taxonomy such as grain versus pore versus precipitate. This representation aligns with quantitative tasks such as counting, size estimation, shape characterization, density estimation, and distribution analysis (Wang et al., 22 Aug 2025).

The dataset is notable for its density and scale heterogeneity. Instance counts per image span four orders of magnitude; many images contain hundreds of instances, a non-negligible fraction contain more than 1,000 instances, and some examples exceed 2,000–5,000 annotated objects in a single $2000 \times 2000$ micrograph. Instance areas range from small segments of about 10 pixels to particles larger than $10^5$ pixels. The paper emphasizes that this extreme density and scale variation is central to the challenge posed by the benchmark (Wang et al., 22 Aug 2025).

Each image is paired with a structured textual description organized into nine attributes: subject, microscopy type, color profile, morphology, particle density, distribution, layering, surface texture, and pixel-size profile. The nested feature-analysis fields specify categories such as spherical, rod-shaped, angular polyhedral, fibrous, irregular polygonal, plate-like, or nanowire morphologies; qualitative density levels from single object to high density; spatial arrangements including uniform, periodic, densely packed, isolated and scattered, agglomerated into clusters, or interconnected network; layering as tiled or multilayer; surface texture including smooth, porous and web-like, crystalline facets, or nanostructured; and pixel-level size profiles such as uniform, wide range of sizes, or mostly uniform with occasional large outliers. Representative attribute distributions are reported across inorganic and organic systems, irregular aggregates and well-defined shapes, sparse to densely overlapped scenes, random to periodic or networked arrangements, and both single-plane and multilayer micrographs (Wang et al., 22 Aug 2025).

3. Data sourcing, curation, and human-in-the-loop annotation

The data are aggregated from three channels: academic collaborations, automated web crawling, and existing public datasets. Approximately 500 challenging images are selected and re-annotated from the Aversa et al. 2018 SEM dataset. All raw images undergo perceptual hash deduplication and quality filtering, including removal of low-resolution images below 200 pixels and GPT-based screening of severely blurred images or images with illegible embedded text (Wang et al., 22 Aug 2025).

Because EM images often contain timestamps, scale bars, manufacturer logos, and other metadata overlays, the authors train a YOLO11 detector to localize text-containing regions and crop them out for generative-model training. The stated rationale is to prevent the generative model from learning spurious associations between text style and microstructure type. The resulting sanitized micrographs focus on the microstructure itself (Wang et al., 22 Aug 2025).

Instance segmentation annotation is performed through a custom web-based EM annotation platform developed to address the limitations of general tools such as LabelMe and Label Studio in settings with thousands of instances per image. The platform supports efficient rendering for tens of thousands of polygonal annotations, smooth pan and zoom, and a magnetic-lasso tool with real-time Sobel edge snapping. The adjustable sensitivity is intended to support low-contrast and heterogeneous boundaries. Internally, annotations are polygonal masks, while the public data are typically distributed as rasterized masks plus instance identifiers; for model training, masks are converted into foreground probability maps and flow fields in the $x$ and $y$ directions (Wang et al., 22 Aug 2025).

The quality-control workflow is explicitly iterative. Domain experts first annotate a high-quality, low-density subset to establish gold-standard masks. An initial segmentation network is then trained on this seed set; a diffusion model synthesizes additional pseudo-EMs for augmentation; the network generates pseudo-labels on unannotated images; trained annotators refine these labels; each image receives independent peer review by two reviewers with inline comments attached to vertices or instances; and a senior materials-science expert performs the final audit. All edits are version-controlled and traceable. For segmentation evaluation, the benchmark uses IoU, Average Precision, and Panoptic Quality, with $[email protected]$ defined as

$$[email protected] = \frac{1}{N} \sum_{i=1}^{N} \text{AP}_i(0.5),$$

and $[email protected]$ reported for both sparse and dense subsets (Wang et al., 22 Aug 2025).

4. Diffusion model, multimodal conditioning, and synthetic-data utility

The generative component of UniEM-3M is built by fine-tuning Stable Diffusion XL, described as a latent diffusion model with a U-Net-like backbone operating in a VAE latent space and standard DDPM-style forward and reverse diffusion processes. Fine-tuning uses LoRA-based adaptation and introduces learnable tokens representing different UniEM-3M categories. Both adapter weights and token embeddings are optimized during training. Conditioning is performed through the SDXL text encoder on structured attribute descriptions derived from the nine-field JSON schema, with additional textual specification of morphology, density, distribution, layering, surface texture, and pixel-size profile through cross-attention (Wang et al., 22 Aug 2025).

The model serves two stated functions. First, it is a data-augmentation tool for segmentation, capable of generating EM-like images conditioned on specific attribute combinations. Second, it functions as a privacy-preserving surrogate for the complete, only partially releasable dataset. Prompts are constructed from structured descriptions and include both typical patterns and rare anomalies to encourage robustness and compositional generalization (Wang et al., 22 Aug 2025).

Quantitative evidence is reported for both distributional fidelity and downstream utility. Against the sanitized real training set, the Fréchet Inception Distance is 28.06 for real data, 213.16 for the untuned SDXL base model, and 34.03 for SDXL fine-tuned on UniEM-3M. Using DINO-ViT CLS tokens and cosine similarity, style similarity is 0.985 for real data, 0.669 for the SDXL base model, and 0.958 for the fine-tuned model. These figures are presented as indicating that the fine-tuned model captures the style and distribution of real EMs much more closely than the generic baseline (Wang et al., 22 Aug 2025).

The segmentation augmentation results are equally central. Training UniEM-Net on real data only yields $[email protected] = 0.767$ and $8000 \times 6000$0 for 1k images, and $8000 \times 6000$1 and $8000 \times 6000$2 for 4k images. Training on generated data only yields $8000 \times 6000$3 and $8000 \times 6000$4 for 5k generated images, and $8000 \times 6000$5 and $8000 \times 6000$6 for 30k generated images. Combining 30k generated images with 4k real images gives the best reported performance, $8000 \times 6000$7 and $8000 \times 6000$8. Pretraining UniEM-Net on 30k generated images and then fine-tuning on real data also outperforms pretraining on SA-1B across all tested real-data sizes. The paper nonetheless notes a residual distribution gap, since the synthetic-data FID remains slightly worse than that of real data, and acknowledges that rare phenomena or subtle instrument artifacts may still be underrepresented (Wang et al., 22 Aug 2025).

5. UniEM-Net and the benchmark for instance segmentation

UniEM-Net is the benchmark’s designated baseline model and is characterized as a flow-based instance segmentation architecture tailored to EM microstructures. It is inspired by Cellpose and is intended to handle extremely high instance density, purely geometric structures, and multiscale variation with computational efficiency in crowded scenes. The representation is based on directional flow fields from instance boundaries toward instance centers. For each training instance, the supervision targets are an $8000 \times 6000$9-gradient field $2000 \times 2000$0, a $2000 \times 2000$1-gradient field $2000 \times 2000$2, and a foreground probability map $2000 \times 2000$3. A modified Euler integration scheme is used to derive these flows from the masks (Wang et al., 22 Aug 2025).

The backbone is SAM-ViT-base. During inference, each pixel is iteratively updated by the predicted flow field: $2000 \times 2000$4 where $2000 \times 2000$5. Pixels that converge to the same point are assigned to the same instance, and the final masks are refined by the foreground probability map. The paper presents this 2D flow-field formulation as especially effective for irregular shapes, overlapping particles, and high-density images in which bounding-box and center-based methods are disadvantaged (Wang et al., 22 Aug 2025).

The benchmark includes anchor-based methods such as Mask R-CNN, Cascade R-CNN, and HTC; anchor-free methods such as YOLACT, Mask2Former, StarDist, and CPP-Net; and field-based methods such as CellViT, Cellpose-SAM, and UniEM-Net. Images are cropped or resized to $2000 \times 2000$6 for training. Anchor-based methods are trained and evaluated only on the sparse subset, defined as images with fewer than 100 instances, because of computational limits in dense scenes. Field-based methods are evaluated on both sparse and dense subsets. All models are fine-tuned for approximately 72 epochs, while the UniEM-Net baseline is trained for 180k iterations on 4×4090 GPUs (Wang et al., 22 Aug 2025).

The reported results show a marked separation between method families. On the sparse subset, UniEM-Net achieves $2000 \times 2000$7 and $2000 \times 2000$8, compared with 0.605 and 0.633 for Cellpose-SAM, 0.589 and 0.573 for CellViT, 0.542 and 0.565 for Mask R-CNN, and 0.204 and 0.064 for Mask2Former. On the dense subset, UniEM-Net achieves $2000 \times 2000$9 and $10^5$0, compared with 0.760 and 0.700 for Cellpose-SAM, 0.701 and 0.604 for CellViT, 0.487 and 0.438 for StarDist, and 0.378 and 0.352 for CPP-Net. UniEM-Net also uses fewer parameters, 93M, than CellViT at 146M and Cellpose-SAM at 304M. The accompanying discussion states that anchor-based detectors are reasonable on sparse images but impractical on dense scenes because of proposal and NMS limitations, while anchor-free and polygon-based methods often over-segment or miss instances, and Mask2Former is sensitive to background texture and produces many false positives (Wang et al., 22 Aug 2025).

6. Scientific uses, public release, and acknowledged limitations

The combined dataset and model suite support automated segmentation of particles, grains, pores, and related microstructural entities, followed by computation of size distributions, equivalent diameters, number density, volume fraction, spatial distributions, and morphological descriptors such as aspect ratio and roundness. The intended applications include materials characterization workflows, process monitoring, comparative studies across processing conditions, and broader automated materials analysis (Wang et al., 22 Aug 2025).

The generative model extends this functionality to attribute-guided synthesis. A structured description containing subject, microscopy type, morphology, density, distribution, layering, texture, and size profile can be used to generate EM-like micrographs consistent with those attributes. The paper also highlights out-of-distribution recombination of attributes, such as a high-density fibrous network with wide size distribution in a multilayered arrangement, as a means of exploring hypothetical microstructures. A plausible implication is that UniEM-3M can operate simultaneously as a benchmarking corpus and as a controlled microstructure prior for synthetic experimentation (Wang et al., 22 Aug 2025).

The public release strategy is deliberately partial. A subset of the dataset is available at Hugging Face under https://huggingface.co/datasets/NNNan/UniEM-3M, and the fine-tuned SDXL model and UniEM-Net baseline are also released. At the same time, privacy-sensitive portions of the full corpus remain unreleased, with the diffusion model positioned as a practical substitute for access to the complete data distribution (Wang et al., 22 Aug 2025).

The paper explicitly notes several limitations. The dataset is dominated by SEM, with TEM and other modalities less represented. Some domains, including complex multiphase alloys and biological EM, may be underrepresented. Extremely dense and low-contrast scenes remain difficult to annotate perfectly even with expert review. The labels focus on geometry-based instances rather than systematic semantic distinctions such as pores versus grains versus precipitates. Finally, the generative model retains a measurable distribution gap and may miss rare phenomena or encode subtle biases, so synthetic data should be used cautiously in precise quantitative studies (Wang et al., 22 Aug 2025).

7. Nomenclature and disambiguation

Within materials and scientific machine learning, “UniEM-3M” refers to the electron micrograph dataset, diffusion model, and benchmarking framework just described (Wang et al., 22 Aug 2025). In a separate and unrelated mathematical context, the expression “UniEM-3M” is used in one exposition as shorthand for “the universal elliptic modular surface picture for modular forms of weight $10^5$1,” summarizing Shouhei Ma’s identification of $10^5$2 with a log canonical ring of the corresponding elliptic modular surface (Ma, 2015).

That mathematical usage concerns a graded-ring isomorphism

$10^5$3

together with the canonical map $10^5$4 on the elliptic modular surface (Ma, 2015). The coexistence of these two usages is purely terminological. In current materials-science discourse, UniEM-3M denotes the multimodal EM dataset and associated generative and segmentation models, whereas the modular-forms usage is a mathematical interpretive label rather than a dataset name.