---
title: Mask Clustering-based Annotation Engine (MCAE)
url: https://www.emergentmind.com/topics/mask-clustering-based-annotation-engine-mcae
type: topic
---

# Mask Clustering-based Annotation Engine (MCAE)

Mask Clustering-based Annotation Engine (MCAE) is an annotation and data-engineering framework for large-scale submeter land-cover mapping that treats semantically consistent mask groups as the minimal annotating units, allowing a single annotation action to label multiple spatially coherent instances simultaneously [2509.24374]. It was proposed for the regime of \(0.3\)–\(0.6\) m remote-sensing imagery, where high-resolution detail makes dense pixel-wise supervision prohibitively expensive. MCAE is motivated by spatial autocorrelation—the principle that nearby objects in the same local region tend to belong to the same class and often share similar visual characteristics—and combines multi-scale mask generation, mask-level feature learning, hierarchical mask clustering, and iterative test-set curation. The framework was used to build HiCity-LC, a dataset of about 14 billion labeled pixels spanning five major Chinese cities, and to support city-scale land-cover maps with classification accuracies above 85% [2509.24374].

## 1. Problem setting and conceptual basis

MCAE addresses a bottleneck specific to submeter land-cover mapping: high-resolution imagery contains extremely detailed, dense, and heterogeneous objects, but obtaining high-quality annotations at that scale is prohibitively expensive [2509.24374]. At \(0.3\)–\(0.6\) m imagery, annotators must handle many small objects, complex boundaries, high intra-class variation, inter-class ambiguity, and a small minimum mapping unit. The paper identifies narrow roads, tiny buildings, paths, roadside trees, greenbelts, and field boundaries as representative examples of the annotation burden [2509.24374].

The framework is explicitly positioned against two prevalent alternatives. Coarse-label transfer or weak supervision from \(10\) m products is described as noisy and often misaligned, while manual pixel-wise or mask-wise annotation is accurate but too slow and costly for city-scale coverage [2509.24374]. MCAE therefore changes the unit of supervision from the single pixel or single object to the semantically consistent mask group.

Its key prior is spatial autocorrelation. The paper states that nearby objects in the same local region tend to belong to the same class and often share similar visual characteristics, with examples including houses in the same village sharing roof styles and cotton, orchard, or aquaculture parcels appearing in spatial clusters due to planning and land-use organization [2509.24374]. This motivates the central operational principle: annotate semantically consistent object clusters, not individual objects.

A semantically consistent mask group is defined operationally as a set of object masks that mostly belong to the same semantic class, lie in a local spatial neighborhood, share similar mask-level features, and have high cluster purity [2509.24374]. Cluster purity is defined as the proportion of masks in a cluster that share the dominant semantic class. Only clusters above a purity threshold are retained for unified annotation [2509.24374].

## 2. Four-stage workflow

MCAE is organized as a four-stage pipeline: multi-scale mask generation, mask-level feature learning, hierarchical mask clustering, and iterative test-set curation [2509.24374].

The first stage, multi-scale mask generation, targets the large object-size variation characteristic of submeter imagery. The original imagery is at \(0.3\) m resolution, and masks are generated at both \(0.3\) m and \(0.6\) m scales before fusion [2509.24374]. The rationale is that off-the-shelf segmentation models such as SAM are sensitive to scale: at coarse scale, small objects may be missed or only partially segmented, whereas at fine scale, large objects may be fragmented or mis-segmented [2509.24374].

The second stage, mask-level feature learning, is introduced because raw pixel features or pre-trained features do not explicitly encode object-level semantics well enough for high-purity clustering [2509.24374]. A self-supervised mask-level feature learner is trained so that masks of the same object or class become closer in feature space and different masks become more separable [2509.24374].

The third stage, hierarchical mask clustering, is the mechanism that converts mask proposals and embeddings into annotatable units. The stated objective is to form groups that are semantically homogeneous, spatially coherent, large enough to reduce annotation effort, and not so large that purity is lost [2509.24374]. This stage uses DBSCAN, a choice justified by the fact that it does not require specifying the number of clusters, is robust to noise, and handles irregular spatial and feature distributions well [2509.24374].

The fourth stage, iterative test-set curation, addresses a distinct requirement: sparse cluster annotations are suitable for training, but evaluation requires dense and reliable labels [2509.24374]. MCAE therefore constructs dense and representative test regions through repeated sampling, prediction, and manual refinement rather than exhaustive dense annotation of entire cities.

Taken together, these four stages define MCAE as a data engine rather than a single segmentation model. A plausible implication is that its principal contribution lies in annotation system design and dataset construction, not in proposing a new end-task classifier alone.

## 3. Multi-scale mask generation and mask-level representation learning

In the \(0.3\) m branch, each image is split into overlapping tiles with 50% overlap, each tile is processed independently by SAM (ViT-L), and the overlapping predictions are reconciled conservatively: if overlapping tiles produce consistent masks in the overlap area, the masks are kept; if inconsistent, both conflicting masks are discarded [2509.24374]. This yields a refined set of fine-scale masks denoted \(M^{L'}\) [2509.24374].

In the \(0.6\) m branch, the image is downsampled to \(0.6\) m and SAM generates a coarser set of masks denoted \(M^L\) [2509.24374]. The two sets are then fused by prioritizing finer masks to preserve purity. Non-overlapping masks are retained directly; if a fine mask is fully inside a coarse mask, the coarse mask is split; and if overlap is partial, a three-part decomposition is used consisting of the overlapping region, the non-overlapping part of the fine mask, and the non-overlapping part of the coarse mask [2509.24374]. The stated design rationale is to balance boundary precision from \(0.3\) m masks with contextual completeness from \(0.6\) m masks [2509.24374].

The feature-learning stage is designed to produce discriminative mask representations for downstream clustering. Two overlapping crops are sampled from an input tile, a Swin Transformer backbone extracts feature maps, features for masks appearing in the overlap region are compared, and a similarity matrix is built [2509.24374]. The learning objective is described as a self-supervised contrastive relationship over mask-level embeddings, encouraging the same mask or same semantic object to have similar features and different masks to remain distinct [2509.24374].

The paper attributes three downstream benefits to this stage: improved intra-class compactness, improved inter-class separability, and improved cluster purity [2509.24374]. The reported ablation states that without mask-level self-supervised learning, the number of valid clusters and masks drops sharply because the system must filter more noisy and impure clusters [2509.24374]. This suggests that MCAE depends not only on clustering mechanics but also on representation quality at the mask level.

## 4. Hierarchical clustering, semantic consistency, and annotation mechanics

Hierarchical mask clustering begins with semantic consistency filtering. MCAE uses a semantic segmentation model, UPerNet-Swin-Small pretrained on OpenEarthMap, to produce pixel-wise predictions [2509.24374]. Each mask is then assigned a class by majority vote over the pixels inside that mask, cluster purity is computed from these mask labels, clusters with sufficiently high purity are retained, and the dominant class acts as a reference label for the annotator [2509.24374].

The clustering itself is performed hierarchically in two neighborhood regimes. Small-neighborhood clustering uses a \(3\times 3\) tile window and is intended to find dense, highly coherent clusters; it tends to be purer but may miss sparse objects [2509.24374]. Large-neighborhood clustering is then applied to masks not covered in the first stage using a \(5\times 5\) tile window; this stage captures spatially dispersed but semantically related masks and improves completeness [2509.24374]. The paper frames this design as a response to a tradeoff: larger neighborhoods improve completeness but increase the risk of impurity, whereas smaller neighborhoods improve purity but may miss sparse instances [2509.24374].

Once clusters are formed, the annotator labels the cluster as a whole, and that single annotation operation labels all instances in the cluster [2509.24374]. The reported average is 67 masks per cluster, so MCAE’s estimated cost is about
\[
\frac{1}{67}
\]
per object relative to one object per action [2509.24374]. The paper accordingly claims annotation-efficiency gains of one to two orders of magnitude [2509.24374].

The quality-preservation strategy is distributed across the pipeline rather than concentrated in a single verification step. The paper lists five mechanisms: multi-scale mask fusion, self-supervised mask features, purity filtering, reference class assignment via majority voting, and the hierarchical neighborhood strategy [2509.24374]. These mechanisms are intended to preserve label quality, semantic diversity, and spatial representativeness while reducing the number of manual actions [2509.24374].

## 5. Iterative test-set curation, HiCity-LC, and evaluation protocol

MCAE distinguishes between sparse training annotation and dense evaluation annotation. For test-set construction, the workflow is: pool each tile’s feature map into a compact vector; use SKATER to partition the large region based on feature similarity with spatial adjacency constraints; uniformly sample a fixed number of tiles from each partition; use a semantic segmentation model trained on sparse annotations to generate initial predictions; overlay predictions with object masks; manually refine or correct them to obtain dense labels; and repeat this process across multiple rounds so that dense labels from one round help retrain the model and improve the next round of sampling and refinement [2509.24374]. The stated purpose is to ensure that the dense test set is spatially representative, semantically diverse, densely annotated, and cost-effective to build [2509.24374].

The resulting dataset, HiCity-LC, is described as the first publicly available submeter city-level land-cover benchmark [2509.24374]. It has \(0.6\) m resolution, image size \(1024 \times 1024\), 55,887 image-label pairs, and about 14 billion labeled pixels [2509.24374]. Of these, 5,116 image-label pairs are densely annotated and 50,771 are sparsely annotated, corresponding to about 3.74 billion dense pixels and about 10.27 billion sparse pixels [2509.24374]. The five cities are Beijing, Tianjin, Shanghai, Chengdu, and Guangzhou [2509.24374].

HiCity-LC uses the OpenEarthMap taxonomy with 8 classes: bareland, rangeland, developed space, road, tree, water, agricultural land, and building [2509.24374]. The paper states that the imagery is subdivided by administrative units, small districts and counties are excluded, sparse cluster annotations are created first, and dense annotations for test regions are then built through iterative sampling and refinement [2509.24374].

The benchmark reports four evaluation metrics:
\[
\text{OA} = \frac{\sum_{i=1}^{K} TP_i}{\sum_{i=1}^{K} (TP_i + FP_i + FN_i + TN_i)}
\]
\[
\text{mF1} = \frac{1}{K} \sum_{i=1}^{K} \text{F1}_i,\quad \text{F1}_i = \frac{2TP_i}{2TP_i + FP_i + FN_i}
\]
\[
\text{mIoU} = \frac{1}{K} \sum_{i=1}^{K} \text{IoU}_i,\quad \text{IoU}_i = \frac{TP_i}{TP_i + FP_i + FN_i}
\]
\[
\text{UA}_i = \frac{TP_i}{TP_i + FP_i}
\]
[2509.24374]

In scale, the paper claims that HiCity-LC contains about 8× more images than GID and Five-Billion-Pixels, about 3× more labeled pixels than those datasets, nearly 3× the annotation volume of OpenEarthMap, and broader geographic coverage [2509.24374].

## 6. Reported performance, ablations, and relation to adjacent research

Under the same annotation budget, MCAE is reported to outperform both pixel-based annotation and mask-based annotation baselines [2509.24374]. The paper gives overall accuracies of 92.38% for Beijing, 87.80% for Tianjin, 85.12% for Shanghai, 91.93% for Chengdu, and 89.23% for Guangzhou, all above 85% [2509.24374]. It attributes this advantage to producing many more labeled objects, better spatial sampling, more semantic diversity, and more representative training data under fixed human effort [2509.24374].

A central empirical finding is that sparse annotations alone already work well. When models are trained with sparse annotations only, performance is close to that of training with sparse plus dense annotations, with only modest gains from adding dense labels [2509.24374]. The paper interprets this as evidence that MCAE’s sparse labels are spatially representative, semantically diverse, and sufficiently informative for training strong segmentation models [2509.24374].

The comparison with SinoLC-1 is presented as a comparison against coarse-resolution source labels and weak supervision. Reported overall accuracies for MCAE versus SinoLC-1 are 92.21 vs 62.77 for Beijing, 91.79 vs 61.65 for Tianjin, 89.04 vs 44.68 for Shanghai, 91.38 vs 64.09 for Chengdu, and 93.78 vs 69.19 for Guangzhou [2509.24374]. The paper also states that MCAE substantially improves IoU, UA, and F1 across classes [2509.24374].

The ablation study identifies multi-scale mask generation and mask-level self-supervised learning as critical. In Shanghai-Fengxian, mean IoU improves from 50.05 / 53.91 to 66.49 and OA improves to 87.80 when multi-scale fusion is used [2509.24374]. The paper further states that \(0.3\) m masks are better for small structures such as roads and buildings, \(0.6\) m masks are better for large homogeneous regions such as trees and water, and multi-scale fusion is best overall [2509.24374].

In a broader research context, MCAE belongs to a family of methods that replace instance-wise supervision with cluster-level operations. "Scaling up instance annotation via label propagation" uses class-specific hierarchical agglomerative clustering over predicted masks, asks humans to verify only a few masks per cluster, and propagates the result to the whole cluster; on 1M unlabeled Places images over 80 classes, it reports 993,677 high-quality masks with 290 hours of total annotation time and a 76× speedup over manual annotation [2110.02277]. MCAE differs in target domain and clustering prior: its emphasis is submeter land-cover mapping, spatial autocorrelation, semantically consistent local mask groups, and city-scale geospatial representativeness [2509.24374].

The term should also be distinguished from works whose acronymic or conceptual overlap is only partial. "Joint-Embedding Predictive Architecture for Self-Supervised Learning of Mask Classification Architecture" is a self-supervised pretraining framework for mask classification architectures such as Mask2Former, focusing on pixel decoders and transformer decoders rather than cluster-level annotation [2407.10733]. "Masked AutoEncoder for Graph Clustering without Pre-defined Cluster Number \(k\)" addresses nonparametric graph clustering through masked autoencoding and an improved density-based clustering decoder, but it is an unsupervised graph clustering framework rather than an annotation engine [2401.04741]. These neighboring lines of work help locate MCAE within a larger methodological landscape, but the defining property of MCAE remains the use of semantically consistent mask groups as the atomic annotation unit for large-scale submeter remote-sensing data [2509.24374].

Source: https://www.emergentmind.com/topics/mask-clustering-based-annotation-engine-mcae