Mask to Concept (M2C): A Cross-Domain Paradigm

Updated 6 July 2026

Mask to Concept (M2C) is a design paradigm where masking techniques isolate parts of the input to reveal semantically rich representations.
It spans applications in medical interpretation, segmentation, visual pre-training, language modeling, and cross-modal tasks using diverse operational strategies.
The approach navigates trade-offs in mask dependency, reconstruction, and supervision, offering robust frameworks for both concept discovery and evaluation.

Mask to Concept (M2C) denotes a family of formulations in which masking is used to expose, retrieve, align, or score semantic concepts. The term does not identify a single canonical algorithm. Across the cited literature, it appears in medical neuron interpretation, semantic segmentation, self-supervised visual pre-training, masked language modeling, dialogue modeling, visio-linguistic compositionality, few-shot medical annotation with SAM3, and evaluation of concept-driven text-to-image generation (Kim et al., 2024, Wang et al., 2018, 2411.09858, Lee et al., 2022, Pandey et al., 2020, Li et al., 11 Jun 2026, Zhou et al., 25 Jun 2026, Bartkowiak et al., 21 May 2026). In all of these settings, a partial observation—whether a binary mask, a masked subset of patches, or a masked token span—is converted into a semantically meaningful concept representation that can support interpretation, prediction, reconstruction, annotation, or evaluation.

1. Terminological scope and historical emergence

The cited record shows that M2C has developed as a cross-domain design pattern rather than a standardized task. The earliest cited work, "Concept Mask: Large-Scale Segmentation from Semantic Concepts" (Wang et al., 2018), formulates segmentation as image segmentation given a semantic concept. Later works use masking to discover dialogue concepts (Pandey et al., 2020), to schedule whole-concept masking in masked language modeling (Lee et al., 2022), to replace mask-dependent neuron annotation in medical CNNs (Kim et al., 2024), to learn concept-level visual representations from masked intra-image views (2411.09858), to build multi-layer concept tokens from masked images (Sun et al., 1 Feb 2025), and to learn or evaluate concept embeddings in cross-modal generation and segmentation systems (Li et al., 11 Jun 2026, Zhou et al., 25 Jun 2026, Bartkowiak et al., 21 May 2026). The surveyed papers indicate that the term is best understood as a family resemblance: masking is the mechanism, while the output concept may be a label, an embedding, a token bank, a curriculum unit, or a similarity score.

Setting	Mask-to-concept operation	Representative work
Medical interpretation	Neuron activations or neuron exemplars mapped to semantic concepts	MAMMI (Kim et al., 2024)
Large-scale segmentation	Region masks or attention maps used to score concepts	Concept Mask (Wang et al., 2018)
Self-supervised vision	Masked patch views mapped to concept-level representations	MiCL (2411.09858)
Masked image concept learning	Masked images mapped to multi-layer concept tokens	MCM (Sun et al., 1 Feb 2025)
Language and dialogue	Masked spans mapped to KG concepts or discrete dialogue concepts	CCM (Lee et al., 2022); Mask & Focus (Pandey et al., 2020)
Vision-language compositionality	Masked attributes and relations reconstructed as compositional concepts	MACCO (Li et al., 11 Jun 2026)
Annotation and evaluation	Masks define reusable concept prompts or concept-specific scores	SAM3 M2C (Zhou et al., 25 Jun 2026); MaSC (Bartkowiak et al., 21 May 2026)

2. Spatial grounding and concept attribution

A central M2C lineage starts from spatial structure. In traditional neuron concept annotation, a neuron’s activation map is converted into a spatial mask and matched to concept masks by IoU; Network Dissection uses maximal IoU assignment, and TSI adapts this logic to medical data. "Mask-Free Neuron Concept Annotation for Interpreting Neural Networks in Medical Domain" replaces this mask dependency with a vision-language pipeline. MAMMI constructs a medical concept set by extracting clinical nouns from MIMIC-CXR reports, selects representative images for each neuron through a neuron-specific adaptive threshold,

$\tau_{\text{example}}^{(l,i)}=\max(A^{(l,i)})-(1-\alpha)\cdot(\max(A^{(l,i)})-\min(A^{(l,i)})),$

and scores concepts by template-normalized cosine similarity,

$s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$

In the reported NIH ChestX-ray14 experiments, MIMIC_Nouns with 1361 medical concepts yields CLIP $0.8599 \pm 0.024$ , mpnet $0.5290 \pm 0.084$ , F1 $0.2619 \pm 0.113$ , and hit-rate $0.2857 \pm 0.121$ , whereas 20K generic CV concepts and 80K WordNet nouns both give F1 $0.000$ and hit-rate $0.000$. Against mask-dependent TSI, MAMMI reports CLIP $0.8599 \pm 0.024$ versus $0.7872 \pm 0.014$ , mpnet $s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$ 0 versus $s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$ 1, and nonzero F1 and hit-rate where TSI reports zero on both metrics (Kim et al., 2024).

The same spatial logic appears in "Concept Mask: Large-Scale Segmentation from Semantic Concepts" through a three-stage concept-to-mask framework: a visual-semantic embedding trained on a stock image dataset with only image-level labels for 18K concepts, an attention-refinement stage on OIVG-750 with bounding boxes, and an attention-driven class-agnostic segmentation network trained on COCO-80. Its dense embeddings and attention maps can be adapted to M2C by scoring a region mask $s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$ 2 with quantities such as $s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$ 3, $s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$ 4, and $s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$ 5. The underlying segmentation system reports Pointing Game accuracy $s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$ 6 and IoU $s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$ 7 on OIVG-750 for the full refined model, and zero-shot IoU $s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$ 8 on 10 unseen concepts, indicating that spatially grounded concept scoring can extend beyond fully supervised categories (Wang et al., 2018).

These works separate two variants of spatial M2C. One begins with model internals and seeks a human-readable concept for a neuron; the other begins with a region and seeks the best-matching concept in a large vocabulary. MAMMI shows that external segmentation masks are not a necessary condition for concept annotation, whereas Concept Mask shows that large-vocabulary concept retrieval can remain spatially conditioned even when categories extend to objects, parts, stuff, and attributes.

3. Masked views as concept-learning signals

A second lineage uses masks not as external supervision but as a view generator for representation learning. "Masked Image Contrastive Learning for Efficient Visual Conceptual Pre-training" defines masked intra-image views by randomly masking patches, splitting the remaining visible tokens into two non-overlapping groups $s_j^{(l,i)}=\frac{1}{n}\sum_{o=1}^{n}\{\cos(v_o^{(l,i)}, t_j)-\cos(v_o^{(l,i)}, t_{tem})\}, \qquad \theta_{\text{concept}}=\beta\cdot \max_j s_j^{(l,i)}.$ 9 and $0.8599 \pm 0.024$ 0, encoding each group with a shared ViT, and using the $0.8599 \pm 0.024$ 1 tokens directly as concept representations. Its contrastive objective uses a T-distributed spherical similarity,

$0.8599 \pm 0.024$ 2

with symmetric InfoNCE-style training,

$0.8599 \pm 0.024$ 3

MiCL omits reconstruction, hand-crafted augmentations, momentum encoders, memory queues, and MLP projection heads. On ImageNet-1K, ViT-L/16 completes pre-training in 133 hours on 4 A100 GPUs, achieves 85.8% fine-tuning top-1 and 77.9% linear-probe top-1, and compares favorably with MAE, CAE, BEiT, I-JEPA, MoCo v3, and DINO under the reported settings (2411.09858).

"Multi-layer Concept Map for Efficient Concept Learning from Masked Images" takes a different path. MCM uses random masking, but instead of contrasting masked views, it learns a hierarchy of concept tokens $0.8599 \pm 0.024$ 4 at multiple encoder depths. Visible tokens are updated by self-attention, concept tokens are updated only by cross-attending to visible tokens, and the decoder reconstructs masked patches by cross-attending to concept tokens selected from layer buckets $0.8599 \pm 0.024$ 5. Its objective is

$0.8599 \pm 0.024$ 6

where $0.8599 \pm 0.024$ 7 is masked reconstruction, $0.8599 \pm 0.024$ 8 is an inverse-frequency weighted MSE to CLIP-derived concept prototypes, and $0.8599 \pm 0.024$ 9 is an antonym-swap cycle-consistency loss. For the base model, the best reported mask ratio is $0.5290 \pm 0.084$ 0, giving accuracy $0.5290 \pm 0.084$ 1, F1 $0.5290 \pm 0.084$ 2, and FID $0.5290 \pm 0.084$ 3; adding both weighted concept loss and disentanglement yields accuracy $0.5290 \pm 0.084$ 4, F1 $0.5290 \pm 0.084$ 5, and FID $0.5290 \pm 0.084$ 6. Against a base-sized MAE with a linear concept head, MCM reports accuracy $0.5290 \pm 0.084$ 7 versus $0.5290 \pm 0.084$ 8 and F1 $0.5290 \pm 0.084$ 9 versus $0.2619 \pm 0.113$ 0, while MAE retains lower FID $0.2619 \pm 0.113$ 1 (Sun et al., 1 Feb 2025).

These two systems clarify a key distinction. MiCL treats masked views as a route to concept-level invariances without reconstruction. MCM treats masked images as a route to explicit, controllable concept tokens that guide reconstruction and editing. A common misconception is that concept learning from masked visual inputs necessarily requires pixel reconstruction; MiCL explicitly rejects that premise, while MCM makes reconstruction the primary supervision signal.

In language modeling, M2C becomes a problem of deciding which masked units correspond to concepts rather than isolated tokens. "Efficient Pre-training of Masked LLM via Concept-based Curriculum Masking" uses ConceptNet 5.5 to map tokens and multiword spans to KG concepts, applies whole concept masking to all WordPiece tokens of a selected concept, and constructs a curriculum $0.2619 \pm 0.113$ 2 by hop-based expansion, $0.2619 \pm 0.113$ 3, with $0.2619 \pm 0.113$ 4 performing best in the reported ablations. The initial concept set is the top- $0.2619 \pm 0.113$ 5 concepts ranked by graph degree among those with corpus frequency at least 100k, and $0.2619 \pm 0.113$ 6 performs best in the reported setting. With BERT Base, CCM improves average GLUE dev score from 80.4 to 82.3 after 1M steps, and the paper reports that it reaches baseline GLUE performance at about half the computational cost. Reverse curriculum yields a marked drop to 66.7 on the reported Medium-model slice, underscoring the importance of easy-to-hard ordering (Lee et al., 2022).

In dialogue, "Mask & Focus: Conversation Modelling by Learning Concepts" uses masking as a probing instrument. A pretrained HRED computes $0.2619 \pm 0.113$ 7, and tokens whose PMI-based scores exceed a threshold enter a concept bank. At inference, context concepts $0.2619 \pm 0.113$ 8 are the words in the context that intersect this bank. Response concepts $0.2619 \pm 0.113$ 9 are then predicted by an intermediate concept decoder, and the final response is generated with copy attention conditioned on both $0.2857 \pm 0.121$ 0 and $0.2857 \pm 0.121$ 1. On Ubuntu Dialogue Corpus v2.0, the reported Entity F1 is 7.82 for Mask & Focus, versus 2.22 for HRED, 2.53 for VHRED, and 6.31 for MrRNN-Noun; human evaluation reports relevance 1.79 and generic-response rate 0.18, versus 1.25 and 0.53 for HRED (Pandey et al., 2020).

In cross-modal representation learning, "Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality" uses targeted masking of attributes and relations in text and grounded regions in images. MACCO reconstructs the masked modality conditioned on the full unmasked other modality through cross-attention predictors, enriches local tokens by global-to-local semantic injection, and adds Masked-augmented Cross-modal Alignment (MCA) and Masked-augmented Intra-modal Regularization (MIR) on masked and unmasked CLS features. The total objective is $0.2857 \pm 0.121$ 2. With CLIP ViT-B/32, MACCO reports ARO Relation/Attribute/Order scores of 73.1 / 68.5 / 76.0, compared with CLIP at 58.7 / 62.7 / 54.1 and CLIP-FT at 64.3 / 66.2 / 49.1. On SugarCrepe, it reports 77.1 / 79.1 for Relation/Attribute, and on VALSE it reports 75.3 on Relation (Li et al., 11 Jun 2026).

Taken together, these methods show that language-centric M2C can operate at several semantic granularities: KG nodes in MLM, discrete salient words in dialogue, and attribute-relation structures in vision-LLMs. The shared principle is that masking is applied to semantically structured units rather than to arbitrary tokens alone.

5. Annotation and evaluation as M2C pipelines

A more recent usage turns M2C into a test-time concept search mechanism inside a frozen foundation model. "Mask to Concept: Auto-Promptable SAM3 via Efficient Test-Time Concept Embedding Search for Few-Shot Annotation" adapts SAM3 for medical few-shot annotation without external modules, parameter retraining, or manual text engineering. It replaces an ambiguous human-written text prompt with a learnable concept embedding $0.2857 \pm 0.121$ 3 in SAM3’s text-embedding space and optimizes only that vector:

$0.2857 \pm 0.121$ 4

The segmentation loss is $0.2857 \pm 0.121$ 5. The method adds Hybrid Uncertainty Estimation, $0.2857 \pm 0.121$ 6, where $0.2857 \pm 0.121$ 7 averages entropy over pixels with entropy above 0.8, and $0.2857 \pm 0.121$ 8 is one minus the IoU between the concept-predicted mask and the box-prompted mask. The reported threshold is $0.2857 \pm 0.121$ 9. On Kvasir-SEG, M2C reports 76.3±8.0, 80.2±4.0, and 82.8±4.4 Dice for 1-shot, 5-shot, and 10-shot; on ISIC-2017, it reports 75.9±8.4, 79.2±2.5, and 82.1±0.5. To reach 90% Dice on Kvasir-SEG, UENT alone requires 345 manually annotated samples, UCGPI alone 230, and HUE 155 (Zhou et al., 25 Jun 2026).

"MaSC: A Masked Similarity Metric for Evaluating Concept-Driven Generation" uses masks not to learn concepts but to separate concept evidence from background evidence during evaluation. Concept Preservation is measured by masked max-cosine over reference foreground patches,

$0.000$0

while Prompt Following compares a background-only pooled image embedding to a subject-stripped prompt embedding. MaSC uses frozen SigLIP2 SO400M-NaFlex features, SAM3 masks on DreamBench++, and dataset-provided masks on ORIDa. On DreamBench++ human ratings, MaSC reports Krippendorff $0.000$1 for Concept Preservation, outperforming all tested non-LLM baselines and GPT-4V and trailing GPT-4o by $0.000$2. On ORIDa, it reports AUC $0.000$3, and on DreamBench++ Prompt Following it reports $0.000$4, above CLIP-T at 0.327. The paper also reports low sensitivity to the segmentation source, with CP $0.000$5 varying by at most 0.012 and PF $0.000$6 by at most 0.005 across several mask sources on intersected DreamBench++ subsets (Bartkowiak et al., 21 May 2026).

These two systems extend M2C beyond representation learning. In the SAM3 setting, a few labeled masks are used to search for a reusable concept prompt. In MaSC, externally supplied masks instantiate the concept boundary needed to evaluate identity fidelity and prompt adherence separately.

6. Recurring design axes, trade-offs, and open issues

The surveyed papers suggest four recurring design axes. The first is the role of the mask. In medical neuron interpretation and MaSC, masks delimit where concept evidence is read out; in MiCL and MCM, masks define partial visual observations; in CCM and Mask & Focus, masking probes or schedules discrete semantic units; in MACCO, masking targets compositional anchors such as attributes and relations; in SAM3 M2C, masks bootstrap a learnable reusable concept embedding (Kim et al., 2024, 2411.09858, Lee et al., 2022, Li et al., 11 Jun 2026, Zhou et al., 25 Jun 2026, Bartkowiak et al., 21 May 2026). The second axis is the form of the concept itself: a neuron label, a dense embedding, a KG node, a discrete word set, a multi-layer token bank, or an evaluation score.

The third axis is supervision. Mask-dependent concept annotation requires pixel-wise masks or at least box-level supervision, which MAMMI explicitly presents as costly in medical imaging; Concept Mask mixes image-level tags, boxes, and masks; CCM depends on ConceptNet and exact string matching; Mask & Focus depends on a pretrained generator for PMI probing; MACCO depends on scene-graph parsing and visual grounding; SAM3 M2C requires only a few labeled images but still needs seed masks; MaSC requires externally supplied foreground masks (Kim et al., 2024, Wang et al., 2018, Lee et al., 2022, Pandey et al., 2020, Li et al., 11 Jun 2026, Zhou et al., 25 Jun 2026, Bartkowiak et al., 21 May 2026). A plausible implication is that M2C methods should be compared not only by downstream metric, but also by the type and cost of concept supervision they assume.

The fourth axis is the bias induced by the masking policy. MiCL reports strong sensitivity to actual mini-batch size and to overall versus per-branch visible ratio, with excessive masking degrading concept extraction; MCM also reports that too high a mask ratio degrades concept prediction and reconstruction; CCM shows that reverse curriculum and suboptimal hop count reduce performance; MAMMI notes reliance on the medical VLM, prompt sensitivity, vocabulary ambiguity, and degraded alignment under domain shift; MACCO notes dependence on concept extraction tools and continued difficulty in attribute binding; SAM3 M2C notes ambiguous or composite concepts and persistent concept-geometry inconsistency in pathological edge cases; MaSC remains limited by external mask quality and excludes style subjects from Prompt Following evaluation on DreamBench++ (2411.09858, Sun et al., 1 Feb 2025, Lee et al., 2022, Kim et al., 2024, Li et al., 11 Jun 2026, Zhou et al., 25 Jun 2026, Bartkowiak et al., 21 May 2026).

Several misconceptions are resolved by the literature itself. One is that M2C necessarily requires segmentation masks; MAMMI, MiCL, CCM, MACCO, and Mask & Focus all operate through mask-free alignment, masked views, or masked spans rather than external pixel annotations (Kim et al., 2024, 2411.09858, Lee et al., 2022, Li et al., 11 Jun 2026, Pandey et al., 2020). Another is that masked concept learning necessarily implies reconstruction; MiCL is explicitly contrastive and reconstructs nothing (2411.09858). A third is that M2C always denotes concept discovery rather than evaluation or annotation; MaSC and SAM3 M2C show that the same design principle can structure metrics and human-in-the-loop labeling pipelines (Bartkowiak et al., 21 May 2026, Zhou et al., 25 Jun 2026).

Open directions are explicit in the cited work. MiCL identifies multi-view $0.000$7, adaptive or scheduled mask ratios, lightweight queues, and alternative metric-learning objectives; CCM proposes multilingual KGs, contextualized concept linking, relation-aware weights, and application to RoBERTa or DeBERTa; MACCO highlights end-to-end concept extraction, larger corpora, multilingual settings, and stronger text encoders; SAM3 M2C suggests multi-concept or mixture prompts and multi-class extensions; MCM points to continual concept expansion and higher-resolution scaling (2411.09858, Lee et al., 2022, Li et al., 11 Jun 2026, Zhou et al., 25 Jun 2026, Sun et al., 1 Feb 2025). The resulting picture is not of a single M2C pipeline, but of a general methodological motif: masking is used to isolate, perturb, or suppress part of the signal so that a model must expose the concept structure that remains.