IConMark: Multimodal Icon Annotation

Updated 4 July 2026

IConMark is a multimodal icon annotation framework that fuses screenshot pixels with view-hierarchy metadata to localize and classify UI icons effectively.
It leverages a single-stage object detector to integrate textual and visual cues, significantly improving detection precision and recall over pixel-only approaches.
IConMark underpins a broader research agenda in content identification, influencing systems for watermarking, provenance, and multimodal governance.

IConMark is not presented in the cited literature as a single universally standardized acronym. In the corpus considered here, the term is used most concretely as a shorthand for a multimodal icon annotation framework for mobile applications, where icon bounding boxes and semantic labels are predicted by fusing screenshot pixels with view-hierarchy metadata, and it is also situated within a broader research space concerned with content identification and marking across text, image, audio, and video (Zang et al., 2021, Li et al., 13 Dec 2025).

1. Terminological scope

The literature assigns IConMark an uneven status. In "Multimodal Icon Annotation For Mobile Applications" (Zang et al., 2021), IConMark corresponds to the paper’s multimodal icon-marking or annotation approach for mobile user interfaces. In "UniMark: Artificial Intelligence Generated Content Identification Toolkit" (Li et al., 13 Dec 2025), the relation is broader: UniMark is described as occupying the same conceptual space of AI content identification and marking, especially where hidden watermarking and visible marking must coexist.

Usage in the corpus	Technical focus	Representative paper
Multimodal icon annotation	Localization and classification of UI icons from screenshots and view hierarchies	(Zang et al., 2021)
AI content identification/marking	Hidden watermarking and visible marking across text, image, audio, and video	(Li et al., 13 Dec 2025)

The ambiguity is explicit in adjacent papers. "EmMark: Robust Watermarks for IP Protection of Embedded Quantized LLMs" states that IConMark is not mentioned explicitly in that paper (Zhang et al., 2024). "MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models" likewise states that the paper does not introduce a model called IConMark and treats MICON-Bench itself as the closest match if that term was intended (Wu et al., 23 Feb 2026). This suggests that IConMark functions less as a settled technical name than as a shorthand attached to several related marking and identification problems.

2. IConMark as multimodal icon annotation

In its clearest instantiated sense, IConMark addresses icon annotation in mobile UIs: localization and classification of meaningful UI elements on a screen, with particular emphasis on object icons such as menu, search, and arrow backward (Zang et al., 2021). The problem is technically difficult because icons often have no explicit labels on screen, can resemble ordinary pictures, and vary widely in shape. The same visual pattern can correspond to different meanings in different contexts, while different icons with the same meaning may look very different.

The paper identifies two dominant prior methodological families. View-hierarchy-only methods rely on leaf nodes in the hierarchy, but Android view hierarchies are often incomplete, noisy, or unsynchronized with the visible screen. Pixel-only methods avoid hierarchy errors and exploit visual shape, but discard informative attributes such as resource-id, content description, and class name. The multimodal formulation therefore combines screenshot pixels with view-hierarchy text metadata in a single-stage object detector rather than separating localization and classification (Zang et al., 2021).

The application context is primarily accessibility and mobile interaction. Accurate icon annotation can support screen readers such as TalkBack, voice-control systems such as Voice Access, and more generally any system that must speak or act on the correct UI component. The same paper also suggests downstream relevance for design mining, UI search, sensitive-icon detection, tappability prediction, GUI prototyping, and extension to platforms where DOM trees or analogous structural metadata are available (Zang et al., 2021).

3. Dataset foundation: Rico $_{clean}$

The empirical basis for this interpretation of IConMark is Rico $_{clean}$ , a manually relabeled dataset derived from Rico, which contains more than 72k unique UI screenshots and their view hierarchies (Zang et al., 2021). The authors used 40 in-house professional crowd workers to annotate all 72k unique UI screenshots. For each screenshot, workers saw only the screenshot and drew bounding boxes around icons while assigning labels from a set of 29 icon classes. The team manually checked about 300 random UIs, refined the instructions, and repeated verification and re-annotation for two more rounds. Average annotation time was 81 seconds per screenshot.

The cleaning materially changed the dataset. The paper states that 51.4% of the icon-type UI elements in original Rico had their labels corrected, and that the cleaned dataset contains 40% more icons than the original Rico. This is presented as evidence that view hierarchies are incomplete and that indirect label generation in the original corpus introduced substantial bias and noise (Zang et al., 2021).

The 29 classes are star, arrow backward, arrow forward, more, menu, search, close, add, expand more, play, check, share, chat, settings, info, home, refresh, time, emoji, edit, notifications, call, pause, send, delete, video cam, launch, end call, and take photo. The paper explicitly notes that end_call and take_photo are newly added beyond the 27 classes that existed in Rico (Zang et al., 2021).

4. Multimodal architecture and formalization

The multimodal IConMark formulation is implemented as a single-stage object detection model built on SSD and CenterNet, with view-hierarchy features fused into an intermediate detector feature layer (Zang et al., 2021). The screenshot is encoded into an image feature map $C \in \mathbb{R}^{H \times W \times D}$ . For each leaf node $V_i$ in the hierarchy, text from the class name and resource-id fields is tokenized and passed through a pretrained text encoder to produce an embedding $t_i \in \mathbb{R}^K$ . Each hierarchy node also provides a normalized bounding box $B_i=(x_{min}, y_{min}, x_{max}, y_{max})$ .

The fusion mechanism is spatially explicit. The model constructs a binary overlay mask $O_i$ over the feature map, tiles the text embedding across the spatial region covered by the node, and then aggregates all node maps into a dense hierarchy feature map:

$O_i = calcOverlay(B_i, H, W, K), \qquad T_i = O_i \circ tile(t_i, H, W),$

$E = stack([O_1,O_2, ..., O_S]), \qquad G = \frac{\sum_{i=0}^{S}T_i}{mean(E)}.$

The resulting hierarchy map $G$ is projected to the same channel dimension as the image feature map by two $_{clean}$ 0 convolution layers and added to the image features. In effect, the model paints text-derived semantics onto the spatial regions where hierarchy nodes lie, then fuses those semantics with visual evidence (Zang et al., 2021).

The paper also evaluates a two-stage classification baseline. Candidate boxes are taken from leaf nodes in the hierarchy, icon crops are resized to $_{clean}$ 1, image embeddings are extracted from the final layer of a pretrained Inception V2, and text embeddings are produced by the pretrained TF Hub nnlm-128 module. Because resource-id exists in only about 70% of icon data, the authors introduce a sampling trick that randomly supplies a fake resource-id from a class-specific dictionary when text is missing, thereby encouraging joint multimodal learning (Zang et al., 2021).

5. Evaluation, results, and diagnostic findings

Evaluation uses micro-averaged precision, recall, and F1, together with [mAP](https://www.emergentmind.com/topics/mean-average-precision-map)@0.1 IOU and [email protected] IOU for detection (Zang et al., 2021). For detection, a predicted box is matched to ground truth if its center lies inside the ground-truth bounding box, predictions with confidence below 0.2 are discarded, and when multiple predictions match the same ground truth the highest-confidence prediction is used.

The strongest result is obtained by CenterNet + VH, where VH denotes view-hierarchy fusion. On Rico $_{clean}$ 2, the paper reports precision $_{clean}$ 3, recall $_{clean}$ 4, and F1 $_{clean}$ 5 for CenterNet + VH, compared with precision $_{clean}$ 6, recall $_{clean}$ 7, and F1 $_{clean}$ 8 for plain CenterNet. The same comparison shows [email protected] = 0.903 and [email protected] = 0.602 for CenterNet + VH, versus 0.863 and 0.592 for CenterNet (Zang et al., 2021).

Model	Precision / Recall / F1	[email protected] / [email protected]
SSD	0.446 / 0.873 / 0.590	0.787 / 0.650
SSD + VH	0.496 / 0.878 / 0.634	0.810 / 0.665
CenterNet	0.725 / 0.876 / 0.793	0.863 / 0.592
CenterNet + VH	0.765 / 0.911 / 0.832	0.903 / 0.602

The paper states that CenterNet + VH beats the best classification baseline by 28.4% F1 and improves recall by 49.5%. Compared to plain CenterNet, adding VH improves precision by about 4%, recall by about 3.5%, and F1 by about 3.9% (Zang et al., 2021). Qualitative examples illustrate why the hierarchy signal matters: a share icon partly occluded in the screenshot is correctly predicted because the resource-id contains “share,” while a logo shaped like an American flag is incorrectly predicted as menu by pure CenterNet but handled correctly by the multimodal model because the hierarchy includes a useful “logo” keyword. Failure cases remain, including visually similar Chinese characters predicted as menu and “#” predicted as settings.

6. Generalization from icon marking to multimodal content governance

A later and broader usage treats IConMark as part of a more general content identification and marking agenda. "UniMark: Artificial Intelligence Generated Content Identification Toolkit" presents an open-source, unified framework for multimodal content governance across text, image, audio, and video, and explicitly positions itself as an evolution from a broader content identification or marking idea toward a practical, production-oriented framework (Li et al., 13 Dec 2025). Its central component is the UnifiedWatermarkEngine, wrapped by a WatermarkTool class using the Facade Pattern, with unified embed and extract APIs, lazy loading for heavy models, config-driven behavior through config/default_config.yaml, and a modular adapter interface.

The major conceptual extension is the dual-operation strategy. UniMark natively supports hidden watermarking through operation='watermark' and visible compliance marking through operation='visible_mark'. Hidden watermarking is intended for copyright protection, technical tracing, post hoc identification, and low visibility or imperceptibility. Visible marking is intended for regulatory compliance, user transparency, and explicit disclosure, including visual overlays for images and videos and audio voice prompts for audio outputs. The paper frames this as a response to the fragmentation of prior tools and to the gap between technical watermarking and legal disclosure requirements (Li et al., 13 Dec 2025).

UniMark also standardizes evaluation through an "Attack-Detect-Score" loop. Quality metrics are PSNR, SSIM, LPIPS, and SNR; robustness metrics are TPR and BA. Its three specialized benchmarks are Image-Bench, based on the W-Bench dataset and resistance to 25 types of distortions; Video-Bench, based on the VideoMarkBench dataset and temporal attacks such as frame dropping and frame averaging; and Audio-Bench, based on the AudioMark dataset and transformations such as noise injection, time-stretching, and MP3 compression. This broadening of scope does not redefine IConMark as a single algorithm, but it shows how the marking problem expands from UI icon annotation into multimodal provenance, disclosure, and governance (Li et al., 13 Dec 2025).

A common misconception is to treat IConMark as interchangeable with any watermarking or marking method whose name resembles “icon” or “mark.” The cited literature does not support that equivalence. EmMark is a watermarking framework for embedded, quantized LLMs, and it explicitly states that IConMark is not mentioned in that paper (Zhang et al., 2024). ProMark is a proactive diffusion watermarking method for causal attribution of generated images to training-data concepts and supports up to $_{clean}$ 9 unique watermarks, but it is presented as causal concept attribution rather than as IConMark (Asnani et al., 2024). BitMark is an inference-time bitwise watermark for Infinity that biases generation toward green-list bit sequences and is evaluated for robustness and radioactivity, again as a provenance watermark rather than as IConMark (Kerner et al., 26 Jun 2025).

The same boundary applies to model watermarking and screen watermarking. ClearMark is a white-box DNN watermarking method based on transposed model training, designed to recover a visible, human-readable secret with 8,544-bit capacity (Krauß et al., 2023). ScreenMark is a three-stage progressive framework for arbitrary Visual Screen Content, trained on a dataset of 100,000 screenshots and using pre-multiplied alpha blending for real-time screen protection (Liang et al., 2024). These methods occupy adjacent technical territory—ownership proof, provenance, causal attribution, or screenshot resilience—but they do not establish IConMark as their canonical name.

The main boundary condition is therefore terminological. In the cited corpus, IConMark is best understood either as the multimodal icon annotation system of (Zang et al., 2021) or as a loose umbrella for content identification and marking problems that later systems such as UniMark attempt to unify (Li et al., 13 Dec 2025). It is not, on the evidence available here, a single benchmark, a single watermarking primitive, or a single universally expanded acronym.