Semantic Co-occurrence Insight Network (SCINet)

• SCINet is a multimodal framework for partial multi-label learning that leverages semantic co-occurrence patterns to resolve label ambiguity.
• It integrates a bi-dominant prompter, cross-modality fusion, and intrinsic semantic augmentation to condition predictions on inter-label and instance similarities.
• Empirical evaluations on datasets like VOC and COCO demonstrate significant performance gains over baseline methods through structured confidence refinement.

Semantic Co-occurrence Insight Network (SCINet) is a framework for partial multi-label learning in which each instance may have known correct labels, known incorrect labels, and unknown labels, and the central modeling premise is that matching co-occurrence patterns between labels and instances is crucial for resolving label ambiguity. Introduced as a multimodal architecture that combines a bi-dominant prompter module, a cross-modality fusion module, and an intrinsic semantic augmentation strategy, SCINet uses vision–language alignment, inter-label correlation, inter-instance similarity, and instance–label co-occurrence to refine label confidence and improve prediction under incomplete supervision (Wu et al., 8 Jul 2025).

1. Problem formulation and motivating principles

SCINet is defined in the setting of partial multi-label learning rather than standard fully supervised multi-label classification. Let the label space be $$\mathcal{Y} = \{\mathbf{l}_1, \dots, \mathbf{l}_q\}$$. In the formulation used by SCINet, the training data are represented as

$$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$

where $\mathbf{s}_i$ is an instance, $C_i$ is the set of known correct labels, and $U_i$ is the set of unknown labels. Known incorrect labels are also part of the annotation regime and are treated as explicit negatives in label-confidence construction. The learning problem is therefore not merely to score labels, but to infer the status of unknown labels while respecting both observed positives and observed negatives (Wu et al., 8 Jul 2025).

The paper identifies the core difficulty as the ambiguous relationship between instances and labels when annotation is incomplete. Unknown labels cannot safely be treated as negatives, but ignoring them discards exploitable structure. SCINet addresses this by treating semantic co-occurrence knowledge as the principal regularizer. In this context, semantic co-occurrence knowledge comprises statistical correlations among labels, similarity among instances, and cross-modal priors provided by a pretrained vision–LLM. The motivating claim is explicit: matching co-occurrence patterns between labels and instances is key to resolving ambiguity in partial multi-label learning (Wu et al., 8 Jul 2025).

This focus places SCINet within a class of methods that use structural priors rather than only per-instance supervision. Prior work cited in the SCINet study includes matrix completion with side information, label-correlation models, losses for missing labels, semantic-aware representation blending, and CLIP-based prompting methods. The architectural distinction of SCINet is that it simultaneously models inter-label correlations, inter-instance relationships, and co-occurrence patterns across instance–label assignments instead of relying on only one of these structures (Wu et al., 8 Jul 2025).

2. Architectural composition

SCINet consists of three tightly coupled components: a bi-dominant prompter module, a cross-modality fusion module, and an intrinsic semantic augmentation strategy. The data flow begins with partially labeled images and ends with refined label confidences and multimodally aligned predictions (Wu et al., 8 Jul 2025).

Component	Primary inputs	Function
Bi-dominant prompter	prompts, label names, images	captures text-image correlations and enhances semantic alignment
Cross-modality fusion	visual features, partial labels	models inter-label, inter-instance, and instance-label co-occurrence
Intrinsic semantic augmentation	weak, original, strong image views	couples label confidence with sample difficulty through consistency and self-distillation

The bi-dominant prompter is built on a modified CLIP model. On the text side, the prompt sequence is

$V = [v_1, v_2, \dots, v_m, \text{CLS}],$

where each $v_i$ is a learnable soft prompt token and CLS denotes the label name. On the image side, the model processes transformed images collected in $X'$. The two dominant encoders are defined as

$z_i = G(V, X'^{(1)}),$

$f = F(X', V^{(1)}),$

where $$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$0 is the text-dominant encoder and $$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$1 is the visual-dominant encoder. This design lets textual prompts absorb visual context and lets visual features be conditioned by textual information, thereby strengthening semantic alignment under sparse supervision (Wu et al., 8 Jul 2025).

The cross-modality fusion module takes the multimodal features and the partial label matrix, then refines label confidence by jointly exploiting label correlation and instance similarity. The intrinsic semantic augmentation strategy then imposes consistency across weak, original, and strong transformations, so that the refined confidence estimates are not tied to a single view of the image. The resulting framework is not a simple prompting scheme layered on top of CLIP; it is a co-occurrence-driven confidence-refinement system in which the CLIP prior is only one component of a larger inference structure (Wu et al., 8 Jul 2025).

3. Modeling label, instance, and instance–label co-occurrence

The mathematical core of SCINet is the label-confidence matrix $$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$2, which is optimized so that it remains close to the observed partial labels while varying smoothly over both the instance graph and the label graph. The instance neighborhood around sample $$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$3 is defined as

$$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$4

and the paper constructs an instance similarity matrix $$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$5 with a Gaussian kernel over these neighborhoods. In parallel, label correlation is computed with the Pearson correlation coefficient

$$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$6

Here, $$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$7 measures how strongly labels $$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$8 and $$\mathbf{X} = \left\{ ( \mathbf{s}_1, C_1, U_1), (\mathbf{s}_2, C_2, U_2), \dots, (\mathbf{s}_n, C_n, U_n) \right\},$$9 co-occur across the dataset (Wu et al., 8 Jul 2025).

The confidence-refinement objective is

$\mathbf{s}_i$0

where $\mathbf{s}_i$1 is the observed partial label matrix, $\mathbf{s}_i$2 encodes inter-instance similarity, and $\mathbf{s}_i$3 encodes inter-label correlation. The first term preserves fidelity to known annotations. The second propagates supervision among similar instances. The third enforces compatibility among correlated labels. Unknown labels are therefore not left unconstrained; they are inferred through smoothness in both sample space and label space (Wu et al., 8 Jul 2025).

This construction determines how SCINet handles the three label states. Known correct labels pull entries of $\mathbf{s}_i$4 toward positive confidence values, known incorrect labels pull them toward negative confidence values, and unknown labels are resolved by interaction with visually similar instances and statistically correlated labels. A plausible implication is that SCINet treats missing supervision as a structured inference problem rather than a missing-data nuisance. That interpretation is consistent with the role assigned to $\mathbf{s}_i$5 in later training stages, where it modulates consistency and self-distillation losses (Wu et al., 8 Jul 2025).

The paper also reports that Pearson correlation is empirically stronger than cosine similarity and the Gini coefficient for constructing the label-correlation structure. On COCO2014, Pearson correlation yielded 77.98 mAP, 72.66 OF1, and 71.40 CF1, compared with 74.30/64.17/55.84 for cosine similarity and 73.97 mAP for the Gini coefficient (Wu et al., 8 Jul 2025).

4. Intrinsic semantic augmentation and optimization

SCINet augments each image into three versions: a weak transformation $\mathbf{s}_i$6, an original or medium transformation $\mathbf{s}_i$7, and a strong transformation $\mathbf{s}_i$8. The corresponding predictive distributions are

$\mathbf{s}_i$9

Weak transformations preserve most of the semantics, while strong transformations increase diversity and encourage robustness. The model uses these three views to couple label confidence with sample difficulty (Wu et al., 8 Jul 2025).

A confident label set $C_i$0 is constructed per instance by thresholding predicted probabilities with $C_i$1. Labels above the threshold are treated as confident positives for consistency learning; the others are treated as non-confident. The study reports that $C_i$2 works best on COCO2014, giving the most favorable balance across mAP, OF1, and CF1 (Wu et al., 8 Jul 2025).

Two consistency losses, $C_i$3 and $C_i$4, enforce agreement between weak and original views, and between original and strong views, respectively. A third loss, $C_i$5, performs self-distillation. In the formulation described by the paper, the medium-view distribution is modulated by label confidence through

$C_i$6

and then aligned with the auxiliary views by KL-style terms. The resulting training signal combines view consistency with confidence-aware refinement, so that agreement is not imposed uniformly across all labels but is instead weighted by the confidence structure inferred in the cross-modality fusion stage (Wu et al., 8 Jul 2025).

The losses are not combined with static hand-tuned coefficients. Instead, SCINet uses Pareto multi-objective optimization: the losses are placed into a list, the Pareto frontier is calculated, and the loss weights are updated dynamically. In implementation, the paper uses Adam with learning rate $C_i$7 and weight decay $C_i$8. The reported hyperparameters are $C_i$9, $U_i$0, and $U_i$1 (Wu et al., 8 Jul 2025).

5. Empirical evaluation and ablation structure

SCINet is evaluated in two regimes: a single-positive-label setting and a partial-label setting. The single-positive-label experiments use VOC2012, COCO2014, and CUB-200-2011 under the LargeLoss and SPLC protocols. The partial-label experiments use VOC2007 and COCO2014 with known-label fractions of 10%, 30%, 50%, 70%, and 90% (Wu et al., 8 Jul 2025).

Evaluation setting	Dataset summary	Reported SCINet result
Single positive label, LargeLoss	VOC2012, COCO2014, CUB	average mAP 64.21
Single positive label, SPLC	VOC2012, COCO2014, CUB	average mAP 64.79
Partial labels	VOC2007	average mAP 92.53
Partial labels	COCO2014	average mAP 77.93
Full ablation gain	four-dataset average	66.61 to 73.27

Under the LargeLoss protocol, SCINet attains 90.97 mAP on VOC2012, 75.52 on COCO2014, and 26.16 on CUB, with an average of 64.21. Under SPLC, the corresponding results are 91.76, 76.46, and 26.16, with an average of 64.79. The paper states that these averages improve over SCPNet by +1.04 and +1.21, respectively. In the partial-label setting on VOC2007, SCINet reaches 92.32 mAP when only 10% of labels are known and 93.97 at 90% known labels, with an average of 92.53. On COCO2014 in the same regime, the average is 77.93 mAP (Wu et al., 8 Jul 2025).

The ablation study decomposes SCINet into the baseline, the bi-dominant prompter, the cross-modality fusion module, and the semantic augmentation losses. The baseline average mAP is 66.61. Adding only the bi-dominant prompter yields +3.59, only the cross-modality fusion module yields +3.90, and only the semantic-augmentation component yields +4.21, raising performance to 70.82. The full system reaches 73.27 average mAP, a total gain of +6.66 over the baseline. This decomposition is important because it shows that SCINet’s empirical gain is not concentrated in the CLIP prompting component alone; the co-occurrence-driven confidence refinement and the augmentation losses each contribute measurably (Wu et al., 8 Jul 2025).

The paper also reports qualitative and systems-level analyses. t-SNE visualizations for VOC2007 categories such as “person,” “chair,” and “motorcycle” show tighter intra-class grouping and better inter-class separation under SCINet than under the baseline. Prompt-length experiments show a trade-off: long prompt sequences can increase false detections in cluttered scenes, while moderate prompt length is preferable. In backbone comparisons on COCO2014, ResNet-50 gives 77.98 mAP with 623.7 ms inference, while ViT-B/16 yields 82.07 mAP but 1387 ms inference, indicating a clear accuracy–latency trade-off (Wu et al., 8 Jul 2025).

6. Relation to broader semantic co-occurrence research, limitations, and outlook

SCINet is specifically a multimodal framework for partial multi-label learning; it is not a generic co-occurrence network constructor, nor a universal model of semantic similarity. That distinction matters because the term “semantic co-occurrence” has been operationalized in several different ways across the literature. In semantic segmentation, ISNet showed that explicit same-category aggregation complements image-level context and improves dense prediction, indicating that context modeling benefits from separating global relations and semantic-level consistency (Jin et al., 2021). In attributed network embedding, CoANE modeled context co-occurrence by aligning structural contexts and attribute channels, using random walks and 1-D convolution to capture latent social circles (Hsieh et al., 2021). In static word embedding, SemGloVe replaced local-window co-occurrence counts with BERT-derived semantic co-occurrences, showing that co-occurrence can be distilled from contextual LLMs rather than counted only from proximity in text (Gan et al., 2020).

Related work outside multi-label vision also clarifies both the potential and the limits of co-occurrence priors. Studies of enriched word co-occurrence networks found that adding semantic edges can improve the informativeness of some graph metrics while degrading others, especially when the enrichment is too dense or stopword handling is poorly controlled (Amancio et al., 2024). Research on semantic roles showed that role combinations are highly constrained: only 107 unique VerbNet role frames occur out of 768,211 possible combinations, which indicates that co-occurrence can reveal hidden structural restrictions rather than just frequency patterns (Huminski et al., 2018). This suggests that SCINet belongs to a broader methodological trend in which co-occurrence is treated as a structured prior over permissible or likely configurations.

The limitations reported for SCINet are concrete. Prompt length must be balanced against false detections, especially in complex scenes with small objects. The framework depends on CLIP and on the semantic adequacy of label names, so its performance is tied to the quality and domain coverage of the underlying multimodal pretraining. It also assumes that label co-occurrence patterns are stable enough to be informative; if co-occurrence is noisy or weakly structured, the benefit of the correlation module may diminish (Wu et al., 8 Jul 2025).

The future directions listed in the study are correspondingly focused. They include more interpretable and fine-grained analysis, language decomposition for understanding prompt semantics, adaptive prompt learning that adjusts prompt length to scene complexity, and architecture optimization for improved robustness and efficiency (Wu et al., 8 Jul 2025). A plausible implication is that later variants may make the co-occurrence structure itself more adaptive, rather than treating label correlation and prompt geometry as largely fixed design choices.