Dual-Level Semantic Supervision

• Dual-Level Semantic Supervision is a training paradigm that couples two semantically meaningful levels, such as coarse and fine labels, to jointly guide model learning.
• It employs explicit coupling mechanisms like shared attention, prototype assignments, and teacher–student transfer to align granularity or representational spaces across tasks.
• The approach enhances performance and interpretability in fields like NLP, semantic segmentation, and multimodal reasoning, especially under weak or sparse annotation regimes.

Searching arXiv for the cited papers to ground the article and verify identifiers. Dual-level semantic supervision is a training paradigm in which semantic information is imposed at two distinct levels and the two levels are explicitly coupled rather than learned in isolation. In the literature, the paired levels vary with the task: sentence and token labels in NLP, category-agnostic and category-aware edges in semantic edge detection, sentence and phrase labels in image-text retrieval, image-space and feature-space consistency in semi-supervised semantic segmentation, global and local teacher guidance in segmentation, feature-level and model-level consistency in biomedical VQA, supervision-level and feature-level guidance in sparse-input NeRF, and semantic and visual supervision of latent reasoning states in medical multimodal LLMs (Rei et al., 2018, Liu et al., 2018, Fan et al., 2021, Tain et al., 2023, Tran et al., 14 Apr 2025, Ji et al., 4 Mar 2025, Zhong et al., 4 Mar 2025, Li et al., 27 May 2026).

1. Conceptual scope and recurring formulations

Across these works, “dual-level” does not denote a single fixed architectural recipe. It denotes a family of couplings between two semantically meaningful strata of supervision. In some settings the two levels differ by annotation granularity, as in sentence-level versus token-level labels or sentence-level versus phrase-level labels. In other settings they differ by representational space, such as logit space versus representation space, image space versus feature space, or supervision level versus feature level. Elsewhere they differ by semantic abstraction, such as global context versus local region structure, or low-level detail versus high-level semantics (Rei et al., 2018, Wang et al., 2023, Tain et al., 2023, Tran et al., 14 Apr 2025, Huang et al., 6 Aug 2025).

A common property is that the two levels are not merely co-trained. They are tied by a mechanism that allows one level to shape the other: shared attention scores, structured losses, prototype assignments, teacher–student transfer, masked cross-modal interaction, or feature querying from a learned codebook. This coupling is the defining feature of the paradigm. It distinguishes dual-level supervision from ordinary multi-task learning in which two losses simply share an encoder.

Formulation	Two supervised levels	Coupling mechanism
Joint sentence/token labeling (Rei et al., 2018)	Sentence labels; token labels	Token predictions reused as attention weights
Diverse deep supervision for SED (Liu et al., 2018)	Category-agnostic edges; category-aware edges	Information converters plus side losses
Image-text retrieval (Fan et al., 2021)	Sentence match; phrase match	Phrase nodes, masked attention, phrase loss
DSSN (Tain et al., 2023)	Image-space; feature-space	Dual-level contrastive and weak-to-strong losses
Space Engage (Wang et al., 2023)	Logit space; representation space	Mix/cross pseudo-labeling and prototypes
IGL-DT (Tran et al., 14 Apr 2025)	Global context; local region structure	Dual teachers, discrepancy learning, swap
BioD2C (Ji et al., 4 Mar 2025)	Feature-level interaction; model-level consistency	Fusion gate plus KL semantic loss
VITAL (Li et al., 27 May 2026)	Semantic supervision; visual supervision	Auxiliary decoder and visual projector
S³NeRF (Zhong et al., 4 Mar 2025)	Supervision level; feature level	BDV filtering plus semantic-aware codebook
DS$^2$Net (Huang et al., 6 Aug 2025)	Detail supervision; semantic supervision	DEM/SEM with uncertainty weighting

This variability also clarifies a frequent misconception: dual-level semantic supervision is not restricted to coarse-versus-fine labels within a single modality. The surveyed papers instantiate it in uni-modal, cross-modal, latent-space, and neural-field settings.

2. Granularity-based supervision in language and retrieval

In "Jointly Learning to Label Sentences and Tokens" (Rei et al., 2018), dual-level semantic supervision is formulated as sentence-level classification coupled to token-level sequence labeling. Each contextual token representation $h_i$ produces a scalar token prediction $\hat{a}_i$, and the same scalar is then normalized into attention weights,

$$\widetilde{a}_i = \frac{\hat{a}_i}{\sum_{k=1}^N \hat{a}_k}, \qquad s = \sum_{i=1}^N \widetilde{a}_i h_i.$$

The crucial design choice is that the token classifier output and the sentence-composition attention are the same quantity. Token-level labels therefore supervise not only local predictions but also the sentence representation itself. Conversely, sentence-level loss and the attention-range loss regularize token predictions, including regimes with sparse or missing token annotations. The model further combines sentence MSE, token MSE, word-level language modeling, character-level language modeling, and an attention-range loss in a joint objective.

In "Constructing Phrase-level Semantic Labels to Form Multi-Grained Supervision for Image-Text Retrieval" (Fan et al., 2021), the two levels are sentence-level match/mismatch and phrase-level match/mismatch. Phrase labels are automatically constructed from matched captions by parsing text scene graphs with the SPICE pipeline and extracting entities, object–attribute pairs, and object–relation–object triples. SSAMT then introduces sentence nodes, phrase nodes, word nodes, image region nodes, and masked attention that constrains phrase nodes to attend only to their spans. Global matching losses supervise sentence-level retrieval, while local and phrase matching losses penalize mismatched phrases inside otherwise partially relevant negative captions.

These two papers exemplify the original granularity-centered interpretation of the concept. In both, the finer level is not an auxiliary decoration. It actively constrains the representation used for the coarser decision. A plausible implication is that dual-level supervision is most effective when the local labels identify the units that actually carry the sentence-level semantics, such as hedge cues, sentiment-bearing phrases, or retrieval-critical phrases.

3. Space- and scale-based supervision in dense prediction

In dense prediction, dual-level semantic supervision often appears as supervision in two spaces or at two semantic scales. "Semantic Edge Detection with Diverse Deep Supervision" (Liu et al., 2018) separates category-agnostic edge supervision at bottom side outputs from category-aware semantic edge supervision at the top side output and fused output. Its central claim is that these objectives conflict when backpropagated naively through the same backbone. The information converter units are introduced precisely to buffer this conflict, allowing low-level binary edge supervision and high-level semantic edge supervision to coexist within one fully convolutional network.

"Improving Semi-Supervised Semantic Segmentation with Dual-Level Siamese Structure Network" (Tain et al., 2023) defines the two levels as low-level image space and high-level feature space. For unlabeled images, two strong image augmentations produce low-level strong views, while a weakly augmented feature map is strongly perturbed twice in feature space by random dropout. Pixel-wise contrastive consistency is imposed at both levels, and weak teacher predictions generate class-aware pseudo-labels that supervise strong views at both levels. The full objective

$$\mathcal L=\mathcal L_{\rm sup} +\gamma_1 \bigl(\mathcal L_{\rm cl}^{ls}+\mathcal L_{\rm cl}^{hs}\bigr) +\gamma_2 \bigl(\mathcal L^{(l)}_{\rm w2s}+\mathcal L^{(h)}_{\rm w2s}\bigr)$$

therefore integrates supervised CE, dual-level contrastive losses, and dual-level weak-to-strong pseudo-supervision.

"Space Engage: Collaborative Space Supervision for Contrastive-based Semi-Supervised Semantic Segmentation" (Wang et al., 2023) uses a different decomposition: logit space and representation space. Teacher logits generate logit-space pseudo-labels, while teacher embeddings are classified by nearest prototypes in representation space. The two spaces then collaborate through mix pseudo-labeling, which keeps only pixels where both spaces agree, and cross pseudo-labeling, which lets one space supervise the other. This formulation makes dual-level supervision a mechanism for pseudo-label refinement and prototype stabilization rather than a purely architectural decomposition.

"IGL-DT: Iterative Global-Local Feature Learning with Dual-Teacher Semantic Segmentation Framework under Limited Annotation Scheme" (Tran et al., 14 Apr 2025) shifts the emphasis from spaces to semantic scale. A SwinUnet global teacher provides global semantic guidance through Global Context Learning, while a ResUnet local teacher provides detailed feature refinement through Local Regional Learning. The student alternates between

$$\mathcal{L}_{s1} = \mathcal{L}_{Glo}(S, T_{SU}) + \alpha \mathcal{L}_{Dis}(S, T_{RU})$$

and

$$\mathcal{L}_{s2} = \mathcal{L}_{Loc}(S, T_{RU}) + \alpha \mathcal{L}_{Dis}(S, T_{SU}),$$

using a swap schedule that alternates global-focused and local-focused iterations. Dual-level supervision is thus realized by specialized teachers, level-specific distillation losses, and discrepancy learning that discourages collapse to a single teacher.

"DS$^2$Net: Detail-Semantic Deep Supervision Network for Medical Image Segmentation" (Huang et al., 6 Aug 2025) makes the same global/local intuition explicit inside a decoder. Its Detail Enhance Module derives a detail mask from low-level features, while its Semantic Enhance Module derives a semantic mask from high-level features. The network supervises three detail-focused outputs and three semantic-focused outputs, then weights them adaptively by uncertainty. This moves from conventional single-view deep supervision to what the paper calls multi-view deep supervision.

Taken together, these dense-prediction papers show that “dual-level” in vision is a flexible abstraction. It can refer to supervision across layers, spaces, teachers, or masks, but in each case the two levels are semantically nonredundant and mutually constraining.

4. Feature-level, model-level, and latent dual supervision in multimodal reasoning and neural fields

In biomedical VQA, "BioD2C: A Dual-level Semantic Consistency Constraint Framework for Biomedical VQA" (Ji et al., 4 Mar 2025) defines the two levels as feature-level semantic interaction and model-level semantic consistency. At feature level, a 12-layer Transformer decoder uses text as queries and multi-scale visual features as keys and values, then a learnable gate forms the text-conditioned image representation

$$X_{v|t}=Proj(X_v)+Proj_g(X_{vt}) \cdot \tanh(\beta).$$

At model level, the method builds a text queue $\mathcal Q_t$, computes semantic distributions $h_i$0 and $h_i$1 over the queue from the image-conditioned features and the question, and minimizes

$h_i$2

The architectural fusion supplies question-aware visual features, while the KL term forces their semantic distribution to match the question semantics.

"VITAL: Visual-Semantic Dual Supervision for Enhanced and Interpretable Latent Reasoning in Medical MLLMs" (Li et al., 27 May 2026) defines dual supervision over latent reasoning states $h_i$3. Each state is supervised semantically by an auxiliary text decoder that reconstructs the corresponding reasoning step $h_i$4, and visually by a projector that regresses a pre-extracted ROI feature $h_i$5 from a frozen, independent medical vision encoder. The total objective

$h_i$6

leaves the latent loop itself unchanged between training and inference. This is an important variation of the paradigm: both supervision heads are read-only scaffolding during training and are removed at inference with zero overhead.

"Empowering Sparse-Input Neural Radiance Fields with Dual-Level Semantic Guidance from Dense Novel Views" (Zhong et al., 4 Mar 2025) defines the two levels as supervision level and feature level. At supervision level, a teacher Semantic NeRF renders dense novel-view semantic labels and depth maps, and a bi-directional verification module validates each rendered label by source-to-novel and novel-to-source projection consistency before using it as pseudo supervision. At feature level, a semantic-aware codebook $h_i$7 with $h_i$8 and $h_i$9 is queried by each implicit feature $\hat{a}_i$0 through dot-product attention to produce $\hat{a}_i$1, which is then added back to $\hat{a}_i$2 before color and density prediction. This is a neural-field analogue of dual-level semantic supervision: label-level filtering regularizes rendered semantics, while codebook-mediated feature regularization injects semantic priors into appearance and geometry.

These multimodal and neural-field variants show that the two levels need not be two output granularities. They may instead be two kinds of semantic anchoring applied to the same latent state, visual feature, or radiance-field representation.

5. Objective design and empirical behavior

The objective functions used in dual-level semantic supervision are heterogeneous. The cited works use MSE for sentence and token objectives (Rei et al., 2018), reweighted sigmoid cross-entropy for semantic edge detection (Liu et al., 2018), triplet losses for global, local, and phrase matching (Fan et al., 2021), cross-entropy plus KL divergence for model-level semantic alignment (Ji et al., 4 Mar 2025), pixel-wise contrastive losses and weak-to-strong pseudo-label CE in semi-supervised segmentation (Tain et al., 2023), prototype-based contrastive loss in representation space (Wang et al., 2023), L1 global alignment and correlation-matrix matching in dual-teacher segmentation (Tran et al., 14 Apr 2025), uncertainty-weighted IoU and BCE losses in DS$\hat{a}_i$3Net (Huang et al., 6 Aug 2025), and answer CE plus auxiliary text and visual losses for latent reasoning (Li et al., 27 May 2026). The common pattern is not a shared loss family but a shared insistence that both semantic levels must shape the same representational substrate.

Paper	Setting	Representative reported effect
(Rei et al., 2018)	Joint sentence/token labeling	CoNLL 2010 sentence $\hat{a}_i$4: 83.87 $\hat{a}_i$5 87.17; FCE sentence $\hat{a}_i$6: 85.21 $\hat{a}_i$7 86.01
(Liu et al., 2018)	Semantic edge detection	SBD ODS: CASENet 71.4, DDS-R 73.3, DDS-U 74.8
(Fan et al., 2021)	Image-text retrieval	MS-COCO 1K RSum: 519.4; phrase-level classification 97.1% vs 30.0% without PM
(Tain et al., 2023)	Semi-supervised semantic segmentation	VOC 1/8, R-101: 76.12 $\hat{a}_i$8 79.58 with dual-level contrastive + CPLG
(Wang et al., 2023)	Collaborative space supervision	92 labels: logit-only 67.11, mix pseudo-labeling 68.41; pseudo-label mIoU 82.17
(Tran et al., 14 Apr 2025)	Global-local dual-teacher segmentation	Cityscapes mIoU: 77.5 / 78.8 / 80.0 / 81.1 across 1/16, 1/8, 1/4, 1/2
(Ji et al., 4 Mar 2025)	Biomedical VQA	Average over SLAKE, RAD-VQA, Path-VQA: 0.641 vs 0.616
(Zhong et al., 4 Mar 2025)	Sparse-input NeRF	ScanNet++: 21.17 / 0.779 / 0.395 $\hat{a}_i$9 22.21 / 0.787 / 0.364
(Li et al., 27 May 2026)	Medical latent reasoning	Full dual supervision: 81.08 / 74.28 / 80.50; latency $$\widetilde{a}_i = \frac{\hat{a}_i}{\sum_{k=1}^N \hat{a}_k}, \qquad s = \sum_{i=1}^N \widetilde{a}_i h_i.$$0s vs explicit CoT $$\widetilde{a}_i = \frac{\hat{a}_i}{\sum_{k=1}^N \hat{a}_k}, \qquad s = \sum_{i=1}^N \widetilde{a}_i h_i.$$1s
(Huang et al., 6 Aug 2025)	Medical image segmentation	Kvasir-SEG mDice 92.72%, mIoU 87.85%; 2018-DSB mDice 90.97%, mIoU 83.95%

Several empirical regularities recur across these results. First, the auxiliary level usually improves the primary task rather than merely adding interpretability. Token supervision improves sentence classification (Rei et al., 2018), phrase supervision improves retrieval (Fan et al., 2021), and feature-level or visual-level supervision improves latent reasoning or NeRF synthesis (Li et al., 27 May 2026, Zhong et al., 4 Mar 2025). Second, the gains are especially marked under weak supervision or sparse annotation. BiLSTM-JOINT reaches approximately $$\widetilde{a}_i = \frac{\hat{a}_i}{\sum_{k=1}^N \hat{a}_k}, \qquad s = \sum_{i=1}^N \widetilde{a}_i h_i.$$2 token $$\widetilde{a}_i = \frac{\hat{a}_i}{\sum_{k=1}^N \hat{a}_k}, \qquad s = \sum_{i=1}^N \widetilde{a}_i h_i.$$3 on CoNLL-2010 cue detection with $$\widetilde{a}_i = \frac{\hat{a}_i}{\sum_{k=1}^N \hat{a}_k}, \qquad s = \sum_{i=1}^N \widetilde{a}_i h_i.$$4 token labels through sentence supervision and attention-range constraints alone (Rei et al., 2018), while DSSN and IGL-DT report their strongest advantages under low-label segmentation splits (Tain et al., 2023, Tran et al., 14 Apr 2025). Third, simple coexistence of two losses is usually insufficient; the strongest results come from explicit cross-level coupling such as shared attention, class-aware thresholding, prototype agreement, or discrepancy-controlled alternation.

6. Interpretability, misconceptions, and research directions

A major attraction of the paradigm is that the second supervision level often becomes an explanatory interface. In joint sentence/token labeling, the token scores $$\widetilde{a}_i = \frac{\hat{a}_i}{\sum_{k=1}^N \hat{a}_k}, \qquad s = \sum_{i=1}^N \widetilde{a}_i h_i.$$5 are simultaneously token predictions and attention weights, so the sentence decision can be traced to specific hedge cues, error tokens, or sentiment-bearing words (Rei et al., 2018). In phrase-supervised retrieval, the model can identify which phrases inside a negative caption are actually mismatched to the image (Fan et al., 2021). In VITAL, the auxiliary decoder and visual projector can be re-attached post hoc to decode latent reasoning steps and generate visual heatmaps, yielding textual and visual explanations without changing inference-time computation (Li et al., 27 May 2026). BioD2C visualizations likewise show question-dependent attention shifts toward regions such as a black arrow or two wires, linking feature-level fusion to visually grounded question answering (Ji et al., 4 Mar 2025).

A second recurrent theme is that dual-level supervision is often introduced to counter a failure mode of single-level training. Diverse deep supervision in semantic edge detection addresses supervision conflict between fine edges and high-level semantics (Liu et al., 2018). DSSN addresses underuse of unlabeled data by supervising both image-space and feature-space invariance (Tain et al., 2023). Space Engage addresses over-fitting to incorrect semantic information in logits by adding supervision from representation space (Wang et al., 2023). VITAL targets modality collapse, train–inference mismatch, and latent opacity (Li et al., 27 May 2026). S³NeRF targets shape–radiance ambiguity under sparse views by replacing unreliable rendered RGB augmentation with verified rendered semantics (Zhong et al., 4 Mar 2025). This suggests that the paradigm is most useful when one semantic view is systematically insufficient or unstable on its own.

The limitations reported across papers are equally consistent. Multi-view or multi-branch formulations increase computation and memory, as noted for DSSN and IGL-DT (Tain et al., 2023, Tran et al., 14 Apr 2025). Prototype-based approaches depend on good initialization or warm-up, as in Space Engage (Wang et al., 2023). Automatically derived fine-grained labels may inherit parser errors or retrieval noise, as in phrase-level supervision (Fan et al., 2021). Teacher-dependent systems inherit teacher biases, as in VITAL and S³NeRF (Li et al., 27 May 2026, Zhong et al., 4 Mar 2025). Several methods also expose unresolved scheduling questions: IGL-DT uses a fixed odd/even alternation, DS$$\widetilde{a}_i = \frac{\hat{a}_i}{\sum_{k=1}^N \hat{a}_k}, \qquad s = \sum_{i=1}^N \widetilde{a}_i h_i.$$6Net replaces heuristic weights with uncertainty scaling but still relies on architectural priors about which mask should come from which feature level, and VITAL fixes reasoning depth at $$\widetilde{a}_i = \frac{\hat{a}_i}{\sum_{k=1}^N \hat{a}_k}, \qquad s = \sum_{i=1}^N \widetilde{a}_i h_i.$$7 (Tran et al., 14 Apr 2025, Huang et al., 6 Aug 2025, Li et al., 27 May 2026).

Future directions are already visible within the cited papers. DSSN explicitly notes the possibility of extending beyond two levels to a multi-level hierarchy (Tain et al., 2023). Space Engage points to richer knowledge exchange and improved prototype modeling (Wang et al., 2023). IGL-DT motivates adaptive or confidence-aware alternatives to fixed alternation (Tran et al., 14 Apr 2025). VITAL proposes adaptive reasoning depth and broader supervision sources such as radiology reports and clinical narratives (Li et al., 27 May 2026). BioD2C suggests broader multimodal integration and more sophisticated semantic reference sets (Ji et al., 4 Mar 2025). A plausible synthesis is that dual-level semantic supervision is best understood not as a terminal design pattern but as the simplest nontrivial instance of multi-level semantic coupling: once two levels are shown to be beneficial, the next research question is how many levels to use, how to align them, and how to keep the resulting system trainable, data-efficient, and interpretable.