Coarse-to-Fine Supervision

• Coarse-to-fine supervision is a hierarchical learning strategy that begins with broad, low-resolution labels and advances to detailed, high-resolution outputs.
• It leverages curriculum learning by gradually increasing task complexity and employs pseudo-labeling to enrich coarse annotations.
• This approach is pivotal in applications such as semantic segmentation, few-shot learning, and 3D reconstruction to enhance optimization and data efficiency.

Coarse-to-fine supervision is a foundational strategy in modern machine learning and pattern recognition that structures model supervision, learning, or adaptation to proceed hierarchically: starting from broad, global, or low-resolution (coarse) targets and progressing toward finer-grained, detailed, or higher-resolution objectives. This paradigm is widely leveraged in semantic segmentation, few-shot learning, mathematical reasoning, multimodal retrieval, named entity recognition, 3D reconstruction, multiple instance learning, and fine-grained classification, among others. The essential principle is to guide learning from easier or less detailed tasks toward harder or more specific ones, thereby stabilizing training, improving generalization, and enabling supervision when fine-grained annotations are scarce.

1. Core Principles and Motivations

Coarse-to-fine supervision decomposes high-complexity learning tasks into a sequence of auxiliary tasks, each operating at a finer granularity. At each hierarchical level, supervision may take the form of:

• Lower-resolution labels (coarse semantic or spatial granularity in segmentation or parsing)
• Clustered, merged, or higher-level classes (grouped categories, coarse entity types, or merged reasoning steps)
• Pseudo-labels or weak supervision that can be refined (label propagation, self-distillation, or bootstrapped annotation enrichment)

The principal motivations include:

• Optimization stability: Each supervision stage eases the learning burden by guiding the model through incrementally harder tasks.
• Data efficiency: Coarse annotations require less expertise or labeling effort and can be algorithmically refined to detailed supervision (Luo et al., 2018).
• Curriculum learning: Gradually exposing the model to more difficult tasks improves convergence and robustness.
• Bridging annotation gaps: Where precise labels are unavailable, coarse-to-fine mechanisms can generate surrogate labels at finer scales (Mekala et al., 2021, Yang et al., 2020).

2. Representative Architectures and Supervisory Mechanisms

Coarse-to-fine supervision can be realized in network architectures, training curricula, or reinforcement learning pipelines.

Semantic Segmentation and Parsing

• Stacked refinement blocks: As in the Label Refinement Network, a deep CNN’s intermediate features at multiple resolutions are passed to a hierarchy of decoder heads. Each head predicts semantic labels at a particular scale; further heads combine upsampled coarse predictions with higher-resolution features to incrementally sharpen detail. Explicit loss terms are imposed at all levels (Islam et al., 2017).
• Progressive parsing: Stacked segmentation modules receive merged class maps for coarse stages and detailed classes for fine stages, often enhanced with skip connections to recover small structures (Hu et al., 2018).

Classification and Embedding Learning

• Pseudo-labeling and clustering: When only coarse labels are available for training, feature embeddings can be clustered within each coarse class to induce pseudo-fine labels; meta-learning or contrastive methods then supervise on these new partitions (Yang et al., 2020).
• Angular contrastive learning: In fine-grained few-shot tasks, embeddings are first separated by coarse classes, then refined within the tangent space to encourage intra-coarse-class angular diversity, supporting downstream specialization (Bukchin et al., 2020).

Mathematical Reasoning and Sequence Tasks

• Hierarchical reward modeling: In process reward modeling, adjacently grouped reasoning steps form a hierarchy from coarse macro-steps to fine atomic steps; learning proceeds from merged to unmerged windows, providing multi-granular supervision (Hu et al., 23 Jan 2025).
• Neural semantic parsing: A sketch decoder first generates a high-level representation (skeleton) of the output; a detail decoder then fills in fine details, jointly maximizing both likelihoods (Dong et al., 2018).

Multimodal and 3D Tasks

• Contrastive + fine-alignment: For speech-image retrieval, initial contrastive alignment at the global level is followed by a matching task focusing on fine-grained distinctions, jointly supervised (Zhou et al., 2024).
• 3D representation alignment: In COREA, coarse supervision via depth maps is first imposed across signed distance field and 3D Gaussian representations, followed by fine-level alignment using depth gradients and surface normals for high-fidelity geometry (Lee et al., 8 Dec 2025).

3. Loss Functions, Supervision Hierarchies, and Training Protocols

Across domains, key mathematical approaches involve multi-level loss summation, hierarchical label mapping, and stagewise training. Commonly observed patterns include:

• Multi-resolution and deeply supervised losses: At each coarse-to-fine stage, outputs are directly supervised by downsampled or merged ground-truth, with total loss given by a weighted sum:

$$L_{\mathrm{total}} = \sum_{i=1}^{T} \lambda_i\,\mathcal{L}_i$$

where each $\mathcal{L}_i$ is a per-stage cross-entropy, L1/L2, or matching loss (Islam et al., 2017, Hu et al., 2018).

• Pseudo-label and annotation enrichment: Graph-based label propagation or clustering generates fine-grained pseudo-labels from coarse user annotations (e.g. scribbles or bounding boxes), often via convex energy minimization on an affinity graph (Luo et al., 2018).
• Progressive residual learning: Initial coarse predictions are further refined by learning residuals over the coarse outputs, guided by color or other auxiliary modalities (Liu et al., 2020).
• Joint or iterative optimization: Models are trained sequentially for each granularity, or jointly for all; sometimes, iterative bootstrapping refines pseudo-labels using classifier predictions (Mekala et al., 2021).

4. Application Domains and Empirical Outcomes

Coarse-to-fine supervision is applied broadly:

Domain	Example(s)	Coarse Stage	Fine Stage	Empirical Benefit
Semantic Segmentation	(Islam et al., 2017, Hu et al., 2018, Luo et al., 2018)	Low-res segmentation, annotation enrichment	High-res labels, graph-propagated fine masks	+1–5 mIoU points, state-of-the-art with weak labels
Fine-grained Classification	(Eshratifar et al., 2019, Zhao et al., 27 Feb 2025, Bukchin et al., 2020)	Coarse class separation	Clustered pseudo-fine classes, angular dispersion	+2–4% accuracy, improved few-shot adaptation
Mathematical Reasoning	(Hu et al., 23 Jan 2025)	Macro-step reward, merged substeps	Micro-step reward, atomic steps	+1–3.4 pp BoN@64 on GSM-Plus, MATH500
NER and Text Class.	(Xue et al., 2020, Mekala et al., 2021)	Entity span/anchor detection	Fine-typed clustering, LM-based data generation	+2–5 F1 points, effective in low-resource settings
3D Reconstruction	(Lee et al., 8 Dec 2025)	Depth alignment	Gradient/normal alignment	SOTA mesh/fidelity, improved relighting
Medical Segmentation	(He et al., 15 Nov 2025, Moris et al., 2022, Wang et al., 2023)	Coarse mask (SAM/UNet), region-level	1D point regression, pixel-level	Robustness in low-data, ±4–10% Dice gains

These results robustly confirm the efficacy of the coarse-to-fine paradigm, especially where fine annotations are costly or training data are limited.

5. Analysis of Benefits, Limitations, and Design Choices

Benefits

• Label efficiency: Aggressively reduces annotation time (e.g. scribbles instead of per-pixel masks (Luo et al., 2018)) while achieving near full-supervised performance.
• Optimization smoothing: Coarse-to-fine deep supervision acts as an auxiliary curriculum, easing gradient flow and facilitating convergence (Islam et al., 2017, Hu et al., 2018).
• Error correction: Fine-level stages correct coarse localization or labeling errors; e.g., 1D normal sampling in lithography (He et al., 15 Nov 2025).

Limitations

• Diminishing returns with abundant data: On large, diverse datasets, single-stage fully supervised models may match or exceed coarse-to-fine pipelines (Moris et al., 2022).
• Sensitivity to design choices: The choice of merging operation, cluster size, or pseudo-label threshold can impact final accuracy (Yang et al., 2020, Luo et al., 2018).
• Pipeline complexity: Multi-stage architectures and iterative refinements increase implementation and compute burden.

A plausible implication is that coarse-to-fine frameworks yield the greatest advantage when annotation resources are limited, task complexity is high, or fine-grained distinctions arise from within broad label classes.

6. Extensions and Cross-Domain Generalization

Recent work integrates coarse-to-fine strategies with:

• Self-supervised and self-distillation protocols: Instance-level pseudo-labeling for MIL (Wu et al., 4 Feb 2025); self-picking for NER (Xue et al., 2020).
• Cross-modal and multimodal learning: Staged alignment in speech-image or vision-language retrieval (Zhou et al., 2024).
• Domain adaptation: Coarse region-level adaptation facilitates robust transfer across domains, especially in medical imaging and polyp segmentation (Wang et al., 2023).
• 3D and physically-based modeling: Bidirectional coarse-to-fine supervisions between different 3D representations for faithful geometry and BRDF learning (Lee et al., 8 Dec 2025).

This breadth of application underscores coarse-to-fine supervision as a general methodological axis—applicable whenever hierarchical structure, label scarcity, or staged optimization can be leveraged for better learning or transfer.

---

References (arXiv IDs):

• (Islam et al., 2017) Label Refinement Network for Coarse-to-Fine Semantic Segmentation
• (Hu et al., 2018) Progressive refinement: a method of coarse-to-fine image parsing using stacked network
• (Dong et al., 2018) Coarse-to-Fine Decoding for Neural Semantic Parsing
• (Luo et al., 2018) Coarse-to-Fine Annotation Enrichment for Semantic Segmentation Learning
• (Eshratifar et al., 2019) Coarse2Fine: A Two-stage Training Method for Fine-grained Visual Classification
• (Yang et al., 2020) Towards Cross-Granularity Few-Shot Learning: Coarse-to-Fine Pseudo-Labeling with Visual-Semantic Meta-Embedding
• (Xue et al., 2020) Coarse-to-Fine Pre-training for Named Entity Recognition
• (Bukchin et al., 2020) Fine-grained Angular Contrastive Learning with Coarse Labels
• (Liu et al., 2020) FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion
• (Mekala et al., 2021) Coarse2Fine: Fine-grained Text Classification on Coarsely-grained Annotated Data
• (Moris et al., 2022) Assessing Coarse-to-Fine Deep Learning Models for Optic Disc and Cup Segmentation in Fundus Images
• (Wang et al., 2023) Unsupervised Adaptation of Polyp Segmentation Models via Coarse-to-Fine Self-Supervision
• (Zhou et al., 2024) Coarse-to-fine Alignment Makes Better Speech-image Retrieval
• (Hu et al., 23 Jan 2025) Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning
• (Wu et al., 4 Feb 2025) LadderMIL: Multiple Instance Learning with Coarse-to-Fine Self-Distillation
• (Zhao et al., 27 Feb 2025) Twofold Debiasing Enhances Fine-Grained Learning with Coarse Labels
• (He et al., 15 Nov 2025) LithoSeg: A Coarse-to-Fine Framework for High-Precision Lithography Segmentation
• (Lee et al., 8 Dec 2025) COREA: Coarse-to-Fine 3D Representation Alignment Between Relightable 3D Gaussians and SDF via Bidirectional 3D-to-3D Supervision