Coarse-to-Fine Alignment

• Coarse-to-Fine Alignment is a hierarchical strategy that first reduces the search space with a global, low-resolution filter and then refines the alignment with detailed, local analysis.
• It is applied in various fields like video grounding, domain adaptation, and generative modeling, significantly enhancing computational efficiency and robustness.
• Empirical and theoretical studies show that this approach reduces noise, avoids local minima, and improves model interpretability across complex tasks.

Coarse-to-Fine Alignment

Coarse-to-fine alignment refers to a hierarchical strategy for aligning representations, features, or structures across data modalities, spatial or temporal scales, or abstraction levels. This paradigm appears frequently in machine learning, computer vision, natural language processing, and multimodal retrieval, where alignment objectives are challenging due to the presence of noise, large search spaces, weak supervision, or the need for high precision. The approach decomposes alignment into (1) an initial coarse (global, low-resolution, or high-abstraction) phase designed to reduce search space and efficiently prune irrelevant candidates, and (2) a fine (local, high-resolution, or detailed) phase that precisely aligns or matches the targets selected in the earlier stage. Empirically, coarse-to-fine alignment reduces computational cost, enhances robustness to outliers and distractors, and yields higher accuracy across a broad range of tasks.

1. Core Principles and Motivations

Coarse-to-fine alignment assumes that the alignment problem contains substructures that can be efficiently screened or approximated at a higher (coarser) level, while finer details require expensive computation or more discriminative modeling. Central motivations include:

• Search space reduction: The coarse stage eliminates a large fraction of irrelevant candidates, allowing the fine stage to focus on promising subsets (Hou et al., 2022, Hou et al., 2022).
• Robustness to distractors and noise: Coarse filters suppress spurious alignments early, making fine-level alignment more reliable (Mei et al., 2015, Huang et al., 2015).
• Hierarchical modeling: Many tasks (e.g., semantic segmentation, multi-modal matching, video grounding) inherently benefit from progressively narrowing focus—from global to local, semantic to spatial, or feature- to instance-level (Wang et al., 2023, Zhou et al., 15 Aug 2024, Qian et al., 2020).
• Improved optimization: Coarse-to-fine decompositions induce better inductive bias, avoid local minima, and facilitate stable training, especially in weakly supervised or unsupervised settings (Chen et al., 2022, Chen et al., 2021).

2. Methodological Frameworks

Coarse-to-fine alignment can be instantiated in diverse applications with different concrete mechanisms. Representative forms include:

• Sliding-window temporal filtering: For temporal grounding in long videos, CONE (Hou et al., 2022, Hou et al., 2022) uses a sliding window scheme, where coarse, query-guided window selection first prunes the video, followed by fine proposal-level ranking within selected windows.
• Selective feature gating: In encoder–aligner–decoder models for text generation from structured data, a pre-selector computes coarse salience probabilities per record, masking downstream fine-grained attention (Mei et al., 2015).
• Prototype-based hierarchical branching: In action quality assessment, CoFInAl (Zhou et al., 22 Apr 2024) uses learnable coarse-grade prototypes and fixed ETF-based sub-grade classifiers, mirroring human hierarchical assessment.
• Sequential regression and patch refinement: For tasks such as facial landmark localization, an initial holistic predictor is refined iteratively with networks attending to local multi-scale patches (Huang et al., 2015).
• Domain adaptation by staged divergence minimization: CALI (Chen et al., 2022) and related approaches (Ma et al., 2021, Tang et al., 2021) begin with global (domain-level, photometric) alignment, then enforce class-conditional (fine) feature distribution regularization.
• Feature-space alignment in generative models: In diffusion networks for text-to-image or person-image synthesis, initial global semantic alignment is followed by spatial or local attribute binding at finer stages (Jiang et al., 2023).
• Attention-based cross-view fusion: In cross-view or multimodal GANs, coarse attention modules reweight input branches; fine distillation stages suppress residual noise and amplify orthogonal, view-consistent features (Qiao et al., 19 Aug 2024).

3. Mathematical Formulations

The mathematical realization of coarse-to-fine alignment is tailored to the task but often follows a two-stage or cascaded composition of objectives, losses, or architectural modules. Examples include:

• Contrastive learning at coarse and fine levels: CONE (Hou et al., 2022) combines a window-level NCE loss,

$$L_{coarse} = -\sum_{i=1}^N \log \frac{\exp(\mathrm{sim}(C_v^i,C_q^i)/\tau)}{\sum_{j=1}^N \exp(\mathrm{sim}(C_v^i,C_q^j)/\tau)},$$

and a fine proposal-level contrastive loss,

$$L_{fine} = -\log \frac{\exp(F_v^{pos} \cdot F_q / \tau)}{\sum_j \exp(F_v^j \cdot F_q / \tau)}.$$

• Two-stage salience and attention: The pre-selector computes

$p_j = \sigma(q^{\top} \tanh(Pm_j)),$

then modulates fine-grained attention,

$$\alpha_{t,j} = \frac{p_j w_{t,j}}{\sum_{k=1}^N p_k w_{t,k}}.$$

(Mei et al., 2015)

• Adversarial games for domain adaptation: Coarse domain alignment is formulated as

$$L_{coarse}(G, D) = \max_\psi \min_\phi V_1(G^\phi, D^\psi),$$

and fine class-conditional alignment as

$$L_{fine}(G, C_1, C_2) = \max_{\theta_1, \theta_2} \min_\phi V_2(G^\phi, C_1^{\theta_1}, C_2^{\theta_2}).$$

(Chen et al., 2022)

• Hierarchical classification with prototypes: CoFInAl (Zhou et al., 22 Apr 2024) reconstructs scores as

$\hat s_i = \hat s_C^i S_C + \hat s_F^i S_F,$

with separate cross-entropy and regression losses at each level.

4. Applications and Empirical Impact

Coarse-to-fine alignment has shown significant gains across several domains:

• Temporal video grounding: CONE outperforms vanilla proposal models by 3–4 percentage points in Recall@1 on benchmarks like MAD and Ego4D-NLQ, with 2–15× acceleration (Hou et al., 2022, Hou et al., 2022).
• Selective content generation: On WeatherGov, the coarse-to-fine aligner attains F-1 ≈ 76.3%, a ~12% relative gain over prior art in content selection, with 59% improvement in BLEU generation (Mei et al., 2015).
• Unsupervised domain adaptation: CALI achieves mIoU improvements up to 8 percentage points over both domain- and class-alignment-only baselines and avoids negative transfer (Chen et al., 2022). Similarly, CFContra (Tang et al., 2021) and photometric-triplelet pipelines (Ma et al., 2021) report analogous gains.
• Multimodal retrieval: Unified coarse-to-fine alignment models outperform CLIP-only baselines in video-text retrieval and speech-image retrieval, offering up to +4.2 points R@1 improvement (Wang et al., 2023, Zhou et al., 15 Aug 2024).
• Medical imaging and dense registration: Three-stage cell-level registration in CORE delivers sub-10μm error rates across stains and imaging modalities, setting new benchmarks in multi-modal histopathology WSI alignment (Nasir et al., 5 Nov 2025).

Task	Coarse-to-Fine Method	Performance Gain
Temporal Video Grounding	CONE (Hou et al., 2022)	+3–4 pp R@1, 2–15× speed
Selective Gen (WeatherGov)	Coarse-to-fine aligner (Mei et al., 2015)	F-1↑ by ~12%, BLEU↑ 59%
UDA Semantic Segmentation	CALI (Chen et al., 2022)	mIoU +6–8 points
Video-Text Retrieval	UCoFiA (Wang et al., 2023)	R@1 +2.4–1.3 pp
Nuclei-Level Image Reg	CORE (Nasir et al., 5 Nov 2025)	SOTA TRE <10μm

Coarse-to-fine pipelines have also become standard in facial landmark detection (Huang et al., 2015, Shao et al., 2016), anomaly detection (Zheng et al., 2021), generative alignment (Jiang et al., 2023), and high-resolution video fusion (Chen et al., 2021).

5. Ablation, Optimization, and Theoretical Insights

Ablations consistently demonstrate that removing either the coarse or fine stage degrades performance. For example, in CONE, omitting coarse contrastive losses returns accuracy to that of the vanilla base VTG model; skipping fine-level ranking costs about 2–3 percentage points (Hou et al., 2022, Hou et al., 2022). In domain adaptation, skipping coarse/domain alignment (DA) or fine/class alignment (CA) each sharply degrades mean IoU (Chen et al., 2022, Ma et al., 2021).

Analysis in CALI establishes that the coarse (domain-level) divergence upper bounds the finer (class-wise) divergence, ensuring that an alternating first-coarse-then-fine process is both stable and avoids negative transfer. Empirical results confirm that purely fine-level adaptation is prone to divergence, while coarse-to-fine training produces monotonic improvements (Chen et al., 2022). A plausible implication is that hierarchical alignment regularizes learning, constraining feature evolution at each stage.

6. Interpretability and Modularity

Coarse-to-fine methods also offer enhanced interpretability:

• Prototype interpretability: In CoFInAl, coarse-grade prototypes align with qualitative human categories, while ETF sub-grade classification yields maximally separated, margin-optimal predictions (Zhou et al., 22 Apr 2024).
• Attention heatmaps: In selective generation, heatmaps over records clearly demonstrate sparse, task-relevant selection at both coarse and fine stages (Mei et al., 2015).
• Region masks and dense captions: In diffusion-based image generation, fine alignment via object tags, masks, and dense region captions clarify how the model binds semantic text to localized image regions (Jiang et al., 2023).
• Hierarchical feature visualization: Medical image registration pipelines visually decompose sample alignment from tissue shape (coarse) down to nuclei-level (fine), allowing debugging and refinement at each scale (Nasir et al., 5 Nov 2025).

7. Limitations and Future Directions

Despite clear empirical success, coarse-to-fine alignment is not universally optimal. Limitations include reliance on suitable hierarchy design (the definition of "coarse" and "fine" is domain dependent), potential information loss if coarse filtering is too aggressive, and increased architectural complexity. Open areas include automatic discovery of coarse-to-fine hierarchies, dynamic multi-stage reward fusion (especially in generative models (Jiang et al., 2023)), and extending the paradigm to new domains such as cross-lingual transfer, massive multimodal retrieval, or ultra-high-resolution 3D image registration.

A plausible implication is that, as models and data complexity increase, hierarchical alignment strategies combining global and local, semantic and structural cues will remain critical for scaling both accuracy and computational efficiency.