Three-Stage Multi-Granularity Design

• Three-Stage Multi-Granularity Design is a hierarchical framework that breaks down complex tasks into three interrelated levels to capture fine, intermediate, and global information.
• It incorporates domain-specific strategies such as adaptive pruning, self-attention, and reinforcement learning to optimize both neural and hardware architectures.
• Empirical results show significant gains in efficiency, accuracy, and robustness, with improvements like up to 40% parameter reduction and enhanced multi-scale performance.

A three-stage multi-granularity design is a class of architectural or training frameworks that explicitly decomposes a learning task into three sub-tasks or modules, each corresponding to a distinct level of granularity in data representation, supervision, or model substructure. Across diverse domains—including neural architecture search, time series modeling, large-scale design space exploration, multimodal recognition, and end-to-end sequence modeling—this approach leverages hierarchical structure to enhance the expressiveness, robustness, and optimization of deep learning systems. The technical details of each stage and granularity are tailored to the application domain, but the conceptual foundation remains the systematic incorporation of multi-scale information flow or multi-level search/optimization objectives.

1. Foundations of Three-Stage Multi-Granularity Design

The core principle of three-stage multi-granularity design is to segment a complex learning or search problem into three interacting sub-stages, each aligned with a specific granularity:

• Stage 1: Addresses the finest granularity (e.g., local features, character-level units, hardware sub-blocks), typically focusing on low-level structure, fine details, or local correlations.
• Stage 2: Operates at an intermediate granularity (e.g., patch- or cell-level, subword/BPE units, bonded pairs, mid-level architectural motifs), incorporating longer-range dependencies or semantic structure.
• Stage 3: Engages the coarsest granularity (e.g., global context, attention-based decoding, cross-scale interactions, policy optimization for high-level objectives), often involving information aggregation, hierarchical fusion, or holistic optimization.

This staged approach is instantiated in various research fields:

Domain	Stage 1 Granularity	Stage 2 Granularity	Stage 3 Granularity
Time-series modeling	Local cross-channel patches	Multi-scale embeddings	Intra-/inter-scale attention
NAS	Op/Filter/Weight	Sub-network level	Progressive re-evaluation
Hardware DSE	Architecture definition	Parametric sweep	Mapping strategies
Multitask OCR	Basic pre-training	Bond/coord auxiliary	RL policy optimization
Speech seq2seq	Char encoder (CTC)	BPE encoder (CTC)	Attention decoder (CE)

A key advantage is enhanced hierarchical compositionality, enabling models to capture dependencies not accessible at a single level and improving both representation and search efficiency (Wang et al., 2024, Liu et al., 2023, Qu et al., 27 Mar 2025, Zhang et al., 21 Nov 2025, Garg et al., 2019).

2. Three-Stage Multi-Granularity in Neural and Hardware Architectures

Three-stage multi-granularity strategies are prominent in both neural network and hardware architecture optimization:

Medformer for Medical Time Series

Medformer utilizes a pipeline where:

1. Cross-Channel Patching: For $$\mathbf{x}_{\mathrm{in}} \in \mathbb{R}^{T \times C}$$, the sample is decomposed into $N_i = \lceil T/L_i \rceil$ non-overlapping cross-channel patches for each $L_i$, immediately encoding fine-scale temporal and inter-channel features.
2. Multi-Granularity Embedding: Each patch $\mathbf{x}_p^{(i)}$ is projected into a $D$-dimensional space; positional and scale-specific embeddings are added, and task-specific augmentations enforce representation robustness.
3. Two-Stage Self-Attention: Separate intra-granularity (within-scale) and inter-granularity (across scales) self-attention allow routers to summarize local/global structure and to coordinate scale fusion. Output embeddings are then pooled and classified. This design achieves state-of-the-art results on multiple health datasets, outperforming CNN and prior transformer-based baselines (Wang et al., 2024).

Multi-Level Hardware DSE

MLDSE formalizes design space exploration as:

1. Modeling: Recursive hardware IR with SpaceMatrix/SpacePoint, capturing diverse multi-level hardware hierarchies.
2. Mapping: Spatiotemporal mapping IR, primitives for tiling, parallel task assignment, communication, and synchronization, enabling detailed mapping strategies to be explored.
3. Simulation: Task-level event-driven simulation and a hardware-consistent scheduler, supporting efficient resource contention resolution and deep pipeline evaluation.

Search proceeds by nesting loops over architectures, parameters, and mapping iterations, yielding a three-tier combinatorial DSE process (Qu et al., 27 Mar 2025).

3. Multi-Granularity in Neural Architecture Search and Training

Multi-Granularity Architecture Search (MGAS)

MGAS leverages three integrated design axes:

1. Granular Search Spaces: Simultaneously searches operation-level (candidate ops), filter-level (output channel weighting), and weight-level (fine per-weight masking) units. Architectures are parameterized by $\alpha, \beta, \omega$ at each respective level.
2. Learned Discretization and Adaptive Pruning: At every granularity, masking thresholds $t_\cdot$ are learned, and units below threshold are pruned, with masking functions $M_\alpha$, $M_\beta$, $M_\omega$ controlling remaining ratios. The network dynamically balances complexity and performance.
3. Progressive Multi-Stage Re-evaluation: The super-net is incrementally divided into subnets along depth, each staged for sequential optimization and adaptive pruning. Regrowth mechanisms allow previously pruned units to reenter to mitigate over-pruning bias.

MGAS achieves state-of-the-art size–accuracy trade-offs and substantial memory savings compared to single-stage or fixed-ratio baselines (Liu et al., 2023).

Multi-Stage Training for Sequence Models

Three stages in online attention-based AED models are:

1. Character-CTC Encoder Pre-training: Stacked ULSTM + CTC loss on character targets, with layer-wise growth and max-pooling to stabilize early training.
2. BPE-CTC Encoder Training: Extend with additional ULSTM layers, high time-reduction, joint CTC objectives for character and BPE targets, and parameter freezing for staged bootstrapping.
3. Attentional Decoder Training: Switch to standard attention-based decoder (MoChA), train with cross-entropy loss over BPE units. The multi-task CTC loss and scheduling of “hand-off” transitions across stages are key for convergence and strong error reductions (Garg et al., 2019).

4. Multi-Granularity Contrastive and Multitask Learning

MicRec demonstrates a plug-and-play three-stage structure for item-based contrastive learning in recommendation:

1. Feature-Level Item CL: Fine-grained feature augmentations (field dropouts), InfoNCE loss for intra-item invariance.
2. Semantic-Level Item CL: Coarse grouping by category, title, or content embeddings; InfoNCE loss for inter-item semantic similarity.
3. Session-Level Item CL: Behavioral co-occurrence mining; InfoNCE loss for learning session-based, global correlations.

These objectives are combined in a multi-task loss, each term modulated by a hyperparameter, supporting domain-agnostic integration with standard retrieval models (Xie et al., 2022).

In MolSight's OCSR system:

1. Large-Scale SMILES Pre-training: Modeling on massive, weakly supervised SMILES-only data gives low-level perceptual grounding.
2. Multi-Granularity Fine-Tuning: Auxiliary heads for chemical bond classification and atom localization refine the backbone for structural and spatial awareness.
3. Reinforcement Learning Post-Training: Group Relative Policy Optimization (GRPO) fine-tunes sequence generation using trajectory-level rewards, emphasizing stereochemical accuracy.

This threefold structure yields substantial accuracy gains for stereochemistry extraction across challenging molecular datasets (Zhang et al., 21 Nov 2025).

5. Key Advantages and Quantitative Outcomes

Three-stage multi-granularity designs provide multiple quantifiable gains across domains:

• Performance: Medformer outperforms 10 baselines on all MedTS metrics, MGAS achieves up to 40% parameter reduction at the same accuracy, MolSight raises stereo exact match by 16.1% on USPTO-stereo, and three-stage AED training reduces WER by 29–36% relative to prior single-granularity baselines (Wang et al., 2024, Liu et al., 2023, Zhang et al., 21 Nov 2025, Garg et al., 2019).
• Resource Efficiency: MGAS halves memory use compared to non-staged methods, while MLDSE dramatically shrinks DSE search space via composability and hierarchical pruning (Liu et al., 2023, Qu et al., 27 Mar 2025).
• Robustness: Adaptive, stage-wise mechanisms (e.g., regrow in MGAS, router attention in Medformer, auxiliary multi-objective heads in MolSight) ensure learned structures are robust to overfitting, underfitting, and catastrophic forgetting.
• Flexibility: Multi-granularity designs are agnostic to data modality (time-series, images, graphs, sequences) and task (classification, generation, DSE), supporting direct transfer (e.g. MolSight's encoder for MoleculeNet) (Zhang et al., 21 Nov 2025).

6. Generalization and Design Patterns

Designing with three distinct stages and granularities is a recurring pattern for complex, hierarchical domains. The empirical evidence shows that staged optimization enhances both capacity and interpretability by:

• Isolating simpler sub-problems for stable learning.
• Explicitly encoding multi-scale or multi-level semantic, spatial, or structural relations.
• Allowing separate or joint optimization objectives for each granularity.
• Enabling staged transfer or hand-off of parameters, which improves gradient flow and accelerates convergence.

A plausible implication is that as model/data complexity increases, further granularity decomposition—beyond three stages—may provide diminishing returns or introduce coordination overhead. Empirical results overwhelmingly support three-stage designs as a practical optimum for balancing efficiency, accuracy, and manageability in contemporary deep learning systems (Wang et al., 2024, Liu et al., 2023, Qu et al., 27 Mar 2025, Xie et al., 2022, Zhang et al., 21 Nov 2025, Garg et al., 2019).