Visual Granularity Sequence Construction

• Visual granularity sequence construction is a method that organizes images into a hierarchical series of tokens, progressing from coarse global layouts to fine local details.
• The NVG framework leverages multi-granularity tokenization and iterative inpainting, ensuring precise structure mapping and effective content refinement at each stage.
• Empirical evaluations demonstrate that this approach outperforms traditional flat quantization models by achieving lower FID scores and higher inception metrics, enhancing controllability and interpretability.

Visual granularity sequence construction refers to the design, extraction, and organization of visual representations at controlled levels of abstraction or “granularity,” enabling hierarchical, interpretable, and often progressive processing of image data. Modern frameworks realize visual granularity sequences via structured tokenizations, multi-resolution geometric or semantic representations, and staged inference or synthesis, resulting in downstream benefits such as improved controllability, efficiency, and interpretability.

1. Theoretical Foundations of Visual Granularity Sequences

A visual granularity sequence is a structured arrangement of image representations that span from coarse, global attributes (e.g., background-foreground segmentation, object bounding boxes) to fine, localized details (e.g., textures, edges, small object parts). Unlike approaches relying solely on spatial downsampling or a flat pixel/raster encoding, granularity-aware frameworks explicitly manipulate the number of unique visual tokens, representations, or partitions per stage to achieve hierarchical abstraction.

The Next Visual Granularity (NVG) framework provides a formal and practical basis for this notion. In NVG, an image is recast into a sequence $\{(c_0, s_0), (c_1, s_1), \ldots, (c_K, s_K)\}$, where each $c_i$ is a set of unique content tokens and each $s_i$ is a structure map (cluster assignment per spatial location), with $n_i = |c_i|$ controlling the granularity at stage $i$ ($n_{i-1} > n_i$). The process begins at the coarsest abstraction (lowest $n_i$) and repeatedly refines the image by increasing the number of distinct tokens, producing a hierarchical, layered representation (Wang et al., 18 Aug 2025).

This overcomes the ambiguities induced by flat quantization (e.g., traditional VQ-VAE or autoregressive models) and builds in interpretability by decomposing global structure before injecting local detail.

2. Methodologies for Hierarchical Construction

a. Multi-Granularity Tokenization and Clustering

NVG models leverage a learned quantized feature space with a codebook $\mathcal{V}\in\mathbb{R}^{n\times e}$ and latent representation $Z\in\mathbb{R}^{h\times w\times e}$. At each stage, a graph-based agglomerative clustering (using $\ell_2$ distance) yields a structure map $s_i\in\{0,1,\ldots,n_i-1\}^{h\times w}$ and corresponding content tokens $c_i$. The process merges similar regions at each step, defining the composition of subsequent stages.

Mathematically, this can be formalized as:

• Structure map: assign cluster ids to spatial positions based on token similarity.
• Content refinement: the quantization error at each stage $$R_i = Z - \sum_{j=0}^{i-1} \phi_j \circ a(c_j, s_j)$$ is predicted and minimized, using a content generator $f_c$ and structure generator $f_s$.

This methodology produces a sequence from rough segmentation (large clusters, low $n_i$) to indices approaching pixel-wise granularity.

b. Iterative Generation and Layerwise Decoding

NVG adopts an iterative painting process: at each stage, a structure generator (often using rectified flow for stochastic denoising) predicts the structure map conditioned on the current canvas, and a content generator predicts the new unique content tokens and their spatial placement.

• Structure generator uses a noise-injection and v-prediction mechanism, e.g., $z_s(t) = t\cdot\epsilon + (1-t)\cdot s_e$ where $s_e$ is the encoded structure. The process advances via progressive inpainting, refining known regions at each hierarchy.
• Content generator minimizes a combined MSE and cross-entropy loss, with structure-aware positional encodings (extending RoPE) that preserve cluster assignments, facilitating explicit correspondence between structural tokens and output pixels.

This architecture allows fine-grained control and direct intervention at each generative stage.

3. Performance Evaluation and Empirical Significance

The granularity sequence approach yields state-of-the-art performance on benchmark generative tasks:

Model	FID ↓	Inception Score ↑	Recall ↑
VAR baseline	3.30	-	-
NVG (comparable)	3.03	higher	higher

Empirical results demonstrate:

• Consistently lower Fréchet Inception Distance (FID) than corresponding visual autoregressive (VAR) baselines.
• Improved Inception Score (IS) and recall, indicating better diversity and fidelity of generated samples.

This is attributed to the explicit modeling of hierarchical structure, which reduces ambiguity in early layers, constrains generative trajectories, and avoids common artifacts associated with flat or purely spatially-downsampling tokenizations.

4. Applications and Downstream Implications

The NVG framework’s explicit granularity control has multifaceted practical implications:

• Conditional Generation & Editing: Given an annotated structure map (e.g., from segmentation), images can be generated or edited by specifying the high-level structure before realizing details, affording interactive design capabilities.
• Medical/Scientific Visualization: Ensures the invariance of critical macroscopic regions (e.g., lesions, organ boundaries) through the hierarchy before interpretable refinement.
• Video Generation & Consistent Animation: By tracking the evolution of structure maps across frames, NVG enables temporally coherent, physically plausible sequences.
• Spatial Reasoning Tasks: The sequence formalism naturally integrates with models requiring global-to-local hierarchical reasoning, such as scene infilling, semantic layout transfer, or guided synthesis.

A plausible implication is that similar approaches will enable task- or domain-specific structural control in other generative modalities (e.g., text-to-image, video, or multi-modal synthesis).

5. Technical Challenges and Model Innovations

a. Structure Embedding and Decoding

NVG designs a structure embedding $s_e(s, i)$ formatted by adding bits per hierarchy level, enabling both efficient encoding and accurate recovery of structure at each step:

$$s_e(s,i)_j = 2\cdot(\left\lfloor \frac{s}{2^{i-1-j}}\right\rfloor \bmod 2)\cdot \mathbb{1}_{j<i} + \mathbb{1}_{j \ge i}$$

The original structure label $s$ can be perfectly decoded from this embedding scheme, which facilitates lossless, bit-efficient structural control.

b. Inpainting and Progressive Refinement

Generating the complete structured map at once presents a cold-start challenge. To mitigate this, NVG employs progressive inpainting: the structure generator updates incremental regions per iteration, allowing the system to boot from an “empty” (unknown) canvas, incorporate low-level cues, and avoid structure collapse.

c. Loss Functions and Training

A composite loss—sum of pixel-space MSE (on final canvas) and cross-entropy (over token assignment)—is minimized:

$$\ell(x_i) = ||X - f_c(x_i)||^2_2 + \mathrm{CrossEntropy}(\hat{c}_i, c_i)$$

Structure-aware attention employs specialized positional encoding to ensure region tokens dominate attention within clusters, stabilizing hierarchical reasoning.

6. Limitations and Prospects for Future Research

Remaining challenges include:

• Structure generator initialization: fully binary structure generation in a single pass remains hard, necessitating sophisticated denoising flows.
• Balancing fidelity and diversity: requires dynamic tuning of TOP_P sampling and classifier-free guidance scaling.
• Extending granularity: while current frameworks support cluster-based hierarchy, domain-specific annotations or complex segmentation maps could further generalize the approach.

Future directions highlighted in the paper include:

• Region-Aware Generation: Leveraging domain-specific segmentation or semantic region annotations for controllable, interpretable synthesis.
• Physically-Aware Video Synthesis: Ensuring structural continuity and object permanence across frames.
• Hierarchical Spatial Reasoning: Integrating granularity sequences with global-local reasoning tasks.

This suggests that the visual granularity sequence paradigm is likely to see increasing adoption in multimodal and physically-constrained generation settings.

---

In summary, visual granularity sequence construction, as realized in the Next Visual Granularity (NVG) framework (Wang et al., 18 Aug 2025), decomposes image synthesis or representation into staged, iterative construction from coarse global layouts to fine local details, using explicit hierarchical clustering and specialized generation modules. This structured approach yields strong empirical performance, facilitates controllable and interpretable image generation, and offers a compelling foundation for future advances in generative modeling, spatial reasoning, and interactive visual design.