H-SPAM: Hierarchical Superpixel Anything Model

Abstract: Superpixels offer a compact image representation by grouping pixels into coherent regions. Recent methods have reached a plateau in terms of segmentation accuracy by generating noisy superpixel shapes. Moreover, most existing approaches produce a single fixed-scale partition that limits their use in vision pipelines that would benefit multi-scale representations. In this work, we introduce H-SPAM (Hierarchical Superpixel Anything Model), a unified framework for generating accurate, regular, and perfectly nested hierarchical superpixels. Starting from a fine partition, guided by deep features and external object priors, H-SPAM constructs the hierarchy through a two-phase region merging process that first preserves object consistency and then allows controlled inter-object grouping. The hierarchy can also be modulated using visual attention maps or user input to preserve important regions longer in the hierarchy. Experiments on standard benchmarks show that H-SPAM strongly outperforms existing hierarchical methods in both accuracy and regularity, while performing on par with most recent state-of-the-art non-hierarchical methods. Code and pretrained models are available: https://github.com/waldo-j/hspam.

• The paper presents a novel unified framework (H-SPAM) that integrates object priors and deep features for precise hierarchical superpixel segmentation.
• The method employs a two-phase agglomerative merging strategy that ensures strict nestedness and robust segmentation accuracy across standard datasets.
• User-driven attention modulation refines segmentation by balancing region regularity with semantic boundary adherence for practical vision applications.

H-SPAM: A Unified Framework for Hierarchical Object-Based Superpixels

Introduction and Motivation

Superpixel segmentation is a key preprocessing step in many computer vision tasks, enabling a compact, region-based representation of images. Traditionally, the trade-off between superpixel regularity and accuracy has limited the utility of these methods, particularly for applications requiring semantic object alignment or multi-scale representations. While recent neural approaches harness deep features for better semantic alignment, they often relax regularity, resulting in fragmented and irregular shapes. Critically, most prior art provides only non-hierarchical partitions, restricting their deployment in pipelines requiring consistently nested multi-scale segmentations.

The H-SPAM framework—Hierarchical Superpixel Anything Model—addresses this limitation by introducing an object-prior-guided, hierarchical, deep-learning-based superpixel construction. It enforces both high segmentation accuracy and region regularity while guaranteeing perfect nestedness across all scales. H-SPAM unifies the strengths of foundation models for object boundary detection with an efficient agglomerative hierarchical process for region merging and introduces modulation via attention and user interaction.

Figure 1: Hierarchical superpixel segmentation example. The proposed H-SPAM method generates very regular and easily interpretable superpixels that are aligned with the image objects through the hierarchy, compared to HHTS.

Methodological Framework

Data Representation and Feature Integration

H-SPAM starts by generating a fine initial superpixel partition using deep features and object priors obtained from large-scale segmentation foundation models (notably SAM). Each initial region (superpixel) is described by a vector comprising color (LAB), spatial coordinates, and high-dimensional deep features. The associated object prior map divides the image into $M$ non-overlapping semantically meaningful entities.

Hierarchical Object-Guided Agglomeration

The core of H-SPAM is a two-phase agglomerative hierarchical merging on the region adjacency graph (RAG):

• Phase 1: Intra-object Merges: Only regions belonging to the same object are merged, under a hard constraint that prohibits inter-object edges. The merging cost fuses appearance (color, deep) and spatial features, adaptive to region scale. This preserves object boundaries through the hierarchy's finer levels.
• Phase 2: Inter-object Merges: After all intra-object merges are exhausted (each object reduced to a single region), inter-object merges are allowed. Here, the cost is modulated by region size (akin to Ward’s criterion), and only feature similarity is used, resulting in nested merges consistent until the root.

At any desired scale, an arbitrary number $K$ of superpixels is extracted by unrolling the first $(N_f - K)$ merges.

Attention Modulation and User Interaction

H-SPAM introduces flexible attention-based modulation strategies:

• Semantic Attention: Attention maps (e.g., DINO-based) or user-provided inputs can be averaged within objects to delay merges of salient regions, controlling the preservation of detail for downstream or annotation-centric tasks.
• Interactive Segmentation: Users can directly influence local superpixel density within objects by placing clicks, which further modulates the merging cost to support application-specific focus.

Figure 2: Illustration of the attention modes. With our object-based framework (middle row), the attention can be averaged within objects to provide a cleaner guide for the merging process. The bottom row shows the user interactive mode, where red/white crosses lead to fewer/more superpixels in an object, offering a complementary alternative to our multi-scale segmentation method.

Quantitative and Qualitative Analysis

Strong Numerical Performance

H-SPAM achieves substantial performance gains compared to both classical and contemporary hierarchical superpixel approaches:

• Accuracy and Regularity: Quantitative analysis on BSD, NYUv2, and SBD datasets demonstrates statistically significant improvement in Achievable Segmentation Accuracy (ASA) and Shape Regularity Criteria (SRC) over prior hierarchical baselines, establishing H-SPAM as the most accurate and most regular method.
• Hierarchy Consistency: H-SPAM guarantees strict nestedness (hierarchiness score of 1), essential for multi-scale processing.
• Efficiency: Hierarchies with up to 1250 levels are constructed in under 0.5s on standard resolution images.

Figure 3: Quantitative comparison of H-SPAM to state-of-the-art hierarchical superpixel methods. H-SPAM is the most accurate while being also the most regular hierarchical method on the three datasets.

• Non-hierarchical Parity: Despite the hierarchical constraint, H-SPAM matches or surpasses non-hierarchical SOTA methods in both accuracy and regularity, trailing only its own single-scale source SPAM in specific configurations.

Figure 4: Quantitative comparison of H-SPAM to state-of-the-art non-hierarchical superpixel methods on the BSD. Although constrained to be hierarchical, H-SPAM is one of the best performing methods in both accuracy and regularity, preserving the performance of single scale object-based approach.

Qualitative Results

H-SPAM consistently yields regular, easily interpretable regions that align tightly with object boundaries across all tested scales, outperforming all hierarchical competitors, whose outputs frequently exhibit irregularity and imprecise object adherence.

Figure 5: Qualitative comparison between hierarchical methods. Number of superpixels from left to right: 1250, 800, 500, 150, 50. H-SPAM produces regular and easily interpretable regions, that align well with object boundaries compared to other methods.

Figure 6: Qualitative example of H-SPAM for different scales. From left to right: 1250, 500, 150, 50 superpixels. H-SPAM produces perfectly nested, very regular and easily interpretable regions, that align well with object boundaries compared to other methods.

Ablation and Parameter Analysis

Ablation studies validate the contribution of object priors and regularization parameters. Explicit use of object masks during both initial partitioning and hierarchy construction substantially raises segmentation accuracy, particularly at low $K$ (few, large regions). Tuning the spatial regularity parameter $w_{pos}$ offers trade-offs between region regularity and fine-grained adherence.

Figure 7: (a) Importance of the prior mask. Using object priors to guide the hierarchy largely improves the segmentation accuracy. (b) Influence of $w_{pos}$ on the accuracy and regularity.

Implications and Future Directions

H-SPAM’s integration of hierarchical consistency, regularity, and object adherence addresses key limitations in existing superpixel frameworks, directly enhancing annotation workflows, graph-based scene analysis, and hierarchical vision pipelines. The capacity to inject automated or user-driven attention further expands use in domains such as active learning, interactive annotation, and semi-supervised labeling.

Theoretically, the H-SPAM architecture clarifies the advantage of propagating semantic object priors into hierarchical structures, suggesting future fusion with multidomain foundation models or multi-objective clustering paradigms. Improvements in region feature embeddings or data-driven modulations (e.g., per-task learned attention) would likely yield further gains.

Conclusion

H-SPAM establishes a new standard for hierarchical superpixel generation by enforcing object-based boundary consistency, superpixel regularity, and strict multi-scale nestedness, all within a computationally tractable agglomerative framework. Its superior accuracy and flexibility open new directions for hierarchical segmentation in vision and graphics, with immediate practical value for multitask pipelines, annotation tools, and multi-level reasoning models. The public release of code and models facilitates adoption and further benchmarking by the community.

---

Reference:

"H-SPAM: Hierarchical Superpixel Anything Model" (2604.11218)