Beyond Patches: Superpixel Token-based Transformers for Attribute-Specific Fashion Retrieval

Abstract: Attribute-Specific Fashion Retrieval (ASFR) aims to improve fine-grained image retrieval by focusing on specific attributes. However, existing patch-based attention and Transformer methods often misalign with irregular attribute regions and are prone to background noise, limiting their ability to capture subtle, pixel-level microstructures. To tackle these challenges, we propose SuperFashion, the first ASFR framework that adopts superpixel tokens within a Transformer architecture. SuperFashion initially employs an attribute-guided attention mechanism to extract attribute-related features, which in turn guide the cropping of semantically meaningful image regions. Superpixel segmentation is then leveraged on these regions to generate compact, semantically coherent superpixel tokens. By incorporating modality-specific embeddings for both attribute and superpixel tokens, the superpixel token-based Transformer facilitates adaptive interaction and fusion, thereby enhancing attribute localization and discrimination. Extensive experiments on FashionAI, DARN, and DeepFashion demonstrate relative overall MAP improvements of 1.84%, 9.27%, and 9.35% over prior SOTA. SuperFashion offers a new solution for web-based image retrieval.

• The paper demonstrates that integrating superpixel tokenization with Transformer architecture and attribute-guided attention significantly improves fine-grained fashion retrieval performance.
• It employs a composite contrastive loss combining attribute-guided triplet ranking and cross-InfoNCE to align and distinguish attribute-specific image features.
• Experimental results reveal up to a 13.29% MAP improvement over patch-based methods, underscoring enhanced attribute localization and semantic tokenization.

Superpixel Token-Based Transformers for Attribute-Specific Fashion Retrieval

Motivation and Problem Formulation

Attribute-Specific Fashion Retrieval (ASFR) targets fine-grained image retrieval by focusing on localized, pixel-level fashion attributes. Conventional patch-based Transformer models exhibit critical limitations for ASFR: patch tokenization is grid-based and fails to align with irregular, attribute-specific regions, while background noise dilutes feature distinctiveness. The central challenge is to precisely localize and distinguish attribute-aware features in images, especially given the heterogeneous and fragmented nature of fashion attributes such as neckline design, fabric, or sleeve length.

SuperFashion Framework Overview

The SuperFashion framework addresses these limitations by integrating superpixel tokenization within a Transformer architecture. It operates in three stages. First, an attribute-guided attention mechanism extracts attribute-related features, influencing region cropping for subsequent superpixel segmentation. Second, the segmentation yields semantically coherent superpixel tokens, augmented with positional encodings. Both attribute and superpixel tokens are projected into modality-specific embeddings and fused via a Transformer, leveraging adaptive multi-head attention to discriminate and localize attributes effectively.

Figure 1: An overview of the SuperFashion framework; the representations $\mathbf{f}_{\text{A}}$ and $\mathbf{f}_{\text{T}}$ are jointly utilized for inference.

Superpixel Tokenization and Transformer Fusion

Superpixel segmentation aggregates visually homogeneous pixels, ensuring semantic alignment with attribute boundaries. Attribute-guided attention, based on cross-modal interaction between image and attribute embeddings, enforces attribute locality before region cropping. Positional encoding via learnable Fourier features ensures spatial information preservation in token embeddings. Dual pooling (average and max) summarizes superpixel tokens, capturing both global context and salient local details.

The superpixel token-based Transformer incorporates modality-specific embeddings, distinguishing attribute tokens (query) from superpixel tokens (keys/values). Multi-head self-attention is explicitly guided by attribute tokens, enabling adaptive feature aggregation and enhanced discrimination for ASFR.

Figure 2: Comparison of patch tokens and superpixel tokens for images with the same attribute: tokenization effects.

Training Objectives and Inference

The framework minimizes a composite contrastive loss, comprising attribute-guided triplet ranking and a novel cross-InfoNCE loss to align attribute-related and attribute-aware representations. The cross-InfoNCE loss facilitates joint learning of correlated, yet distinct, features across modalities. At inference, similarity computation between images for a given attribute integrates both attribute-related and attribute-aware features, weighted by a hyperparameter $\lambda$.

Experimental Validation

SuperFashion demonstrates robust performance across FashionAI, DARN, and DeepFashion, achieving relative overall MAP improvements of 1.84%, 9.27%, and 9.35% over previous SOTA models. On DeepFashion, improvement in texture and fabric attributes exceeds 10%. The framework shows effective generalization in cross-dataset evaluation, outperforming previous models by 7.11% and 12.96% MAP for transfer tasks.

Ablation studies reveal that removal of either the attribute-guided attention mechanism or the superpixel token-based Transformer degrades performance significantly. Substitution of superpixel tokens with conventional patch tokens yields MAP drops of up to 13.29%. Superpixel segmentation choice has a minor impact compared to tokenization strategy.

Figure 3: Overall MAP vs. superpixel token count and hyperparameters $\alpha$, $\beta$, $\lambda$ on DeepFashion dataset.

Analysis of Retrieval and Visualizations

SuperFashion excels in discriminating subtle attribute variations, as observed in retrieval cases where baseline models generate false matches. Visualization of superpixel segmentation confirms semantic alignment with attribute regions, validating the framework’s capacity to extract pure and discriminative tokens.

Figure 4: Retrieval case with incorrect retrievals highlighted.

Figure 5: Attribute-based superpixel segmentation map.

Time Efficiency and Practical Implications

While SuperFashion incurs a slight throughput reduction relative to patch-based RPF due to superpixel generation overhead, this is outweighed by its substantial retrieval performance gains. The computational trade-off is favorable, supporting practical deployment for web-based, real-time ASFR systems.

Theoretical and Practical Implications

SuperFashion demonstrates that semantic-aware tokenization is critical for fine-grained visual retrieval. By enabling token-level alignment with non-rectangular attribute regions, it advances the modeling of micro-structural vision features. The approach is extensible to various web-scale image retrieval scenarios and potentially to broader semantic segmentation tasks. The synthesis of modality-specific embeddings and cross-modal contrastive learning opens new directions for multi-modal retrieval in fashion and other domains.

Conclusion

SuperFashion establishes the efficacy of superpixel token-based Transformers for attribute-specific fashion retrieval, overcoming the limitations of patch-based tokenization and substantially improving retrieval precision. The paradigm shift toward semantically coherent tokenization informs future vision Transformer designs for fine-grained attribute modeling. Further advancements may integrate attribute-guided superpixel segmentation and broader application across domains requiring interpretability and discrimination at microstructural granularity.