FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding

Abstract: Fashion understanding requires both visual perception and expert-level reasoning about style, occasion, compatibility, and outfit rationale. However, existing fashion datasets remain fragmented and task-specific, often focusing on item attributes, outfit co-occurrence, or weak textual supervision, and thus provide limited support for holistic outfit understanding. In this paper, we introduce FashionStylist, an expert-annotated benchmark for holistic and expert-level fashion understanding. Constructed through a dedicated fashion-expert annotation pipeline, FashionStylist provides professionally grounded annotations at both the item and outfit levels. It supports three representative tasks: outfit-to-item grounding, outfit completion, and outfit evaluation. These tasks cover realistic item recovery from complex outfits with layering and accessories, compatibility-aware composition beyond co-occurrence matching, and expert-level assessment of style, season, occasion, and overall coherence. Experimental results show that FashionStylist serves not only as a unified benchmark for multiple fashion tasks, but also as an effective training resource for improving grounding, completion, and outfit-level semantic evaluation in MLLM-based fashion systems.

• The paper introduces FashionStylist, a novel multimodal fashion dataset with dual-level expert annotations bridging item-level perception and outfit-level semantic evaluation.
• It demonstrates significant performance gains in outfit-to-item grounding, outfit completion, and holistic outfit evaluation using advanced benchmarks and metrics.
• The study underscores the impact of structured expert knowledge and multimodal active learning in enhancing both generative and retrieval-based AI systems for fashion.

FashionStylist: An Expert Knowledge-Enhanced Multimodal Dataset for Fashion Understanding

Motivation and Context

Comprehensive fashion understanding remains a fundamentally multimodal and hierarchical task, demanding not just visual perception but also expert-level semantic reasoning about style, layering, occasion, and coordination logic. Prevailing datasets in this domain typically partition supervision into either fine-grained item-level annotation (e.g., DeepFashion [liu2016DeepFashion], DeepFashion2 [ge2019deepfashion2], Fashionpedia [jia2020fashionpedia]) or outfit-level compatibility (e.g., Polyvore-U [lu2019learning], iFashion [chen2019pog], IQON3000 [song2019gp]), but seldom bridge these levels or supply systematically structured, expert-informed outfit-level semantics. This limitation restricts progress on holistic, context-sensitive tasks central to both real-world user and designer workflows.

FashionStylist (2604.09249) addresses these deficits by introducing a dataset annotated under an expert-driven, dual-level schema. The result is a resource that supports not only item perception, but also knowledge-rich outfit evaluation and completion—enabling systematic benchmarking and training for multimodal LLMs (MLLMs), vision-LLMs, and hybrid recommendation engines in the fashion domain.

Figure 1: FashionStylist dataset overview—illustrating the expert annotation pipeline (green), representative data samples (purple), and three core task formalizations (blue).

Dataset Construction and Semantic Coverage

FashionStylist is derived from extensive data collection on Taobao and Dewu, with rigorous sampling spanning garment types, genders, age brackets, seasonality, occasions, and stylistic families. Stringent linkage processes ensure each outfit is precisely mapped to its constituent item records. Annotation was conducted by a nine-member expert team using an iterative, cross-reviewed protocol, achieving a Cohen’s $\kappa$ of 0.64, indicating substantial inter-annotator agreement.

The dataset comprises 1,000 outfits (500 womenswear, 300 menswear, 200 childrenswear) and 4,637 items, averaging 4.6 items per outfit and encompassing both main garments and contextual accessories (shoes, hats, bags, scarves, ties). Item-level fields include category, outline, material, pattern, detail, and layering role. Outfit-level fields span free-text style rationales and categorical labels for season and occasion. A quality verification study, involving independent expert review of 10% of the data, confirmed a 91.3% attribute accuracy consensus.

Figure 2: (Left) Count of unique attribute values at item and outfit levels, evidencing annotation diversity. (Right) Normalized style-color co-occurrence frequencies, capturing nontrivial visual-semantic regularities in expert fashion knowledge.

The annotation richness is evident in the breadth of discrete attributes and the statistically significant coupling of style and color at the outfit level, substantiating the dataset’s capacity for modeling non-obvious, expert-driven fashion priors.

Benchmark Task Suite

FashionStylist supports three benchmark tasks calibrated for increasing cognitive complexity:

1. Outfit-to-Item Grounding: Given a full-look image and item category, models must extract or generate the corresponding standalone item image, thus requiring rigorous spatial grounding and disentanglement of overlapping and occluded garments.
2. Outfit Completion: For a partial outfit (with missing item(s)), the system is to retrieve or generate compatible items from the item pool, demanding context-sensitive composition and expert-level compatibility reasoning.
3. Outfit Evaluation: Models must infer an outfit’s style, season, and occasion from constituent item images, and flag incompatible items, necessitating both granular perception and holistic evaluation.

This hierarchical suite jointly targets perception, composition, and semantic understanding, aligning with core tasks in both consumer-facing recommendation and professional design curation.

Experimental Results

Empirical benchmarking is conducted across leading open-source MLLMs and proprietary models, with experiments ablated by fine-tuning (via LoRA) and expert knowledge inclusion.

Task 1: Outfit-to-Item Grounding

Open-source models (FLUX.1 Kontext, Qwen-ImageEdit, LongCat-Turbo) and the proprietary NanoBanana2 are evaluated using Recall@10, NDCG@10, PSNR, SSIM, FID, and KID metrics. Fine-tuning on FashionStylist yields consistent improvements for all open models, with relative gains of up to +26.1% in NDCG@10, +10.7% in PSNR, and a substantial FID reduction (28.4%) for Qwen-ImageEdit. The gap to NanoBanana2 is appreciably narrowed post-supervised fine-tuning.

Performance degrades on inner tops relative to outerwear, attributed to FashionStylist’s emphasis on layering and occlusion, evidencing the benchmark’s real-world challenge.

Figure 3: Performance across representative item categories for Task 1 (100/FID shown; higher is better), highlighting increased complexity in layered item extraction.

Task 2: Outfit Completion

Both retrieval-based (POG, CIRP, CLHE) and generative (DiFashion) paradigms are probed. Inclusion of expert-level semantic annotations (“Eke.” mode) consistently and substantially increases both Recall@10 and NDCG@10. DiFashion, with rich expert cues, achieves a 200.9% improvement in Recall@10. These results underscore that expert semantic guidance significantly enhances compatible item generation and retrieval—particularly vital when surface-level tags or titles are insufficient for nuanced compatibility.

Task 3: Outfit Evaluation

Evaluation accuracy is reported for prediction of style, season, occasion, and mismatch detection. LoRA-based supervised fine-tuning on FashionStylist yields clear improvements, e.g., Qwen2.5VL-7b sees +111.5% accuracy gain on season prediction. Notably, attribution to season benefits most from generic model reasoning (“Think” mode) while style/occasion require explicit expert supervision for strong performance. Even state-of-the-art proprietary models (e.g., Gemini 3.1-Pro) are outperformed in fine-grained attribute prediction by open models fine-tuned with FashionStylist.

Implications and Theoretical Significance

FashionStylist demonstrates that knowledge-aware, expert-annotated multimodal datasets can close key gaps in both generative and retrieval-based recommendation, and significantly enhance the transfer and alignment of LLM/MLLMs to fashion-specific semantic reasoning. The strong numerical improvements reported—especially for retrieval/generation with expert cues and for semantic prediction via LoRA-finetuning—highlight the limitation of existing models trained solely on weak or surface-level signals. These results provide evidence that progress toward robust, practically deployable AI in fashion is contingent not merely on model scale, but on access to high-fidelity, semantically structured supervision.

Theoretically, FashionStylist serves as an empirical probe of cross-modal reasoning in complex, compositional domains; its dual-level annotation structure is a practical blueprint for bridging object-level and collection-level understanding, directly translatable to other structured, compositional AI tasks. The rich co-occurrence structure and expert-curated correlations between visual and semantic attributes further enable research into interpretable, causal, and attribute-driven learning paradigms.

Future Directions

A noted limitation of FashionStylist is dataset scale, dictated by the high annotation throughput required for dual-level, expert-verified labeling. The authors indicate plans to expand the data toward higher-complexity outfits, richer layering, and broader real-world diversity, including leveraging MLLM-assisted iterative annotation pipelines.

Methodologically, FashionStylist invites exploration of multimodal active learning, hybrid symbolic-neural relational reasoning, and joint training protocols that can better fuse item-level perception with collection-level semantics, not only in fashion but in any context where compositional structure and expertise-aligned decisions are critical.

Conclusion

FashionStylist offers a benchmark that unifies, for the first time, expert-grounded multimodal item and outfit annotation at scale, supports rigorous evaluation across grounding, composition, and holistic assessment tasks, and empirically demonstrates the necessity of structured expert knowledge for closing critical gaps in current MLLM and generative retrieval systems in fashion. Its construction and findings have broad relevance for both the advancement of domain-specific AI and the design of datasets that capture the hierarchical, relational, and expert knowledge structures demanded by real-world tasks.