Human-Preference FashionIQ (HP-FashionIQ)

• The paper introduces an evaluation benchmark that bridges the gap between standard CIR and nuanced human satisfaction in fashion retrieval.
• HP-FashionIQ employs paired result sets from different models, leveraging binary human preference ratings and Likert-scale scores for comprehensive analysis.
• The benchmark supports multi-task evaluation including preference prediction, attribute explanation, and textual rationale generation for transparent model assessment.

Human-Preference FashionIQ (HP-FashionIQ) defines a paradigm shift in the evaluation and interpretation of multimodal fashion retrieval systems. Its inception addresses limitations of standard Composed Image Retrieval (CIR) protocols—particularly, the disconnect between target-image retrieval and nuanced user satisfaction. HP-FashionIQ comprises annotated benchmarks and supporting methodologies for assessing how well retrieval systems align with—and can explain—true human preferences over sets of images in response to complex queries. The associated resources are structured to drive both fine-grained qualitative model evaluation and advance the field toward transparent, user-centric recommendation.

1. Motivation and Conceptual Foundation

Traditional CIR tasks evaluate performance by measuring Recall@k on a single iconic target image per ⟨reference image, modification text⟩ query, as seen in datasets such as FashionIQ and CIRR. However, in operational settings, user satisfaction is inherently multivariate: users judge result sets in aggregate and expect their subtle multimodal intent to be honored across all top-ranked outputs. HP-FashionIQ was developed to bridge this gap—providing both (i) an explicit, human-annotated preference benchmark over ranked sets, and (ii) an interpretable, cross-modal annotation protocol for analyzing why a user likes or dislikes specific fashion images, as facilitated by Fashionpedia-Taste (Kwak et al., 16 Jul 2025, Shi et al., 2023).

2. Data Collection, Annotation, and Schema

HP-FashionIQ as introduced in (Kwak et al., 16 Jul 2025) re-purposes the official FashionIQ validation split, covering 3,050 queries in "Shirts" and "Top tees." For each query, two independent CIR models (e.g., CLIP4CIR, Bi-BLIP4CIR, CoVR-BLIP, SPRC) each retrieve their own top-5 result sets. Human annotators (n=61; 50 queries per subject) compare these paired sets. Each iteration records: (a) Likert-scale relevance ratings of each result set, (b) an overall preference for set 1 or 2, (c) notes on whether either set ignored a modality, and (d) a sanity check to ensure the preference matches higher Likert-assigned relevance.

After filtering for modality redundancy (307 queries) and failed sanity checks (28 queries), the final dataset yields 2,715 preference-labeled queries. Each entry is characterized by its reference image $x_I$, modification text $x_T$, the two top-5 result sets from different models, annotator-preferred set ($y \in \{1,2\}$), and Likert relevance scores $(\ell_1, \ell_2 \in \{1,2,3,4,5\})$. HP-FashionIQ is strictly an evaluation benchmark—it does not provide train/test splits and is evaluated solely on held-out data.

The foundation for expanded human-interpretability analysis is provided by Fashionpedia-Taste (Shi et al., 2023), a large-scale, multi-modal resource comprising:

• 1,512 unique fashion images,
• 100 annotators (young, female, ethnically diverse) providing 10,000 user-image "expressions" (4,766 like, 5,234 dislike),
• Multimodal annotation: 52,962 localized attribute selections (≈5.3 per pair), 20,000 human-drawn attention maps (as polygons, rasterized into $H_{gt} \in \{0,1\}^{H \times W}$), and 20,000 free-form rationales (mean 11.3 words),
• User profiles: demographic information, Big-Five personality vectors $P \in \mathbb{R}^5$, favorite length, brands, and style categories.

3. Benchmark Structure and Evaluation Protocol

In HP-FashionIQ (Kwak et al., 16 Jul 2025), each query $Q_i = (x_{I_i}, x_{T_i})$ yields two result sets $S_{i,1}$ and $S_{i,2}$, each containing five images from distinct retrieval models. Human preference labels are binary: $y_i = 1$ if set 1 is preferred, $x_T$0 if set 2. The data supports set-level evaluation: for any scoring model $x_T$1,

$x_T$2

The core metric is Preference Rate, defined as the conditional probability that model-estimated preference between $x_T$3 and $x_T$4 matches the human-annotated choice:

$x_T$5

This protocol emphasizes holistic satisfaction across ranking sets, explicitly moving beyond single-target evaluation.

Among tasks outlined via Fashionpedia-Taste—preference prediction (binary), attribute explanation (multi-label), attention map prediction, rationale generation, retrieval with textual feedback—a multi-task, end-to-end model $x_T$6 with heads $x_T$7 may be trained with combined losses (cross-entropy, BCE, dice, and sequence likelihood) (Shi et al., 2023). Evaluation involves a suite of metrics: preference accuracy; attribute F1; attention alignment via IoU, AUC-Judd, and NSS; rationale quality via BLEU, METEOR, CIDEr, BERTScore; the composite Preference Consistency Score (PCS); and Recall@K for retrieval.

4. Task Definitions and Formalization

The explicit tasks defined by HP-FashionIQ and its underpinning Fashionpedia-Taste schema are:

Task	Input	Model Output
Preference Prediction	Image $x_T$8, user profile $x_T$9	$y \in \{1,2\}$0
Attribute Explanation	$y \in \{1,2\}$1, $y \in \{1,2\}$2	Subset $y \in \{1,2\}$3 of attribute–values
Attention Explanation	$y \in \{1,2\}$4, $y \in \{1,2\}$5	Attention heatmap $y \in \{1,2\}$6
Textual Rationale	$y \in \{1,2\}$7, $y \in \{1,2\}$8	Sentence $y \in \{1,2\}$9 justifying the prediction
Retrieval with Feedback	$(\ell_1, \ell_2 \in \{1,2,3,4,5\})$0, target feedback	Image $(\ell_1, \ell_2 \in \{1,2,3,4,5\})$1 matching the preferences

Preference Consistency Score (PCS) unifies attribute overlap, attention IoU, and caption similarity:

$(\ell_1, \ell_2 \in \{1,2,3,4,5\})$2

Aggregated PCS across test samples serves as a composite interpretability and fidelity measure (Shi et al., 2023).

5. Methodological Implications and Applications

HP-FashionIQ enables researchers to systematically compare CIR models’ alignment with real user preference at the set level, a property not directly assessed by classic Recall@k. For any two CIR models A and B, applying both to all 2,715 queries establishes paired result sets. Models are then judged according to how often their set-level relevance scores concord with empirical human choices—providing a ranking of models relative to actual user judgments. This protocol encourages development of retrieval architectures that are sensitive to nuances in user intent as expressed through multimodal queries.

Extending Fashionpedia-Taste, HP-FashionIQ supports nuanced analysis of preference formation by conditioning on psychographic profiles, attention foci, and rationales, affording interpretable modeling of individual and group-level taste. This supports preference-driven, explanation-oriented recommendation and retrieval.

6. Extensions, Limitations, and Open Challenges

Planned or suggested extensions include adapting the binary like/dislike judgment to a 5-point Likert scale for finer supervision, increasing annotator demographic diversity for fairness and cross-demographic generalization, collecting longitudinal ratings for temporal modeling of taste, and considering social-graph contextualization to model exposure effects (Shi et al., 2023).

Current limitations arise from attribute imbalance (instances with ~3× more “liked” than “disliked” attribute selections), sparse high-frequency brand labels (K ≈ 20 brands), limited linguistic diversity in rationales, and the non-trivial challenge of grounding free-form rationales in spatially arbitrary image regions. Suggested mitigations include focal-loss weighting, open-ended brand collection, paraphrase augmentation, and CLIP-style contrastive pre-training on region–caption pairs.

7. Significance and Outlook

HP-FashionIQ formalizes an evaluative and interpretive bridge between algorithmic retrieval quality and authentic user-centric satisfaction. Its combination of set-level human preference annotation, rich multi-modal contextualization, and comprehensive evaluation metrics enables rigorous benchmarking of CIR algorithms not only on their accuracy but on the alignment of their outputs with human taste and explanation. This supports the development of more transparent, user-aware, and trusted fashion retrieval and recommendation systems (Kwak et al., 16 Jul 2025, Shi et al., 2023).