---
title: Human-Preference FashionIQ (HP-FashionIQ)
url: https://www.emergentmind.com/topics/human-preference-fashioniq-hp-fashioniq
type: topic
---

# Human-Preference FashionIQ (HP-FashionIQ)

Human-Preference FashionIQ (HP-FashionIQ) defines a paradigm shift in the evaluation and interpretation of multimodal fashion retrieval systems. Its inception addresses limitations of standard Composed Image Retrieval (CIR) protocols—particularly, the disconnect between target-image retrieval and nuanced user satisfaction. HP-FashionIQ comprises annotated benchmarks and supporting methodologies for assessing how well retrieval systems align with—and can explain—true human preferences over sets of images in response to complex queries. The associated resources are structured to drive both fine-grained qualitative model evaluation and advance the field toward transparent, user-centric recommendation.

## 1. Motivation and Conceptual Foundation

Traditional CIR tasks evaluate performance by measuring Recall@k on a single iconic target image per ⟨reference image, modification text⟩ query, as seen in datasets such as FashionIQ and CIRR. However, in operational settings, user satisfaction is inherently multivariate: users judge result sets in aggregate and expect their subtle multimodal intent to be honored across all top-ranked outputs. HP-FashionIQ was developed to bridge this gap—providing both (i) an explicit, human-annotated preference benchmark over ranked sets, and (ii) an interpretable, cross-modal annotation protocol for analyzing why a user likes or dislikes specific fashion images, as facilitated by Fashionpedia-Taste [2507.12416] [2305.02307].

## 2. Data Collection, Annotation, and Schema

HP-FashionIQ as introduced in [2507.12416] re-purposes the official FashionIQ validation split, covering 3,050 queries in "Shirts" and "Top tees." For each query, two independent CIR models (e.g., CLIP4CIR, Bi-BLIP4CIR, CoVR-BLIP, SPRC) each retrieve their own top-5 result sets. Human annotators (n=61; 50 queries per subject) compare these paired sets. Each iteration records: (a) Likert-scale relevance ratings of each result set, (b) an overall preference for set 1 or 2, (c) notes on whether either set ignored a modality, and (d) a sanity check to ensure the preference matches higher Likert-assigned relevance.

After filtering for modality redundancy (307 queries) and failed sanity checks (28 queries), the final dataset yields 2,715 preference-labeled queries. Each entry is characterized by its reference image $x_I$, modification text $x_T$, the two top-5 result sets from different models, annotator-preferred set ($y \in \{1,2\}$), and Likert relevance scores $(\ell_1, \ell_2 \in \{1,2,3,4,5\})$. HP-FashionIQ is strictly an evaluation benchmark—it does not provide train/test splits and is evaluated solely on held-out data.

The foundation for expanded human-interpretability analysis is provided by Fashionpedia-Taste [2305.02307], a large-scale, multi-modal resource comprising:
- 1,512 unique fashion images,
- 100 annotators (young, female, ethnically diverse) providing 10,000 user-image "expressions" (4,766 like, 5,234 dislike),
- Multimodal annotation: 52,962 localized attribute selections (≈5.3 per pair), 20,000 human-drawn attention maps (as polygons, rasterized into $H_{gt} \in \{0,1\}^{H \times W}$), and 20,000 free-form rationales (mean 11.3 words),
- User profiles: demographic information, Big-Five personality vectors $P \in \mathbb{R}^5$, favorite length, brands, and style categories.

## 3. Benchmark Structure and Evaluation Protocol

In HP-FashionIQ [2507.12416], each query $Q_i = (x_{I_i}, x_{T_i})$ yields two result sets $S_{i,1}$ and $S_{i,2}$, each containing five images from distinct retrieval models. Human preference labels are binary: $y_i = 1$ if set 1 is preferred, $y_i = 2$ if set 2. The data supports set-level evaluation: for any scoring model $m$,

$$
s_{rel}(S) = \frac{1}{|S|} \sum_{I \in S} s_m(x_{I_i}, x_{T_i}, I)
$$

The core metric is Preference Rate, defined as the conditional probability that model-estimated preference between $S_{i,1}$ and $S_{i,2}$ matches the human-annotated choice:

$$
\text{Preference Rate} = P( \text{model picks Set 1} \mid s_{rel}(S_{i,1}) > s_{rel}(S_{i,2}) )
$$

This protocol emphasizes holistic satisfaction across ranking sets, explicitly moving beyond single-target evaluation.

Among tasks outlined via Fashionpedia-Taste—preference prediction (binary), attribute explanation (multi-label), attention map prediction, rationale generation, retrieval with textual feedback—a multi-task, end-to-end model $f$ with heads ${f_\mathrm{pref}, f_\mathrm{attr}, f_\mathrm{att}, f_\mathrm{cap}}$ may be trained with combined losses (cross-entropy, BCE, dice, and sequence likelihood) [2305.02307]. Evaluation involves a suite of metrics: preference accuracy; attribute F1; attention alignment via IoU, AUC-Judd, and NSS; rationale quality via BLEU, METEOR, CIDEr, BERTScore; the composite Preference Consistency Score (PCS); and Recall@K for retrieval.

## 4. Task Definitions and Formalization

The explicit tasks defined by HP-FashionIQ and its underpinning Fashionpedia-Taste schema are:

| Task                     | Input                           | Model Output                                                  |
|--------------------------|---------------------------------|---------------------------------------------------------------|
| Preference Prediction    | Image $I$, user profile $U$     | $y \in \{\text{like},\text{dislike}\}$                        |
| Attribute Explanation    | $I$, $U$                        | Subset $\hat{A}$ of attribute–values                          |
| Attention Explanation    | $I$, $U$                        | Attention heatmap $\hat{H} \in [0,1]^{H\times W}$             |
| Textual Rationale        | $I$, $U$                        | Sentence $\hat{C}$ justifying the prediction                  |
| Retrieval with Feedback  | $U$, target feedback            | Image $I^+$ matching the preferences                          |

Preference Consistency Score (PCS) unifies attribute overlap, attention IoU, and caption similarity:

$$
PCS_i = \frac{
    |\hat{A}_i \cap A_i|/|A_i| + IoU(\hat{H}_i, H_i) + sim(\hat{C}_i, C_i)
}{3}
$$

Aggregated PCS across test samples serves as a composite interpretability and fidelity measure [2305.02307].

## 5. Methodological Implications and Applications

HP-FashionIQ enables researchers to systematically compare CIR models’ alignment with real user preference at the set level, a property not directly assessed by classic Recall@k. For any two CIR models A and B, applying both to all 2,715 queries establishes paired result sets. Models are then judged according to how often their set-level relevance scores concord with empirical human choices—providing a ranking of models relative to actual user judgments. This protocol encourages development of retrieval architectures that are sensitive to nuances in user intent as expressed through multimodal queries.

Extending Fashionpedia-Taste, HP-FashionIQ supports nuanced analysis of preference formation by conditioning on psychographic profiles, attention foci, and rationales, affording interpretable modeling of individual and group-level taste. This supports preference-driven, explanation-oriented recommendation and retrieval.

## 6. Extensions, Limitations, and Open Challenges

Planned or suggested extensions include adapting the binary like/dislike judgment to a 5-point Likert scale for finer supervision, increasing annotator demographic diversity for fairness and cross-demographic generalization, collecting longitudinal ratings for temporal modeling of taste, and considering social-graph contextualization to model exposure effects [2305.02307].

Current limitations arise from attribute imbalance (instances with ~3× more “liked” than “disliked” attribute selections), sparse high-frequency brand labels (K ≈ 20 brands), limited linguistic diversity in rationales, and the non-trivial challenge of grounding free-form rationales in spatially arbitrary image regions. Suggested mitigations include focal-loss weighting, open-ended brand collection, paraphrase augmentation, and CLIP-style contrastive pre-training on region–caption pairs.

## 7. Significance and Outlook

HP-FashionIQ formalizes an evaluative and interpretive bridge between algorithmic retrieval quality and authentic user-centric satisfaction. Its combination of set-level human preference annotation, rich multi-modal contextualization, and comprehensive evaluation metrics enables rigorous benchmarking of CIR algorithms not only on their accuracy but on the alignment of their outputs with human taste and explanation. This supports the development of more transparent, user-aware, and trusted fashion retrieval and recommendation systems [2507.12416] [2305.02307].

Source: https://www.emergentmind.com/topics/human-preference-fashioniq-hp-fashioniq