Fashion-AlterEval: Alternative-Aware CRS Evaluation

• The paper introduces a dataset that enriches conversational recommendation systems with human-judged alternative items to overcome single-target evaluation limitations.
• It incorporates meta-user simulators, MetaSimTol and MetaSimProb, to model realistic user behavior including dynamic switching and finite patience.
• Empirical results reveal substantial performance gains, with MRR improvements up to 94%, when using alternative-aware evaluation over traditional protocols.

Fashion-AlterEval is a dataset and evaluation methodology for conversational recommendation systems (CRS) in fashion that augments standard single-target simulation with human judgments for alternative relevant items. It was introduced to address a specific limitation of offline CRS evaluation: conventional simulators critique recommendations only with respect to one hidden target item and assume effectively unlimited patience, whereas real fashion shopping often permits several acceptable alternatives and dynamic shifts in intent. In Fashion-AlterEval, the simulator is given access not only to an original target but also to human-judged alternative items, allowing simulated users to express preferences about alternatives, change their mind, and vary their level of patience during the interaction (Vlachou, 24 Jul 2025).

1. Conceptual motivation and evaluation philosophy

Fashion-AlterEval is motivated by the claim that existing conversational image recommendation settings are too rigid, too patient, and too narrow. In the standard setup, the user simulator has one target item in mind and critiques the current retrieved item only relative to that target. The paper characterizes this as close to a known-item search task rather than a realistic shopping process, because actual users may be satisfied with several alternatives, may revise preferences after seeing recommendations, and may stop insisting on the original target after repeated unsatisfactory turns (Vlachou, 24 Jul 2025).

A second motivation is the incompleteness of relevance ground truth. The paper explicitly draws an analogy to information retrieval test collections: search evaluation relies on pooled judgments to approximate completeness, whereas CRS evaluation often relies on a single hidden target. Under that framing, a system can retrieve an item that would be acceptable to a human shopper and still be scored as wrong, simply because only one item is labeled relevant. Fashion-AlterEval addresses this by collecting binary relevance judgments for alternative acceptable items, thereby making offline CRS evaluation closer to test collection construction in information retrieval (Vlachou, 24 Jul 2025).

The resulting evaluation philosophy is not to replace conversational recommendation with a different task, but to refine the notion of success. A successful interaction is no longer restricted to recovering one exact item; it can also end at an alternative that a human assessor judged to be a sufficient substitute for the target. This changes the interpretation of effectiveness from exact target recovery to satisfactory conversational resolution.

2. Dataset construction and annotation protocol

Fashion-AlterEval extends two existing fashion CRS datasets, Shoes and FashionIQ Dresses. In both cases, the original data were designed for relative captioning or conversational image recommendation, where a target image is paired with a candidate image and a human-written relative caption describing visual differences. Fashion-AlterEval adds a new layer of supervision: binary relevance judgments over candidate items, conditioned on a target, indicating whether each candidate is a sufficient alternative (Vlachou, 24 Jul 2025).

To make the new judgments representative, the dataset uses a pooling strategy. The authors selected 200 target items from each category, sampled to cover varying difficulty levels using preliminary query performance prediction analysis. For each target, the candidate pool was formed using two CRS systems, GRU and EGE, by combining top nearest neighbors in embedding space and top retrieved items from the final turn of each model, with overlap removed and lower-ranked items substituted when needed. The final pool contains 14 candidate images per target (Vlachou, 24 Jul 2025).

Annotations were collected on Amazon Mechanical Turk. Workers were restricted to the US, were shown the target and candidate images, and were asked to select one or more candidates that would be acceptable alternatives and to provide a reason for their choice. Quality control included a requirement for a full-sentence justification and a second assessment round on 40 targets. The reported inter-annotator agreement is high, with Cohen’s $\kappa = 0.87$ (Vlachou, 24 Jul 2025).

Dataset	Coverage	Alternative-judgment statistics
Shoes	4658 total targets in original dataset; 200 assessed	190 targets with at least one relevant alternative; average of 3.5 relevant alternatives per target; 14 annotations per target
Dresses	2454 total targets; 200 assessed	181 targets with at least one relevant alternative; average of 3.5 relevant alternatives per target; 14 annotations per target

These statistics show that acceptable alternatives are common rather than exceptional. That empirical regularity is central to the dataset’s rationale: if most assessed targets admit several relevant substitutes, then a single-target protocol is structurally incomplete.

3. Meta-user simulators and formal evaluation protocol

Fashion-AlterEval introduces two meta-user simulators that wrap a standard single-target simulator rather than replacing it. The baseline simulator is defined as

$$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$

where the simulator generates a critique $\mathcal{T}$ from the current top-ranked item and the target item (Vlachou, 24 Jul 2025).

The first extension, MetaSimTol, models fixed tolerance. Before a tolerance threshold is reached, the user behaves like the base simulator. After that threshold, the simulator may switch from the original target to the alternative most similar to the current top-ranked recommendation:

$$Metausersim.critique(user, top1_k, all\_targets) \rightarrow \mathcal{T}$$

with the operational rule

$$\text{if } turn > tolerance,\quad target = alts[\argmax(sims(alts, top1_k))].$$

The critique is then produced with respect to the switched target. This implements a deterministic notion of patience: after enough turns, the user “gives up” on the original target and accepts the best-matching relevant alternative (Vlachou, 24 Jul 2025).

The second extension, MetaSimProb, adds a gain-loss condition. It compares the current recommendation to the previous one relative to the target:

$$\delta = sim(top1_k, target) - sim(top1_{k-1}, target).$$

If

$$turn > tolerance \text{ and } \delta < 0 \text{ and } random() < P_{switch},$$

then the simulator switches to

$target = alts[\argmax(sims(alts, top1_k))].$

This makes switching contingent not only on patience but also on perceived deterioration. The paper interprets this as a more behavioral model, since a worse current recommendation increases the chance that the user changes their mind (Vlachou, 24 Jul 2025).

The evaluation protocol uses three CRS models—GRU-SL, GRU-RL, and EGE—and compares SimBase, MetaSimTol, and MetaSimProb. Each system retrieves top 100 items per turn. Conversations terminate at turn 10. If the target is found earlier, metrics are set to 1 thereafter. The reported metrics are Success Rate @ 1, MRR@10, and nDCG@10. Under the alternative-aware protocol, a conversation is considered successful if the system retrieves either the original target or any alternative relevant item (Vlachou, 24 Jul 2025).

4. Empirical findings and reinterpretation of CRS effectiveness

The central empirical result is that single-target evaluation underestimates CRS effectiveness. When the simulator is allowed to consider alternative relevant items, measured performance often rises substantially, and in many cases the system appears to satisfy the user sooner. The effect is especially strong for MRR@10, though not uniformly positive for every metric-model combination (Vlachou, 24 Jul 2025).

Setting	SimBase $\rightarrow$ alternative-aware	Reported summary change
Shoes, GRU-SL	nDCG@10: 0.209 $\rightarrow$ 0.248; MRR@10: 0.196 $$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$0 0.536	+17.06% nDCG; +94.19% MRR
Shoes, GRU-RL	nDCG@10: 0.303 $$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$1 0.211–0.264; MRR@10: 0.291 $$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$2 0.573	+64.80% MRR; -13.75% nDCG
Shoes, EGE	nDCG@10: 0.277 $$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$3 0.332; MRR@10: 0.263 $$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$4 0.329	+18.06% nDCG; +85.90% MRR
Dresses, GRU-SL	nDCG@10: 0.072 $$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$5 0.139; MRR@10: 0.068 $$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$6 0.388	+63.50% nDCG; +140.35% MRR
Dresses, GRU-RL	nDCG@10: 0.099 $$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$7 0.121; MRR@10: 0.269 $$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$8 0.333	+20% nDCG; +21.26% MRR
Dresses, EGE	nDCG@10: 0.085 $$Usersim.critique(user, top1_k, target) \rightarrow \mathcal{T},$$9 0.258; MRR@10: 0.084 $\mathcal{T}$0 0.587	+100.87% nDCG; +149.92% MRR

Several secondary findings qualify this result. MetaSimProb is usually better than MetaSimTol, especially for MRR@10 and for the GRU variants. The paper attributes this to the fact that GRU models rely heavily on previous-turn information, which aligns well with the pairwise gain-loss switching logic. By contrast, EGE already models the full dialogue history more strongly, so the additional benefit of MetaSimProb over MetaSimTol is smaller (Vlachou, 24 Jul 2025).

Tolerance also matters. Earlier switching to an alternative leads to a larger change in evaluation, whereas higher patience reduces the gap because the CRS has more turns to improve before switching occurs. The paper explicitly states that when users are allowed to consider alternatives, the system can rapidly respond to more quickly satisfy the user (Vlachou, 24 Jul 2025).

A further methodological claim is that more targets are not necessarily better than deeper judgments. In the paper’s RQ3 analysis, evaluation with 200 targets and deeper alternative judgments can yield higher MRR@10 than using all target items in the traditional single-target setup. This is presented as evidence that judgment completeness matters more than sheer target count.

5. Position within fashion recommendation and alteration research

Fashion-AlterEval occupies a specific niche within a broader body of research on fashion retrieval, alteration, and evaluation. Earlier work such as Fashionista focuses on fashion-aware visual exploration: given an item image or item ID, the system retrieves items that are visually similar and fashionable according to learned community preferences and trends. Its retrieval pipeline ranks top-$\mathcal{T}$1 nearest neighbors in a learned visual style space after category filtering and a fashionability threshold, and its interface supports substitutes, complements, and trend inspection. However, it is a demo or system paper and does not report a formal quantitative retrieval benchmark or controlled user study for substitute or complement quality (He et al., 2016).

Fashion++ moves closer to explicit alteration evaluation. It asks, given an outfit image $\mathcal{T}$2, for an edited version $\mathcal{T}$3 that is more fashionable while remaining similar to the original. The method factorizes garment latents into texture and shape, performs activation-maximization-style editing, and evaluates outcomes in terms of fashion improvement, amount of change, and human actionability. Reported human-study results include 92% of changed outfits judged more fashionable at $\mathcal{T}$4 and 72% judged actionable, with a mean clarity score of $\mathcal{T}$5 (Hsiao et al., 2019).

FashionStylist extends evaluation from item alternatives or local edits to holistic outfit judgment. It is an expert-annotated benchmark supporting outfit-to-item grounding, outfit completion, and outfit evaluation. The outfit evaluation task requires prediction of style, season, and occasion and identification of a style-incompatible item. The dataset contains 1,000 outfit-level samples, averages 4.6 items per outfit, and constructs a 2,000-sample evaluation set from original and corrupted outfits. The reported results show that mismatch detection remains the hardest subtask, with Gemini 3.1-Pro achieving the best mismatch detection accuracy of 0.2600 (Feng et al., 10 Apr 2026).

Taken together, these works suggest a continuum. Fashionista addresses retrieval of visually coherent substitutes and complements; Fashion++ evaluates minimal edits to a current outfit; FashionStylist evaluates expert-level outfit coherence; Fashion-AlterEval addresses conversational recommendation under alternative relevance. This suggests that Fashion-AlterEval is best understood not as a generic “fashion evaluation” benchmark, but as a specialized test collection for interactive recommendation where acceptable alternatives are central.

6. Scope, limitations, and terminological ambiguity

Fashion-AlterEval is deliberately narrow in scope. It is designed for offline evaluation of conversational recommendation with simulated users, not for garment generation, explicit outfit editing, or expert-authored semantic rationales. Its added supervision consists of human relevance judgments over pooled candidate items for selected targets in Shoes and FashionIQ Dresses, and its principal innovation lies in alternative-aware simulation rather than in a new retrieval backbone (Vlachou, 24 Jul 2025).

That scope yields clear strengths and clear limits. The strength is methodological: the dataset directly addresses incompleteness of relevance judgments and unrealistic single-target persistence. The limit is that it does not, by itself, explain why an alternative is good in terms of style, season, occasion, silhouette, or material; nor does it provide a formal model of edited-item quality comparable to minimal-edit frameworks. A plausible implication is that future evaluation designs could combine Fashion-AlterEval-style alternative judgments with expert-level outfit semantics or edit-level constraints.

There is also a terminological ambiguity in adjacent literature. One 2024 study on fashion product popularity describes an end-to-end framework, also referred to in its details as “Fashion-AlterEval,” that identifies influential image features, ranks them with an influence score, edits product images with diffusion models, and validates predicted popularity changes using the Fashion Demand Predictor and human surveys. In that setting, the term denotes evaluation of altered fashion products rather than conversational recommendation with alternative relevant items (Li et al., 2024). This suggests that the 2025 dataset gives the term its most specific and stable meaning, while nearby work uses similar language for a broader family of fashion alteration and evaluation problems.

In its most precise sense, however, Fashion-AlterEval denotes the 2025 fashion CRS test collection and its accompanying meta-user simulators. Its core contribution is to replace the exact-target assumption with a richer relevance model in which satisfactory alternatives, shifting intent, and finite patience become first-class objects of evaluation.