Style Discrimination Benchmark Overview

• Style discrimination benchmarks are evaluation frameworks that assess a model’s ability to disentangle stylistic features from invariant factors like semantic content, speaker identity, or object category across modalities.
• They employ controlled data variations and rigorous methodologies—such as paired comparisons and specific metrics like WER, mAP@k, and cosine similarity—to accurately measure style separation.
• Empirical findings indicate that while models perform strongly on semantic tasks, they often struggle with subtle, fine-grained style discrimination and require improved disentanglement techniques.

Searching arXiv for papers on style discrimination benchmarks across modalities. Style discrimination benchmarks are evaluation frameworks that test whether a model can recognize, compare, preserve, or manipulate stylistic factors while separating them from variables that should remain invariant, such as semantic content, speaker identity, object category, or executable program behavior. Recent work treats style as a structured and interacting phenomenon rather than a single label: xSLUE argues that style is formed by a complex combination of factors such as formality markers, emotions, and metaphors (Kang et al., 2019), while ISSE defines style discrimination in speech as separating style attributes from linguistic content and speaker identity during instruction-guided editing (Chen et al., 29 Sep 2025). Across modalities, the same core question recurs: can a system determine what aspect of an artifact counts as style, and can it do so without collapsing style into topic, identity, or other shortcuts?

1. Conceptual foundations

In benchmark design, “style” is operationalized differently across domains, but the underlying concern is consistent. In text, xSLUE frames cross-style language understanding around 15 styles organized into four groupings—figurative, personal, affective, and interpersonal—and studies co-variation, inter-dependencies, and composition rather than isolated labels (Kang et al., 2019). In speech, ISSE defines style editing as modifying stylistic properties such as emotion, speaking rate, pitch, timbre-like expressive delivery, and prosodic manner while preserving transcript and speaker identity (Chen et al., 29 Sep 2025). VStyle extends the same principle to spoken LLMs by defining Voice Style Adaptation as determining whether an SLM can modify its speaking style in response to spoken instructions (Zhan et al., 9 Sep 2025).

Visual benchmarks make a parallel move from coarse labeling to fine-grained discrimination. MegaStyle argues that style is multi-dimensional and highly discriminative, with variation in color composition, light distribution, medium, texture, and brushwork even within a single named style (Gao et al., 9 Apr 2026). HCSU defines historical calligraphy style in terms of subtle brushwork evidence, including ink density, stroke termination, rhythm, structure, and spatial organization, and explicitly separates authentic ink manuscripts from stone rubbings to prevent style from being reduced to source-medium artifacts (Yao et al., 5 Jul 2026). Anime Style Recognition treats style as the abstract painting style of an anime work rather than the identity of a particular role (Li et al., 2022).

Some benchmarks generalize style beyond the arts. TacSIm defines football tactical style as the spatio-temporal organization of a team—formation shape, positional structure, player coordination, tempo, and movement flow—rather than reward-based success proxies such as goals or win rate (Wen et al., 26 Mar 2026). CSB treats code style as a property of program structure and idiom, but emphasizes that code style transfer is only meaningful if the system identifies exactly what to change, changes it correctly, and leaves the rest of the program untouched (Munson et al., 2024). This suggests that “style discrimination” is best understood as a disentanglement problem whose invariants depend on the domain.

2. Benchmark construction and control of invariants

A recurring design principle is to vary style while holding non-style factors fixed as tightly as possible. ISSE constructs source-target speech pairs from an anchor speech, its transcript, and its speaker identity, then uses style reference speeches to define the target style. EmoCapTTS preserves the anchor transcript while adopting the reference style, and Chatterbox voice conversion is then used to match timbre back to the anchor speaker. Retained samples must satisfy WER < 10, SIM_sty > 0.5, and SIM_spk > 0.5, explicitly operationalizing the requirement that content and speaker remain stable while style changes (Chen et al., 29 Sep 2025).

Other datasets impose the same separation through different controls. MegaStyle separates style prompts from content prompts using Qwen3-VL captioning, balances them through exact, fuzzy, and semantic deduplication plus hierarchical clustering, and then synthesizes 1.4M style images from 170K fine-grained styles via content-style prompt combinations (Gao et al., 9 Apr 2026). HCSU addresses “modal mixture” by separating Tie and Bei, canonicalizing foreground and background, normalizing geometry, and adding hierarchical expert annotations such as stroke style, ink style, character structure, overall charm, and layout (Yao et al., 5 Jul 2026). StylisticBias fixes identity as much as possible by generating 500 photorealistic base faces and then creating about 50 controlled variations per face, each changing exactly one visual attribute, yielding 15,726 evaluated images after filtering (Kolli et al., 18 Jun 2026).

Benchmark construction can also isolate style from semantic or contextual leakage. SB-Bench starts from ambiguous BBQ contexts, applies a Visual Information Remover to remove names, occupations, and explicit visual cues, and then retrieves real web images so that stereotype-driven answers arise from the image-conditioned judgment rather than textual leakage (Narnaware et al., 12 Feb 2025). In personalized text evaluation, the discrimination benchmark samples $T_{ref}$, $T_+$, and $T_-$ so that the positive and negative candidates differ systematically by domain, author, or personalization condition (Jangra et al., 8 Aug 2025). In code, CSB typically modifies original programs to create model inputs and uses the original code as the gold target, ensuring that the task is restoration of the desired style rather than unconstrained program synthesis (Munson et al., 2024).

3. Task protocols and operationalizations

Style discrimination is not tied to a single protocol. Existing benchmarks instantiate it as editing, retrieval, ranking, multi-way choice, imitation, or counterfactual comparison.

Benchmark	Modality	Operational test
ISSE	Speech	Instruction-guided source-to-target style editing
VStyle	Speech	Spoken-instruction voice style adaptation
MegaStyle / StyleRetrieval	Image	Same-style retrieval across different content
HCSU	Image-text	8-way style selection and aesthetic reasoning
STEB	Text embeddings	Pair classification, clustering, retrieval, order alignment, probing
TacSIm	Football simulation	Tactical style imitation from broadcast-derived states
CSB	Code	Exact style transfer with untouched non-target code
StylisticBias / SB-Bench	Multimodal bias	Counterfactual or MCQ judgment shifts under visual cues

ISSE formulates instruction-guided editing with triplets $(x_i, y_i, c_i)$, where $x_i$ is source speech, $y_i$ is target speech, and $c_i$ is the style instruction. The model is trained under teacher forcing to minimize the negative log-likelihood objective

$\min\; - \log p(y_i \mid x_i, c_i).$

The benchmark therefore tests whether the model can infer from the instruction which aspect of speech should change and which should remain invariant (Chen et al., 29 Sep 2025).

In text, the style-personalized generation benchmark defines a binary discrimination function

$f : (T_{ref}, T_+, T_-) \mapsto \{+, -\},$

where the correct choice is the candidate stylistically closer to the reference. It evaluates this in three settings: domain discrimination, authorship attribution, and LLM personalized vs non-personalized discrimination (Jangra et al., 8 Aug 2025). STEB generalizes textual style evaluation further by organizing 96 datasets across 7 languages into five task types—clustering, pair classification, order alignment, retrieval, and probing—so that style embeddings are compared under a standardized protocol (Soto et al., 30 Jun 2026).

Visual benchmarks adopt both retrieval and constrained selection. MegaStyle’s StyleRetrieval benchmark samples 2,400 fine-grained styles from the top 800 overall artistic styles, uses 4 images as queries and 28 as gallery per style, and asks whether the representation retrieves same-style images despite content changes (Gao et al., 9 Apr 2026). HCSU’s fine-grained style discrimination protocol presents one target image and 8 candidate styles—1 correct candidate and 7 distractors—and requires the model to select the correct candidate without explanation; the random baseline is 12.5% (Yao et al., 5 Jul 2026).

Counterfactual and ambiguity-based protocols test a different form of discrimination. StylisticBias compares a base face $x_b$ with a single-attribute variation $T_+$0 and measures the prediction shift caused by that one cue (Kolli et al., 18 Jun 2026). SB-Bench presents visually grounded MCQs in ambiguous social scenarios, where the correct answer is always “not known” and any other answer indicates stereotype-driven inference (Narnaware et al., 12 Feb 2025). TacSIm turns real broadcast footage into a real-to-virtual imitation task in which a model receives only initial player and ball positions and must infer the future tactical sequence (Wen et al., 26 Mar 2026).

4. Evaluation metrics and scoring regimes

The metrics used in style discrimination benchmarks are typically factorized to match the benchmark’s notion of invariance. In ISSE, performance is measured along four axes: WER for content preservation, style similarity $T_+$1 via emotion2vec, speaker similarity $T_+$2 via ECAPA-TDNN embeddings, and UTMOS for naturalness. Speaker preservation is computed by cosine similarity between generated and anchor embeddings,

$T_+$3

VStyle instead uses a hierarchical 5-point MOS scale: failure on textual adherence yields 1; style failure after content success yields 2 or 3; and only then is naturalness scored as 4 or 5. Final benchmark scores are obtained by averaging across categories with equal weights (Chen et al., 29 Sep 2025, Zhan et al., 9 Sep 2025).

Retrieval-style benchmarks rely on ranking metrics. MegaStyle measures style similarity in the MegaStyle-Encoder feature space with cosine similarity on $T_+$4-normalized embeddings and evaluates with mAP@k and Recall@k, $T_+$5 (Gao et al., 9 Apr 2026). STEB uses AUROC for pair classification, V-measure for clustering, MRR for retrieval, and task-specific accuracies for order alignment; for long documents it applies chunk-and-pool mean aggregation rather than naive truncation (Soto et al., 30 Jun 2026). HCSU reports Accuracy, Precision, Recall, and F1 for 8-way discrimination, then evaluates the reasoning protocol with BERTScore-F1 × 10 and a judge model scoring Terminology and Richness on a 0–10 scale (Yao et al., 5 Jul 2026).

Benchmarks centered on motion or structured behavior often require specialized similarity measures. TacSIm evaluates spatial occupancy similarity with a Jaccard index over occupancy tensors and movement vector similarity with cosine similarity over flattened motion vectors, then combines them as

$T_+$6

The benchmark is therefore explicitly balanced between where the team is and how the team moves (Wen et al., 26 Mar 2026).

Counterfactual bias benchmarks quantify shifts induced by stylistic edits. In StylisticBias, the preference score for image $T_+$7 and scenario $T_+$8 is

$T_+$9

with $T_-$0 prompt orderings and $T_-$1 random seeds. The prediction shift for a variation is $T_-$2, and the global effect is summarized by Signed Bias Shift,

$T_-$3

Identity-level disparity is separately measured by Variation Strength (VS) (Kolli et al., 18 Jun 2026). SB-Bench uses MCQ accuracy under the benchmark semantics that choosing “not known” is correct and any other choice is biased (Narnaware et al., 12 Feb 2025).

Code and speaker benchmarks demonstrate that style-sensitive evaluation often needs more than a single scalar score. CSB combines functional testing with DiffCorrect, which checks whether lines were added and removed exactly where they should be, using the submetrics Added Correctly, Removed Correctly, No Unexpected Removed, and No Unexpected Added (Munson et al., 2024). Speaker discrimination under style variability is evaluated with EER and $T_-$4, reflecting both hard decision performance and calibration quality under read–read, conversation–conversation, and read–conversation conditions (Zou et al., 2020).

5. Empirical findings across modalities

Benchmark results consistently show that style discrimination remains difficult even when semantic competence is strong. In speech editing, LlasaEdit trained on ISSE outperforms the same model trained on ESD on its own test set across all reported metrics: WER 8.06 vs. 10.07, style similarity 0.68 vs. 0.64, speaker similarity 0.58 vs. 0.49, and UTMOS 4.29 vs. 4.01. Cross-domain evaluation is harsher: the ESD-trained model collapses on ISSE with WER 68.17, whereas the ISSE-trained model generalizes well to ESD (Chen et al., 29 Sep 2025). VStyle similarly finds that style adaptation is significantly harder than content generation: GPT-4o = 4.05 is the best English overall, Doubao = 4.10 is the best Chinese overall, and open-source systems mostly remain in the 2–3 range, with variation and composite acoustic control identified as especially difficult (Zhan et al., 9 Sep 2025). The older speaker-discrimination study reaches a related conclusion: both humans and machines perform best under style-matched conditions and worst under read–conversation mismatch; native listeners achieve 6.96% EER on read–read versus 14.35% for the x-vector/PLDA system, but the gap disappears in style-mismatched trials (Zou et al., 2020).

Visual results point to the same gap between coarse recognition and fine-grained style perception. HCSU reports that all tested LVLMs remain only around 19%–38% accuracy on 8-way calligraphy style discrimination; Doubao-1.5-Vision-Pro-250328 reaches 37.29%, GPT-5.2 34.58%, and the best open-weight model, Qwen3-VL-235B-Instruct, 30.25%. The analysis concludes that current models are often sensitive to script-level, textual, and source-specific cues and are “knowledgeable but unperceptive” with respect to micro-level brushwork evidence (Yao et al., 5 Jul 2026). MegaStyle shows the opposite pattern: when intra-style consistency and inter-style diversity are explicitly engineered, MegaStyle-Encoder reaches 87.26 mAP@1, 85.98 mAP@10, 87.26 Recall@1, and 97.61 Recall@10 on StyleRetrieval, and remains best on StyleBench, FLUX-Retrieval, and OmniStyle-150K (Gao et al., 9 Apr 2026). Anime Style Recognition is harder again: LSASRD baselines remain near 42% mAP, with the abstract highlighting 42.24% mAP for TransReID and the table reporting 42.76% mAP for ViT-Small (Li et al., 2022). The infant-scale attribute benchmark reveals strong attribute asymmetry rather than uniformly weak performance: in prototype mode, CVCL reaches 62.3% on size against CLIP’s 50.5%, whereas CLIP reaches 69.2% on color against CVCL’s 19.6%; both are high on texture in image space but near chance for size and texture in text-vision grounding (Batsell et al., 22 Dec 2025).

Text, code, and multimodal social-judgment benchmarks show analogous patterns. In personalized text evaluation, the difficulty ordering is stable: Domain discrimination (DD) > Authorship attribution (AA) > LLM personalized vs non-personalized (LLM). The best single metric, gpt-4.1, scores 0.961 on DD, 0.807 on AA, 0.678 on LLM, and 0.815 mean, while the best ensemble, $T_-$5, reaches 0.821 mean (Jangra et al., 8 Aug 2025). STEB finds that semantic embeddings consistently fail on stylistic tasks and that there is no style embedding universally superior across all tasks (Soto et al., 30 Jun 2026). CSB reports that large prompted code models and a fine-tuned CodeT5-small generally fail to perform non-trivial code style transfer correctly; CodeBLEU and pass rate can be high even when DiffCorrect shows that the model left required edits undone or changed unrelated lines (Munson et al., 2024). In bias-sensitive multimodal settings, StylisticBias reports that about 15 attributes explain nearly 80% of total $T_-$6 and that age and body type dominate identity-level effects, while fashion style and related self-presentation cues drive the largest attribute-level shifts (Kolli et al., 18 Jun 2026). SB-Bench finds that vision-enabled models are on average 13% more biased than their underlying text-only LLMs, with Nationality showing a 26.2% increase; among evaluated systems, GPT-4o has the best average bias score at 10.79%, while the best open-source model, InternVL2-8B, is at 62.00% (Narnaware et al., 12 Feb 2025).

6. Limitations, controversies, and future directions

A central limitation is that style is rarely cleanly separable from semantics or source artifacts. STEB states this explicitly: content and style are hard to disentangle, so the benchmark includes both tasks where they are entangled and tasks designed to suppress semantic shortcuts (Soto et al., 30 Jun 2026). HCSU shows that even when annotations are expert-written and modality is carefully controlled, models may still rely on script-level, OCR-like, or source-specific cues rather than genuine brushwork evidence; its ablations further indicate that raw images can sometimes improve discrimination precisely because they contain contextual cues that may be spurious shortcuts (Yao et al., 5 Jul 2026).

Evaluation itself is another source of controversy. VStyle introduces LALM-as-a-Judge to reduce the cost and variability of human evaluation, but the paper also acknowledges that LALMs can hallucinate and that future improvements in auditory perception and reasoning are needed (Zhan et al., 9 Sep 2025). The personalized text benchmark reaches a similar conclusion from another angle: no single evaluator is sufficient, binary discrimination is only a proxy for richer stylistic judgments, and the benchmark does not capture first-person user preference (Jangra et al., 8 Aug 2025). CSB shows that functional correctness alone is insufficient because a model can pass tests by making no stylistic transformation at all, motivating exact-edit metrics such as DiffCorrect (Munson et al., 2024).

Several benchmarks also expose domain-specific interpretive caveats. StylisticBias notes that clothing-related edits are presented as full-body portraits, unlike most other head-and-shoulders variations, so fashion effects should be interpreted with that fuller visual context in mind (Kolli et al., 18 Jun 2026). SB-Bench uses paired-image stitching for categories such as Nationality, Race/Ethnicity, Religion, and Socio-Economic Status when single real images are hard to obtain, which improves realism relative to synthetic generation but also defines a particular benchmark geometry (Narnaware et al., 12 Feb 2025). xSLUE further warns that correlation analyses over stylistic factors are not causal and points toward causal analysis, handling of style drift, and ethical interpretation—especially for personal styles—as future work (Kang et al., 2019).

Taken together, these benchmarks show that style discrimination is not a narrow subproblem of classification. It is a family of evaluation problems centered on controlled variation, invariance preservation, and sensitivity to subtle expressive structure. The most mature benchmarks therefore do not ask only whether a model can recognize style labels, but whether it can distinguish style from content, identity, or reward proxies under conditions where shortcut solutions are deliberately constrained.