GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models

Abstract: We present a robust benchmark for measuring cultural value orientation in LLMs on the two Inglehart-Welzel axes over several domains and roles (hence GYROval - Gridded Yielding of Robust value Orientation), together with the results of administering it to twenty models. Items are binary contrastive scenarios in the sense introduced by CDEval: both options are legitimate courses of action, neither is correct, there is no answer key, and a model's score on an axis is the proportion of its responses falling on the counted pole. Eleven of the twenty models were additionally administered a paired Russian translation of the identical items and a second sampling temperature. The instrument is publicly released in both languages. Stability was assessed by treating the vignette as the unit of analysis, ranking the models within the levels of each perturbation factor, and summarising the agreement between levels by tie-corrected Kendall's \emph{W} against an empirical permutation null.

• This paper presents GYROval, a benchmark to measure cultural value orientation in large language models (LLMs) using binary contrastive vignettes, focusing on the stability of model rankings across different contexts rather than simple accuracy.
• GYROval assesses 20 models and finds that relative rankings show strong concordance under various perturbations, but notable changes in absolute scores occur across themes and roles, reinforcing the need for nuanced evaluation.
• Mechanistic analysis reveals that cultural value orientations in LLMs are decodable from model activations and can be causally manipulated, though the generalizeability remains limited.

GYROval addresses a specific measurement problem in LLM evaluation: whether cultural value orientations can be measured as behavioral properties that remain interpretable when the evaluation context changes. Rather than treating an item as correct or incorrect, the benchmark presents binary contrastive vignettes in which both options are legitimate and correspond to opposing poles of an Inglehart–Welzel value axis. A model’s score is therefore the proportion of responses selecting the designated pole, not an accuracy measure. The paper’s central methodological claim is deliberately narrower than measurement invariance: relative model rankings can remain stable even when absolute value scores shift substantially across contexts.

The benchmark is situated within a literature that has repeatedly documented sensitivity of LLM responses to prompt wording, option order, deployment context, language, and elicitation format. Existing value instruments differ in theoretical framework and often provide limited evidence that their rankings or scores replicate under altered administration conditions. GYROval adopts the semantic-vignette format associated with CDEval (Wang et al., 2023), but replaces its dimensional framework with the two principal Inglehart–Welzel axes and introduces a fully crossed design, paired bilingual administration, explicit rank-concordance analysis, and a mechanistic sub-study.

Benchmark construction and measurement target

The instrument contains 224 unique design cells represented by 225 released rows, formed by crossing seven themes, four decision modalities, and eight value facets. The themes are Arts, Education, Family, Lifestyle, Scientific, Wellness, and Work. The modalities are Advisor, Agent, Choice on behalf of the person, and Judgment of another’s behaviour. Four facets operationalize each axis: Agnosticism, Relativism, Scepticism, and Defiance for the sacred–secular axis; and Autonomy, Choice, Equality, and Voice for the obedient–emancipative axis.

The resulting factorial structure is:

$7 \times 4 \times 8 = 224$

The public files contain one duplicate row, giving 225 rows in total. The authors conduct analyses both with and without the duplicate and include the duplicate in a broader specification-sensitivity analysis. The items were generated with Gemini 2.5 Flash at temperature 0.4. Three domain experts accepted 91 seed items, which were used to generate the remaining scenarios. The paper reports the reviewer count and accepted seed count but does not report item-level inclusion criteria or inter-rater agreement.

The response format is intentionally forced-choice: a model must select Option 1 or Option 2. This design avoids answer keys and reduces direct contamination from standardized human questionnaires, but it creates a serious presentation-order issue. Among polarity-identifiable items, Option 2 carries the counted pole in 129 of 141 cases, or 91.5%. Option 2 is also longer on average than Option 1, with mean lengths of 73.37 and 67.59 characters, respectively, and contains more hedging markers, at means of 0.40 and 0.22 per option. The authors therefore identify a structural risk that lexical or slot preferences could be misinterpreted as value orientation.

Figure 1: Option-2 selection rates by model and decision modality, illustrating the interaction between model behavior and the structurally dominant counted-pole slot.

The primary evaluation nevertheless uses fixed option presentation order after a pilot that reportedly found no substantial order effect. Because the pilot’s sample size, model subset, and effect sizes are not documented, the study cannot establish that it had adequate power to detect smaller model-specific position biases. This is an important qualification: the benchmark’s principal robustness claim concerns rank preservation under several perturbations, but option-order robustness is not evaluated with the same level of transparency.

Experimental design and statistical framework

Twenty models are evaluated in English, including proprietary and open-weight systems from several model families. Eleven models are additionally evaluated on the paired Russian translation and at a second vendor-recommended sampling temperature. Seventeen models receive 30 samples per vignette, while three receive 50. The primary condition uses temperature zero.

The authors correctly distinguish response repetition from independent observational units. Repeated responses are first aggregated into a per-model, per-vignette pole-share proportion. Concordance is then calculated across vignettes rather than treating the 6,750 or more raw responses per model as independent observations. At temperature zero, self-consistency ranges from 91.7% to 100%, with a median near 99.5%. The vignette-level intraclass correlation ranges from 0.748 to 1.000, and the effective sample size is approximately 214–296 per model despite thousands of raw responses. Treating runs as independent would therefore produce severe pseudoreplication and substantially underestimated uncertainty.

The main stability statistic is tie-corrected Kendall’s coefficient of concordance, $W$. Models are ranked separately within each level of a perturbation factor, with the factor levels functioning as raters. The four perturbations are decision modality, theme, language, and temperature. Because $W$ has different null expectations depending on the number of raters, the authors generate a 20,000-replicate empirical permutation null for each cell. Under independent random rankings, the expected value is $1/m$, where $m$ is the number of factor levels. The 95th percentile of the relevant null, rather than a universal threshold for $W$, is used for inference.

Uncertainty intervals are obtained by resampling vignettes 2,000 times, stratified within design cells. These intervals quantify item-sampling variance, not prompt variance or run-to-run stochasticity. The authors explicitly exclude prompt rephrasing, template variation, and alternative scoring rules from the study. This distinction is consequential because prior work has shown that prompt templates can materially alter LLM rankings (Biderman et al., 2024, Moore et al., 2024).

Rank-order stability under perturbations

The benchmark produces above-null rank concordance in all twelve factor-by-axis cells. The strongest evidence concerns the obedient–emancipative axis, while the sacred–secular axis is consistently less stable.

Perturbation	Axis	Kendall’s $W$	Mean pairwise Spearman correlation
Modality	Obedient–emancipative	0.740	0.653
Modality	Sacred–secular	0.510	0.347
Theme	Obedient–emancipative	0.651	0.593
Theme	Sacred–secular	0.497	0.413
Language	Obedient–emancipative	0.951	0.901
Language	Sacred–secular	0.895	0.791
Temperature	Obedient–emancipative	0.983	0.966
Temperature	Sacred–secular	0.994	0.989

The two-level language and temperature comparisons must be interpreted against null expectations near 0.50, not against the multi-level modality and theme coefficients. All observed coefficients exceed their corresponding empirical null 95th percentiles. For example, modality has null 95th-percentile thresholds of approximately 0.373, while theme has thresholds near 0.219. Thus, the relatively modest sacred–secular modality concordance of 0.510 still substantially exceeds its calibrated chance baseline.

Figure 2: Tie-corrected Kendall concordance for each perturbation-by-axis cell, with bootstrap intervals and factor-specific permutation-null thresholds.

The rank structure is most stable at the extremes. Claude Sonnet 4.5 remains near the top across decision modalities, while Sonar Pro and GPT-5 remain in the lower part of the emancipative ranking. Middle-ranked systems are substantially more mobile. T-Lite changes from rank 19 to rank 4 across modalities on the emancipative axis, and gpt-oss-120b changes from rank 4 to rank 18 on the sacred–secular axis. These movements support the paper’s distinction between robust ordering of separated model profiles and unstable fine-grained comparisons among nearby models.

The empirical ceiling analysis strengthens the interpretation. Since theme and modality use different subsets of items for each level, perfect rank stability is not expected to yield an observed correlation of exactly one. Relative to simulated ceilings, temperature preserves 96.6% of achievable stability on the emancipative axis and 98.9% on the sacred–secular axis. Language preserves 90.1% and 79.1%, respectively. Theme preserves 86.4% and 94.7%, while modality preserves only 81.8% and 62.0%. Decision modality is therefore the most consequential perturbation for relative ordering, especially on the sacred–secular axis.

Absolute level shifts and role dependence

The principal empirical qualification is that rank stability does not imply score invariance. Pooled across models and axes, the mean counted-pole share is 0.669 for Advisor, 0.573 for Agent, 0.619 for Choice on behalf of the person, and 0.797 for Judgment of another’s behaviour. The modality range is therefore 0.224 on a scale from zero to one. Theme produces a comparable spread: Family reaches 0.756 and Wellness 0.753, whereas Scientific reaches 0.549 and Work 0.578, for a range of 0.207.

The direction of the role effect is substantively asymmetric. Models select more emancipative responses when judging another person’s behavior than when acting agentically on behalf of a user. The paper does not establish whether this reflects training-data regularities, safety policy, social-desirability behavior, or a more general distinction between evaluating and executing an action. It does establish that a single pooled value score conceals a role-dependent shift exceeding one-fifth of the scale.

Figure 3: Option-2 selection rates by model and theme, showing substantial theme-dependent shifts in absolute pole shares.

Role sensitivity is also model-specific. For each model, it is measured as the range between the highest and lowest mean pole share across the four modalities. On the obedient–emancipative axis, the range extends from 0.071 to 0.461, a 6.5-fold difference. On the sacred–secular axis, it extends from 0.094 to 0.500, a 5.3-fold difference.

Several models are comparatively role-invariant: Mistral Medium 3.1, Claude Sonnet 4.5, and Gemini 2.5 Pro have ranges no greater than 0.12 on both axes. Conversely, GigaChat 2 Max, A-vibe, and T-Lite have ranges of at least 0.35 on both axes. A-vibe’s sacred–secular range is 0.500, representing a shift across half the measurement scale. This result makes role sensitivity an evaluative property in its own right rather than merely a nuisance factor. A model may have a stable global ranking while still producing materially different value profiles depending on whether it advises, acts, decides for another person, or judges that person.

Language, temperature, and reasoning mode

The bilingual condition uses identical item identifiers and a paired Russian translation for eleven models. Russian administration shifts ten of eleven models toward the counted pole on the emancipative axis and seven of eleven on the sacred–secular axis. Mean absolute displacements are 0.029 and 0.036, respectively. The largest displacement is 0.071 for DeepSeek v3.2 on the sacred–secular axis. These shifts exceed split-run noise, whose median displacement is zero and maximum is 0.0013.

At the vignette level, only two of 22 model-by-axis contrasts are uncorrected-significant, and none survives Bonferroni correction. At the individual-response level, 11 contrasts survive correction, illustrating the dependence of inference on the observational unit. The paper appropriately treats the vignette as the generalizable unit for benchmark conclusions. Language concordance remains high, with $W=0.951$ for the emancipative axis and $W=0.895$ for the sacred–secular axis. Developer origin does not explain the displacement pattern: the largest shifts are distributed across Russian and non-Russian developers.

Sampling temperature is the least disruptive factor. No model-by-axis comparison between temperature zero and vendor-recommended temperatures is significant. The pooled concordance is $W=0.962$, with mean pairwise Spearman correlations of 0.966 on the emancipative axis and 0.989 on the sacred–secular axis. This finding supports temperature as a benign factor for relative ranking in this particular evaluation, but it does not eliminate concerns that temperature-zero scores may be biased estimators. The authors explicitly retain this limitation.

Reasoning-mode outputs agree with base-mode choices on 90%–97% of items, reaching 97% for Alice AI LLM. Reasoning-trace centroid similarities range from 0.576 to 0.733. A gradient-boosted classifier predicts final choices from trace embeddings with 0.854 accuracy, 0.812 balanced accuracy, and 0.885 ROC-AUC.

Figure 4: Transitions between base-mode and reasoning-mode choices, showing high persistence with limited answer switching.

These results show that explicit reasoning changes relatively few final choices and that traces contain predictive information about those choices. They do not establish that the traces causally determine the selections. In particular, the analysis does not distinguish reasoning that drives a decision from reasoning that rationalizes a decision already produced by other internal processes.

Mechanistic evidence for value-relevant representations

The mechanistic sub-study examines five open-weight models, with four treated as primary models and Qwen2.5-7B-Instruct as a control. It uses the same benchmark items but a distinct protocol: one deterministic forward pass and nine stochastic passes at temperature 0.7. The activation set is enriched for disagreement: 120 items are selected from the highest-variance items within the eight axis-by-modality cells, supplemented by 40 unanimous controls. However, activations are extracted only for the 120 signal items, and 25 of those 120 items have zero observed inter-model disagreement because the selection procedure imposed no minimum variance threshold.

Linear probes recover the axis label above the majority-class baseline in all eight model-by-axis cells. Accuracy ranges from 68.3% to 96.7%, with margins over the majority baseline ranging from 6.7 to 40.0 percentage points. The largest margin is achieved by T-Lite on the sacred–secular axis: 93.3% accuracy against a 53.3% baseline.

Figure 5: Probe accuracy at peak layers relative to majority-class baselines, demonstrating decodable axis information across all evaluated models.

Probe decodability alone would not establish behavioral relevance. The authors therefore construct difference-of-means directions and perform causal activation patching. Within models, injecting the direction into the residual stream flips choices at rates between 50% and 100%. Qwen3 reaches 76.0% on the emancipative axis and 100.0% on the sacred–secular axis; A-vibe reaches 88.0% and 96.0%; T-Lite reaches 90.9% and 76.0%. These results show that the probed directions can causally influence emitted choices under intervention.

Figure 6: Within-model activation-patching effects across depth, showing strong choice-flip rates and architecture-dependent causal profiles.

The cross-model evidence is more heterogeneous. Difference-of-means vectors are nearly orthogonal for most model pairs within the 4,096-dimensional architecture group. The sole strong within-group alignment is between A-vibe and T-Lite on the sacred–secular axis, with cosine similarity 0.296 and a corresponding isotropic-null z-score of 18.9. Four of six within-group comparisons are statistically indistinguishable from random orthogonal vectors. Cross-architecture similarities after PCA and Procrustes alignment are larger, but the authors correctly note that dimensionality reduction and alignment can induce similarity in sample-limited settings.

RSA and CKA reveal a separate representational partition. A-vibe has mean RSA values of 0.024–0.037 and mean CKA values of 0.090–0.107 with the other models, whereas non-A-vibe pairs have mean RSA values of 0.630–0.693 and mean CKA values of 0.701–0.730. The paper relates this isolation to A-vibe’s tokenizer and embedding configuration, but this remains an observational hypothesis rather than a demonstrated causal explanation.

Cross-model activation patching provides evidence of causal transfer but only on eight emancipative and nine sacred–secular disagreement items. On the sacred–secular axis, T-Lite-to-Qwen3 produces 9/9 choice flips, whereas Qwen3-to-T-Lite produces 1/9. Fisher’s exact test gives $W$0. The analogous emancipative comparison, 7/8 versus 5/8, is not significant, with $W$1.

Figure 7: Cross-model activation patching within the 4,096-dimensional model group, showing asymmetric transfer of value directions on disagreement items.

The result is technically notable but statistically fragile because each individual item changes the estimated flip rate by approximately 11–12 percentage points. The wide Wilson intervals and small item counts prevent broad claims about general cross-model representational alignment. The mechanistic evidence establishes that benchmark-relevant directions are decodable and manipulable in these models; it does not demonstrate that the directions naturally control ordinary generation, nor that they generalize to the broader twenty-model panel.

Limitations and open questions

The benchmark has no human baseline, human performance ceiling, or external criterion measure. Consequently, its scores cannot be interpreted as calibrated estimates of population values, and the model rankings cannot be validated against human responses. This limitation is particularly important because the Inglehart–Welzel axes themselves have contested measurement-invariance properties across populations, and the paper measures model responses to newly authored scenarios rather than reproducing a validated human survey instrument.

The item bank also contains structural asymmetries: the counted pole is concentrated in Option 2, Option 2 is longer and more hedged, and the primary evaluation fixes presentation order. The pilot evidence against order effects is insufficiently documented to rule out smaller systematic offsets. The analyses use one prompt template and one scoring rule, leaving prompt-level variance unmeasured. The bootstrap intervals quantify vignette-sampling variance but have known attenuation behavior; two intervals exclude their own point estimates, and the upper interval bound should not be read as a stability ceiling.

The baseline condition’s temperature-zero decoding creates another unresolved validity issue. Near-deterministic responses support reproducibility, but deterministic estimation does not guarantee construct validity. The study also does not provide a full Generalizability Theory decomposition of model, item, modality, theme, language, and interaction effects. Facet-level concordance and ipsative stability are not evaluated.

Finally, the mechanistic conclusions are limited by the small open-weight panel, 60 items per probe cell, omission of the control items from activation extraction, and cross-model patching on only 8–9 disagreement items. The most direct open empirical questions are whether the observed sacred–secular instability reflects item construction or model representations; whether the tokenizer difference explains A-vibe’s representational isolation; whether activation directions remain specific on unanimous control items; and whether the same causal structures appear in proprietary models.

Conclusion

GYROval provides a carefully specified benchmark for evaluating LLM cultural value orientation as a contrastive behavioral profile rather than as task accuracy. Across 20 models, relative rankings exceed calibrated permutation nulls under changes in decision modality, theme, language, and sampling temperature. The strongest stability appears under temperature variation, while decision modality produces the largest ranking and score perturbations.

The paper’s most important qualification is that rank-order stability and level invariance are different properties. Absolute pole shares shift by approximately 0.20–0.22 across themes and roles, and model-specific role sensitivity varies by factors of 6.5 and 5.3 across the two axes. The mechanistic sub-study further indicates that benchmark-relevant distinctions are internally decodable and causally manipulable, but its small and selective design does not establish general representational universality. GYROval therefore supports robust conclusions about broad relative positioning and role-dependent behavior, while requiring that scores be reported with their modality, theme, language, decoding condition, and methodological limitations.