Value-oriented Evaluation (VQ)

• Value-oriented Evaluation (VQ) is a framework that redefines model performance by assessing real-world impacts such as economic viability, social impact, ethical alignment, and environmental sustainability.
• It shifts the focus from traditional benchmark metrics to context-sensitive evaluations that incorporate human judgment, operational costs, and applicable task benefits.
• VQ is applied across domains like LLMs, generative VQA, power system forecasting, and embodied AI, offering modular assessments tailored to specific deployment contexts.

Value-oriented Evaluation (VQ) denotes an evaluation perspective in which a model, forecast, representation, or agent is judged not only by benchmark score or statistical fit, but by the value it creates for a specified task, observer, user, or deployment context. In the LLM evaluation literature, VQ is explicitly defined as a deployment-facing layer that assesses economic viability, social impact, ethical alignment, and environmental sustainability (Wang et al., 26 Aug 2025). In adjacent technical literatures, the same evaluative logic appears in domain-specific form: semantic correctness rather than exact match in generative VQA, downstream operational cost rather than RMSE in unit commitment, observer-usable task information rather than ideal information in medical imaging, and latent task progress rather than single-frame reactivity in long-horizon manipulation (Ji et al., 2024, Ghazanfariharandi et al., 17 Mar 2025, Lu et al., 30 Sep 2025, Honghui et al., 10 Mar 2026). Taken together, these works suggest that VQ is less a single metric than a family of task- and context-sensitive evaluation doctrines.

1. Conceptual foundations

The central motivation for VQ is a recurrent disconnect between benchmark performance and real-world utility. The survey "Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap" presents this most explicitly: benchmark-centric evaluation is described as largely capability-centric and task-centric, whereas VQ is impact-centric and value-centric (Wang et al., 26 Aug 2025). In that framing, a model may score well on classical datasets and still be economically inefficient, socially unhelpful, ethically risky, or environmentally expensive.

A closely related critique appears in other domains. In generative VQA, Exact Match and standard VQA Score are characterized as too brittle for open-ended multimodal outputs because they reject semantically correct but morphologically different responses and discourage rich answers (Ji et al., 2024). In power systems, the same logic is stated operationally: a forecast can be statistically good yet operationally bad because asymmetric recourse costs mean that minimizing RMSE need not minimize system cost (Ghazanfariharandi et al., 17 Mar 2025). In medical imaging, task-specific information based on mutual information is said to quantify the total task-relevant information present in an image, but not the amount of that information a sub-ideal observer can actually exploit (Lu et al., 30 Sep 2025).

These formulations share a common structure. First, they reject the sufficiency of surface-level performance surrogates such as exact string match, pointwise prediction error, or ideal-observer information. Second, they re-anchor evaluation in an external notion of value: human semantic judgment, operational savings, usable task information, or deployment impact. A plausible implication is that VQ is best understood as an evaluative shift from intrinsic performance descriptors to extrinsic consequence-sensitive criteria.

2. Taxonomic position and evaluative architecture

In the anthropomorphic roadmap for LLM evaluation, VQ is the fourth layer after IQ, PQ, and EQ. The paper ties these layers to the development lifecycle as follows: IQ corresponds to pre-training, PQ to supervised fine-tuning, EQ to reinforcement learning or post-training alignment, and VQ to the final deployment-facing layer that asks whether the model produces real-world value across economic, social, ethical, and environmental dimensions (Wang et al., 26 Aug 2025). The authors summarize the sequence as “IQ (pre-training knowledge acquisition), PQ (supervised fine-tuning expertise), EQ (reinforcement alignment), and VQ (value-oriented impact).”

Within that framework, VQ is not presented as a replacement for capability evaluation. It is described as a complementary impact layer on top of existing capacity, expertise, and alignment assessments. The same paper also states that there is no single mathematical VQ score or optimization function; no finalized weighted aggregation is given. Instead, VQ is embedded in a modular evaluation architecture consisting of a benchmark or dataset hub, model hub, prompting module, metrics, monitoring and experiment management, and arena or leaderboard, and it is said to leverage three paradigms: metrics-centered assessment, human-centered assessment, and model-centered peer review (Wang et al., 26 Aug 2025).

The four named VQ dimensions and their indicators are as follows.

Dimension	Named indicators	Assessment approach
Economic viability	CBR, ROI, PI, MA	Cost accounting; downstream utility gains; business metrics
Social impact	US, KDE, PSI, EQI	Surveys, feedback, usage studies, case studies
Ethical alignment	F, T, PP, BD	Statistical tests, expert review, security audits
Environmental sustainability	EE, CF, S	Energy measurement, carbon accounting, lifecycle assessment

The same source also places VQ at the end of a six-tier roadmap: statistical rigor, composite evaluation/ranking, interpretability and explainability, user-centric experience, human-in-the-loop evaluation, and value-oriented dimensions (Wang et al., 26 Aug 2025). This makes VQ cumulative rather than isolated: it depends on technical validity, interpretability, and human judgment instead of bypassing them.

3. Semantic correctness, human values, and preference-sensitive evaluation

One major operationalization of VQ appears in generative visual question answering. "Towards Flexible Evaluation for Generative Visual Question Answering" reframes VQA evaluation as semantic correctness assessment for open-ended responses rather than exact string matching (Ji et al., 2024). The paper formalizes evaluator output through cosine similarity between contextualized text representations and evaluates the evaluator itself by Spearman correlation with human annotations. It further argues that VQA evaluators should be judged by three properties: Alignment, Consistency, and Generalization. Alignment is measured by Spearman rank correlation with human judgments; Consistency is defined through score variance across semantically equivalent responses with different morphology or length; Generalization is defined through variance of Alignment across datasets. The AVE dataset introduced for this purpose contains 3,592 samples, is split 3:7 into validation:test, and reports Krippendorff’s alpha of 0.713 (Ji et al., 2024). In this setting, VQ takes the form of human-aligned semantic evaluation.

A second operationalization appears in value-preference auditing for vision-LLMs. "Value-Spectrum: Quantifying Preferences of Vision-LLMs via Value Decomposition in Social Media Contexts" builds a benchmark around Schwartz’s ten value dimensions using 50,191 unique video records from TikTok, YouTube Shorts, and Instagram Reels (Li et al., 2024). The benchmark evaluates whether a model finds value-linked content “interesting,” converting yes/no responses into a value-specific preference score in $[0,1]$. It reports that videos related to Achievement, Hedonism, and Power appear most frequently in the collected corpus, while Tradition is relatively rare, and it finds distinct model-specific value profiles, with a broad tendency toward Hedonism and Self-direction across models (Li et al., 2024). Here VQ is explicitly about latent preference structure rather than conventional task accuracy.

A third operationalization appears in AI-generated video quality assessment. "VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learning" treats video quality understanding as a reasoning-style VLM problem and introduces a progressive training pipeline with image quality warm-up, temporal learning for video-specific quality understanding, and joint optimization with the generation model (Zhang et al., 23 Jun 2025). Its reward design is multi-dimensional: image scoring reward, temporal modeling reward, length control reward, multi-dimension scoring reward, and preference comparison reward. For AIGC videos, the three named dimensions are spatial quality, temporal quality, and text-video alignment. Reported results include tau 50.80 and diff 75.71 on GenAI, tau 61.20 and diff 74.51 on MonetBench, and multi-dimension scoring results such as SRCC 0.911, KRCC 0.744, and PLCC 0.927 for temporal quality (Zhang et al., 23 Jun 2025). In this formulation, value is expressed as fine-grained quality judgment aligned with human preferences and temporally grounded reasoning.

Across these works, VQ does not denote a single annotation format. It can be instantiated as semantic similarity scoring, value-profile elicitation, or reward-shaped quality reasoning. What remains constant is that the evaluated output is judged by contextual meaning or human significance, not by rigid lexical or surface-form agreement.

4. Temporal and embodied formulations

In embodied AI, VQ is extended to long-horizon settings in which instantaneous observation is insufficient. "Beyond Short-Horizon: VQ-Memory for Robust Long-Horizon Manipulation in Non-Markovian Simulation Benchmarks" introduces RuleSafe, a benchmark of safe-opening tasks with 10 safe instances and 20 rules/tasks, including key locks, password locks, and logic locks (Honghui et al., 10 Mar 2026). The benchmark is described as non-Markovian because the current visual observation does not fully reveal the latent state needed for action choice; the paper formalizes this hidden structure through part-phase and task-phase variables.

The accompanying VQ-Memory module compresses a window of past proprioceptive joint states into discrete latent memory tokens with a VQ-VAE. The training loss combines reconstruction and commitment terms with $\lambda = 4$, and a post-hoc K-means clustering step reduces the vocabulary from 256 codes to 4 tokens. With a window of 50 and stride 20, the paper reports about a 20× compression ratio (Honghui et al., 10 Mar 2026). Memory tokens are then integrated into VLA models such as $\pi_0$, RDT, and CogACT, and into the diffusion policy DP3.

The reported results are framed as evidence that short-horizon evaluation is insufficient for realistic articulated manipulation. On rule_001, $\pi_0$ alone achieves 30.0% SR / 56.7% PS, raw memory gives 55.0% / 70.1%, and VQ-Memory gives 80.0% / 89.3%. On the harder rule_020, $\pi_0$ gives 0.0% / 10.6%, raw memory 0.0% / 16.3%, and VQ-Memory 45.0% / 67.3% (Honghui et al., 10 Mar 2026). Multi-task training across all 20 tasks improves average performance from 25.0% SR / 48.8% PS to 56.3% SR / 76.5% PS.

This formulation broadens VQ from human-value assessment to temporal task-value assessment. The criterion is whether a policy can maintain and exploit latent task progress over long horizons. A plausible implication is that, in embodied systems, VQ evaluates the utility of state representations by their contribution to temporally extended success rather than by instantaneous perceptual sufficiency.

5. Decision-focused and observer-specific formulations

A strongly formalized version of VQ appears in power-system forecasting. "Value-Oriented Forecast Combinations for Unit Commitment" defines value-oriented forecasts as forecasts designed or combined to reduce expected downstream decision cost in a two-stage unit commitment plus real-time dispatch system (Ghazanfariharandi et al., 17 Mar 2025). Forecasts from multiple providers are combined by convex linear pooling, and the combination weights are selected not to minimize forecast error directly but to minimize expected total operational cost. The paper evaluates performance with the two-stage total system cost, reporting ${\rm TST}(\lambda_1,\lambda_2)$, ${\rm TST}^*$, and savings $\Delta_a, \Delta_b, \Delta_c$ relative to single-provider and naive-average baselines. Its core empirical claim is that RMSE-optimal weighting can yield slightly better RMSE but worse downstream operational cost than the value-oriented combination. The paper also reports about 1.8% average cost reduction and one-year training on the 2736-bus Polish system in about 20 hours with SPH (Ghazanfariharandi et al., 17 Mar 2025). In this domain, VQ is explicitly decision-focused evaluation.

A distinct but related formulation appears in medical imaging. "Observer-Usable Information as a Task-specific Image Quality Metric" proposes predictive $\mathcal{V}$-information as a task-specific image quality metric for sub-ideal observers (Lu et al., 30 Sep 2025). The central definition is

$$I_\mathcal{V}[\mathbf{X} \rightarrow \mathbf{Y}] \equiv H(\mathbf{Y}) - H_\mathcal{V}[\mathbf{Y}|\mathbf{X}],$$

where $\lambda = 4$0 is defined by minimizing expected negative log-likelihood over a restricted observer family $\lambda = 4$1. When $\lambda = 4$2, predictive $\lambda = 4$3-information reduces to mutual information or task-specific information. The paper emphasizes that $\lambda = 4$4-info can increase after preprocessing or restoration because a sub-ideal observer may find the processed image more usable, even though classical mutual information obeys the data processing inequality. In a stylized MRI restoration study, $\lambda = 4$5-info exhibits a highly linear relation with AUC for binary tasks with $\lambda = 4$6, extends naturally to three-class tasks, and remains sensitive when AUC or accuracy saturates (Lu et al., 30 Sep 2025). Here VQ is observer-usable information rather than ideal information.

A third decision-oriented example appears in wireless communications. "Precoding-Oriented CSI Feedback Design with Mutual Information Regularized VQ-VAE" evaluates discrete CSI feedback not by reconstruction fidelity alone but by the utility it provides for downstream precoding under a fixed feedback budget (Chen et al., 22 Jan 2026). The principal metric is sum achievable rate, and the paper states that the proposed method outperforms a DNN baseline with sign quantization and achieves comparable performance to variable-length neural entropy coding, especially below 10 bits of feedback overhead per UE. It also reports more uniform codeword usage and interpretable codeword structure correlated with channel state information (Chen et al., 22 Jan 2026). This is not presented as VQ in the LLM-survey sense, but it follows the same downstream-value principle: representation quality is evaluated by operational utility.

These engineering formulations show that VQ need not be tied to social or ethical judgments. It can also mean evaluating predictive objects by the decision quality, observer usability, or communication utility they induce.

6. Misconceptions, limitations, and open problems

A common misconception is that VQ names a single standardized score. The LLM survey explicitly rejects that interpretation: it provides no formal equation for VQ and no finalized aggregation over the four dimensions (Wang et al., 26 Aug 2025). Instead, VQ is a conceptual and modular framework. Another misconception is that VQ replaces capability evaluation. The same source defines it as a complementary deployment-facing layer on top of IQ, PQ, and EQ, not as a substitute for them (Wang et al., 26 Aug 2025).

Methodological limits recur across the literature. The LLM survey identifies lack of statistical rigor, benchmark fragmentation, interpretability gaps, immature user-centric and human-in-the-loop evaluation, dynamic and agentic settings, reproducibility and reliability issues, and the absence of a finalized VQ scoring standard (Wang et al., 26 Aug 2025). In semantic VQA evaluation, AVE is bounded by its covered datasets and model responses, human annotation is based on semantic similarity only, and broader generalization remains open (Ji et al., 2024). In Value-Spectrum, Schwartz values are only one value framework, first-frame screenshots are a simplification, and results are platform-dependent; the paper reports that 90.4% of screenshots were representative of the video’s main content, while around 8.8% were not representative, especially for dynamic or complex videos (Li et al., 2024). In medical imaging, $\lambda = 4$7-info depends on the chosen observer family, is estimated from finite data, and was demonstrated in a stylized synthetic setting rather than a full clinical deployment scenario (Lu et al., 30 Sep 2025).

These limitations indicate that VQ is presently more mature as an orientation than as a universally standardized protocol. The surveyed works nonetheless converge on several durable principles. Lower RMSE does not necessarily imply better operational value; Exact Match does not necessarily reflect semantic correctness; ideal information does not necessarily equal usable information; and benchmark performance does not necessarily imply acceptable economic, social, ethical, or environmental impact (Ghazanfariharandi et al., 17 Mar 2025, Ji et al., 2024, Lu et al., 30 Sep 2025, Wang et al., 26 Aug 2025). This suggests that the long-term research problem is not merely to invent new metrics, but to specify defensible mappings from model behavior to the forms of value that matter in deployment.