Qworld: One-Question-One-World Evaluation

• One-Question-One-World (Qworld) is a method for evaluating open-ended LLM responses by generating question-specific criteria using a Recursive Expansion Tree.
• The approach adapts evaluation to each question’s unique context by hierarchically expanding scenarios, perspectives, and binary criteria to capture varied response requirements.
• Empirical results on benchmarks like HealthBench demonstrate Qworld's high coverage (89%) and uniqueness (79%), offering refined model differentiation over traditional methods.

Searching arXiv for the Qworld paper and closely related evaluation work to ground the article. arxiv_search(query="Qworld question-specific evaluation criteria LLM open-ended questions HealthBench", max_results=10, sort_by="relevance") arxiv_search(query="HealthBench open-ended evaluation criteria LLM rubric generation arXiv", max_results=10, sort_by="relevance") One-Question-One-World (Qworld) is a method for evaluating open-ended LLM responses by generating question-specific evaluation criteria rather than applying a fixed dataset-level rubric. In this formulation, each question implies its own “world”: a structured set of scenarios, perspectives, and fine-grained criteria that specify what a high-quality answer must address. Qworld operationalizes that idea with a Recursive Expansion Tree (RET) that explores the evaluation space induced by a single question and converts it into weighted, binary, checkable criteria (Gao et al., 6 Mar 2026).

1. Problem setting and core concept

Qworld is motivated by a specific difficulty in benchmarking LLMs on open-ended tasks. Response quality depends on the question’s context, intent, audience, stakes, and implicit constraints, so dataset-level rubrics and binary overall scoring can collapse distinctions that matter at the level of individual prompts. The method is designed for settings in which a medical counseling query may require safety warnings, uncertainty communication, and triage guidance, while a scientific explanation may instead require rigor, pedagogical clarity, or handling of assumptions. In that sense, the relevant evaluation dimensions are induced by the semantics of the question itself, not by the benchmark label alone (Gao et al., 6 Mar 2026).

The term “world” in Qworld does not denote a world model in the usual knowledge-representation sense. It refers to the question-implied set of elements that a strong answer should cover. The authors state that they “refer to this set of elements as a question’s ‘world,’ which specifies what a high-quality answer must address.” A central misconception is therefore that Qworld is merely a better checklist generator. Its actual claim is stronger: criterion generation should be treated as structured coverage of the evaluation axes implied by a question, not as one-shot rubric drafting.

This framing places Qworld against three classes of baseline practice. First, dataset-level criteria assume a benchmark-wide common rubric. Second, coarse rubric dimensions such as “accuracy / completeness / communication” are often too blunt. Third, single-pass criterion generation explores only one view of the question and therefore tends to miss alternative evaluation axes. Qworld is proposed as a response to all three.

2. Formalization and Recursive Expansion Tree

Let $Q$ denote a question and $A$ a candidate answer. For each question $Q_i$, Qworld generates a criterion set $\mathcal{C}_i$. Each criterion $c \in \mathcal{C}_i$ specifies a verifiable condition and an associated score $s_c(A,Q_i)$. In the main method description, criterion scoring is binary with a positive importance weight: $s_c(A, Q_i) \in \{0, \alpha_c\},$ and the overall score is normalized after aggregation: $$S(A, Q_i) = F_{\mathrm{norm}}\!\left(\sum_{c \in \mathcal{C}_i} s_c(A, Q_i)\right).$$

The appendix gives the fuller signed formulation. Each criterion is a tuple $(r_c,\alpha_c)$, where $r_c$ is a verifiable criterion statement and $A$0. Positive $A$1 rewards desirable behavior; negative $A$2 penalizes harmful or undesirable behavior. Criterion-level scoring is

$A$3

If $A$4 is the subset of positive-weight criteria, the normalized question-level score is

$A$5

Qworld generates $A$6 through a three-level hierarchy. The levels are:

1. Scenarios ($A$7): contexts or materially distinct settings implied by the question that change what “good” means.
2. Perspectives ($A$8): evaluation axes that matter within a scenario.
3. Criteria ($A$9): fine-grained, binary, checkable conditions with importance weights.

The output criterion set is the set of leaf nodes: $Q_i$0

The core algorithm is the Recursive Expansion Tree (RET). RET uses two expansion operators at each non-leaf level. Hierarchical expansion

$Q_i$1

decomposes a node $Q_i$2 at level $Q_i$3 into finer-grained child nodes at level $Q_i$4. Horizontal expansion

$Q_i$5

adds missing sibling nodes at the current level to improve coverage. The paper’s conceptual distinction is that hierarchical expansion contributes granularity, whereas horizontal expansion contributes coverage.

The algorithm is given as: $Q_i$6 followed, for $Q_i$7 to $Q_i$8, by repeated horizontal expansion

$Q_i$9

and then hierarchical decomposition

$\mathcal{C}_i$0

The implementation uses expansion widths $\mathcal{C}_i$1 for scenarios, $\mathcal{C}_i$2 for perspectives, and $\mathcal{C}_i$3 for criteria (Gao et al., 6 Mar 2026).

3. Criterion construction, prompting structure, and scoring semantics

Qworld is implemented as a sequence of specialized prompt-driven agents over JSON-structured outputs. The pipeline includes a Scenario Analyzer, Scenario Expander, Perspective Analyzer, Perspective Expander, Perspective Reviewer, Criteria Generator, Criteria Expander, Criteria Reviewer, Negative Checker, and Score Assigner (Gao et al., 6 Mar 2026).

The scenario stage seeks a “minimal, non-redundant set of distinct scenarios that materially change evaluation criteria.” The perspective stage produces “4–7 scenario-specific, non-overlapping evaluation perspectives” for a scenario, with each perspective including 3–5 sub-aspects. The criteria stage then instantiates perspective-specific binary items.

The criterion semantics are tightly specified. Each criterion must be:

• Binary: evaluable as YES/NO.
• Scenario-specific: tied to the scenario and perspective.
• Balanced: including positive criteria and negative criteria, with negative criteria reserved for harmful, misleading, or critically wrong behavior.
• Structured: in the form $\mathcal{C}_i$4.
• Diverse: covering all sub-aspects of the perspective.

The scoring scale is also explicit. Positive criteria are assigned $\mathcal{C}_i$5 to $\mathcal{C}_i$6, where $\mathcal{C}_i$7 is “critical safety/core,” $\mathcal{C}_i$8–$\mathcal{C}_i$9 are “important completeness,” $c \in \mathcal{C}_i$0–$c \in \mathcal{C}_i$1 are “quality enhancers,” and $c \in \mathcal{C}_i$2–$c \in \mathcal{C}_i$3 are “minor.” Negative criteria are assigned $c \in \mathcal{C}_i$4 to $c \in \mathcal{C}_i$5, where $c \in \mathcal{C}_i$6 is “dangerous,” $c \in \mathcal{C}_i$7 to $c \in \mathcal{C}_i$8 are “major omission,” $c \in \mathcal{C}_i$9 to $s_c(A,Q_i)$0 are “quality issue,” and $s_c(A,Q_i)$1 to $s_c(A,Q_i)$2 are “minor.”

Review and consolidation are a substantive part of the method rather than a postprocessing detail. The Perspective Reviewer removes perspectives that are off-topic, redundant, or vague, and merges those that express the same evaluation angle. The Criteria Reviewer deduplicates and merges overlapping criteria, keeps the most precise wording, replaces hard-coded facts with “current/official/latest” requirements, and balances positive and negative criteria. The pruning logic is therefore prompt-driven rather than derived from a separate formal objective.

A useful technical point is that Qworld’s unit of evaluation is the individual weighted binary criterion, not a coarse rubric category. Perspectives are intermediate organizational structures, and later analysis may cluster criteria into semantic dimensions, but answer scoring is defined criterion-by-criterion.

4. Empirical evaluation and benchmark behavior

Qworld is evaluated on two datasets with distinct roles. HealthBench contains 5,000 open-ended medical user–assistant queries and includes physician-designed, question-level criteria. It is used both for intrinsic criterion-quality evaluation and for model benchmarking. Humanity’s Last Exam (HLE) is a PhD-level benchmark without expert criteria and is used to test generalization beyond expert-annotated medical evaluation (Gao et al., 6 Mar 2026).

The paper reports two headline intrinsic results on HealthBench: Qworld covers 89\% of expert-authored criteria and generates 79\% novel criteria validated by human experts. On a 1K subset of HealthBench, its criterion-quality comparison against prior methods is:

Method	Coverage	Uniqueness
TICK	0.46	0.24
RocketEval	0.53	0.26
OpenRubrics	0.54	0.37
EvalAgent	0.83	0.50
Qworld	0.89	0.79
Qworld$s_c(A,Q_i)$3	0.90	0.82

Human evaluation further reports Insight = 0.83 and Granularity = 0.85 for Qworld, with Insight = 0.84 for Qworld$s_c(A,Q_i)$4. Figure 1 reports that the Value of unique criteria remains above 0.90 as the number of criteria increases from 15 to 60, which the paper interprets as evidence that later expansion steps continue to add useful constraints rather than mostly redundancy.

The paper also evaluates Specificity and Implicitness. Specificity is defined via NIWF: $s_c(A,Q_i)$5 with an appendix variant replacing $s_c(A,Q_i)$6 by corpus $s_c(A,Q_i)$7. Implicitness is defined through lexical overlap: $s_c(A,Q_i)$8

$s_c(A,Q_i)$9

Qworld reports Specificity = 0.09 and Implicitness = 0.87, while Qworld$s_c(A, Q_i) \in \{0, \alpha_c\},$0 reports 0.10 and 0.89.

The ablation study isolates the contribution of RET. Compared with CoT, Self-Reflection (SR), and Tree Decomposition (TD), RET yields the strongest intrinsic results: Coverage = 0.89 and Uniqueness = 0.79, versus 0.67 / 0.40 for CoT, 0.84 / 0.70 for SR, and 0.77 / 0.65 for TD. This supports the paper’s claim that both hierarchical decomposition and horizontal expansion are necessary.

Qworld is also used to benchmark 11 frontier LLMs: GPT-5, GPT-5-mini, GPT-4.1, GPT-4.1-mini, GPT-4.1-nano, Gemini 3 Flash, Claude Sonnet 4.5, Grok-4.1-Fast, DeepSeek-V3.2, Qwen3-30B, and Llama-3.1-70B. On HealthBench, Qworld-based scores are approximately 20\% lower than the official $s_c(A, Q_i) \in \{0, \alpha_c\},$1, which the authors interpret as reduced saturation and increased separation. The ranking changes are substantive: Qwen3-30B moves from rank 6 to rank 2, while Grok-4.1-Fast moves from rank 4 to rank 6. On HLE, GPT-5 moves to rank 1 and Claude Sonnet 4.5 moves from 6th to 3rd. The paper’s claim is not merely that Qworld reorders leaderboards, but that it distinguishes models on dimensions that final-answer correctness alone does not capture.

5. Relation to prior evaluation paradigms and substantive significance

Qworld is positioned against four prior methodological families. Dataset-level criteria define a benchmark-wide rubric and therefore assume a common evaluative structure across all questions. Question-level single-pass prompting, represented by methods such as WildBench, TICK, and RocketEval, generates criteria per question but usually from a single view. Contrastive or preference-based induction, represented by OpenRubrics, infers criteria by comparing better and worse responses. Retrieval-grounded methods, represented by EvalAgent, add external evidence but do not by themselves perform structured expansion of the evaluation space (Gao et al., 6 Mar 2026).

The paper’s substantive claim is that Qworld changes not only the source of criteria but the ontology of evaluation itself. A question is treated as inducing a structured space of latent requirements. Those requirements may include not only factual correctness and completeness but also dimensions such as long-term impact, equity, error handling, and interdisciplinary reasoning. On HealthBench, the paper lists dimensions such as Clarity, Empathy & Support, User Empowerment, Evidence Quality, Health Equity & Accessibility, Transparency, Sustainability & Long-term Impact, Emergency Recognition, Safety & Risk Management, Factual Correctness, and Guideline Adherence. On HLE, the generated dimensions shift toward Assumption Clarity, Mathematical Rigor, Visual / Geometric Reasoning, Pedagogical Effectiveness, Notation / Format Precision, and Creativity.

This supports a narrower but technically precise interpretation of the phrase “One-Question-One-World.” The “world” is the question-implied evaluation structure, not a literal environment model and not a static task-level rubric. A plausible implication is that Qworld is best understood as an adaptive evaluation formalism for open-ended QA, where semantic variation across prompts is treated as first-class signal rather than noise.

The appendix’s heat-rash case study makes that point concrete. For one medical question, Qworld generates 13 scenarios, 26 perspectives, and 36 criteria. The scenarios include Prevention-Focused Advice, Treatment-Focused Advice for Mild Cases, Portable Solutions for On-the-Go Relief, Long-Term Lifestyle and Environmental Modifications, Workplace or Occupational Exposure, Emergency or Acute Exacerbation Management, Cultural or Religious Clothing Constraints, and Remote or Resource-Limited Settings. The resulting perspectives and criteria extend beyond standard safety/completeness checks to include affordability, eco-friendliness, public discretion, patch-testing, reusable-item hygiene, and workplace constraints. This case is meant to show that even a seemingly narrow question can imply a broad evaluation world.

6. Limitations, dependencies, and open technical questions

The paper does not present a dedicated limitations section, but several constraints are explicit or directly implied by the experiments. First, Qworld depends materially on the capability of the generator model. With the GPT-4.1 family as generator, criterion quality scales monotonically: Coverage / Uniqueness are 0.89 / 0.79 for GPT-4.1, 0.84 / 0.77 for GPT-4.1-mini, and 0.75 / 0.68 for GPT-4.1-nano (Gao et al., 6 Mar 2026).

Second, the intrinsic metrics inherit dependence on judge reliability. Coverage and Uniqueness are computed with an LLM judge, even though robustness is checked across judge/model pairs. The paper reports Coverage = 0.89 with GPT-4.1 as both generator and judge, 0.96 when GPT-4.1-generated criteria are judged by Qwen3-30B, 0.87 when Qwen3-30B generates and GPT-4.1 judges, and 0.93 when Qwen3-30B does both generation and judging. This suggests robustness, but it does not remove the general issue that semantic coverage judgments are model-mediated.

Third, RET uses fixed expansion widths $s_c(A, Q_i) \in \{0, \alpha_c\},$2, $s_c(A, Q_i) \in \{0, \alpha_c\},$3, and $s_c(A, Q_i) \in \{0, \alpha_c\},$4. The paper treats this as a practical design choice rather than a theoretically optimal stopping rule. Too little expansion may miss relevant dimensions; too much expansion may increase redundancy and review burden. Consolidation is handled by reviewer prompts rather than a separate formal criterion.

Fourth, Qworld is more computationally involved than single-pass generation. The paper does not provide a runtime or dollar-cost table, but the pipeline includes multiple expansion and review stages, which makes it structurally heavier than direct prompting. This suggests a tradeoff between evaluation richness and evaluation cost.

Fifth, domain transfer is asymmetric across benchmarks. HealthBench permits intrinsic evaluation because it includes physician-authored criteria. HLE does not. As a result, the paper can show that Qworld changes model differentiation on HLE, but it cannot directly report expert-criterion coverage in the same way. A plausible implication is that evaluation of question-specific criterion generators remains easier in domains with existing high-quality expert annotations.

At a broader level, Qworld is more diagnostic than normative. It demonstrates that question-level evaluation worlds can be generated and that they reveal model differences hidden by coarse rubrics. It does not claim that RET is the final or unique decomposition scheme for open-ended evaluation. Its main lasting contribution is the formulation itself: evaluation should adapt to the semantic demands of each question rather than rely on fixed task-level criteria.