---
title: 'CresOWLve: Creative Problem-Solving Benchmark'
url: https://www.emergentmind.com/topics/cresowlve
type: topic
---

# CresOWLve: Creative Problem-Solving Benchmark

CresOWLve is a benchmark for creative problem-solving over real-world knowledge. It is designed to test whether large language models can do more than retrieve facts or execute isolated reasoning primitives: the benchmark requires models to combine reasoning, lateral thinking, analogy-making, abstraction, commonsense reasoning, and language-based creativity in order to solve single-answer puzzles grounded in authentic knowledge [2604.03374]. The benchmark is built from questions from the Russian intellectual game *What? Where? When?*, and its central premise is that the main difficulty lies not in raw factual access alone, but in making the non-obvious creative connection that links disparate clues into a correct solution.

## 1. Conceptual scope and benchmark gap

CresOWLve was introduced against the observation that prior LLM evaluations tend to isolate components of creative cognition rather than test the broader process of creative problem-solving. The motivating contrast is between benchmarks that focus on factual reasoning, math or logical inference, commonsense QA, lateral-thinking brainteasers, divergent-thinking tests, or analogy and abstraction tasks, and real-world puzzle solving, which may require vertical reasoning, lateral thinking, convergent and divergent thinking, analogy-making, and commonsense background knowledge within a single item [2604.03374].

A defining feature of CresOWLve is that it uses problems grounded in real-world knowledge rather than contrived setups. The source material comes from *What? Where? When?*, where expert players solve carefully crafted questions by combining facts across domains in surprising ways. This makes CresOWLve distinct from creativity benchmarks centered on open-ended ideation or creative writing. Each puzzle has a single answer, or a few acceptable answers, so the challenge is not unrestricted generation but arriving at the intended solution through a creative inferential path.

The benchmark is positioned relative to prior resources such as BRAINTEASER, RiddleSense, Briangle, SPLAT, LatEval, Columbus, Connections, DAT, McGyver, OnlyConnect, CREATE, analogical reasoning benchmarks, coding and math creativity benchmarks, and OMEGA. The paper distinguishes CresOWLve by combining convergent and divergent thinking, lateral thinking, analogy, commonsense reasoning, real-world knowledge, and broad domain coverage, whereas most earlier benchmarks cover only a subset of these dimensions.

## 2. Dataset construction and annotation

CresOWLve is built from **3,789** public questions from `db.chgk.info`, the archive of *What? Where? When?* questions spanning more than 50 years. Each source question includes a short answer, an explanation, and a human-assigned difficulty rating from 1 to 5. The construction process attempts to preserve difficulty balance by collecting roughly **700 questions per difficulty level** [2604.03374].

The curation pipeline removes items that would not survive translation or would require extra materials beyond text. Questions requiring handouts, images, hidden text, or other external artifacts are filtered; Russian-specific or untranslatable questions are also excluded; the remaining questions are translated into English using GPT-4o; and three authors then conduct human validation to remove additional items that fail answerability, translation, or portability criteria.

| Aspect | Value | Notes |
|---|---:|---|
| Initial pool | 3,789 | Public questions from `db.chgk.info` |
| Removed: external materials | 295 | Handouts, images, hidden text, related artifacts |
| Removed: Russian-specific / untranslatable | 799 | Wordplay or deeply culture-bound references |
| Removed after human validation | 282 | Three-author validation |
| Final dataset | 2,413 | Total after filtering |
| Core creative subset | 2,061 | Used for main benchmark evaluation |
| Factual subset | 352 | Separate comparison subset |

The resulting benchmark is released in two language versions, **CresOWLve-Ru** and **CresOWLve-En**. It is also annotated along several axes. Knowledge domains are labeled first with **541 fine-grained labels** and then collapsed to **34 coarse domains**. Additional annotations cover creative concepts or thinking strategies, including lateral thinking, analogy, abstraction, commonsense reasoning, jokes, puns, and metaphors, as well as cultures and demographics involved in the clues. The paper reports that most questions involve **2–4 knowledge domains** and often **multiple creativity concepts**. This suggests that the benchmark is structured to capture compositionality across both epistemic content and cognitive strategy.

## 3. Cognitive skills and task structure

CresOWLve is explicitly designed to probe a bundle of cognitive abilities rather than a single capability. The benchmark targets reasoning, lateral thinking, analogy-making, abstraction, commonsense reasoning, convergent and divergent thinking, and language-based creativity such as puns, metaphors, idioms, jokes, and figurative language [2604.03374]. Although divergent thinking is part of the conceptual framing, the benchmark is primarily convergent in form because puzzles usually have one correct answer.

This design matters because it separates creative problem-solving from generic fluency. A model can possess broad factual coverage and still fail if it does not reinterpret a clue, transfer structure across domains, or notice the intended associative bridge. The paper’s framing therefore treats creativity as a mode of constrained integration: the model must retrieve relevant knowledge, suppress plausible but incorrect semantic trajectories, and converge on the one answer compatible with the puzzle’s clue structure.

The benchmark’s complexity does not reduce to obvious structural proxies. The authors test three such proxies—number of domains involved, semantic distance between domains, and number of atomic facts needed—and find **no significant correlation** with either assigned difficulty or model performance. This is an important negative result. It indicates that CresOWLve’s difficulty is not simply an effect of “more facts” or “more domains,” but is tied to the need for a specifically creative synthesis.

## 4. Evaluation protocol and model classes

Evaluation is conducted in an **open-ended QA** setting. For non-thinking models, the paper uses **Chain-of-Thought prompting**. For thinking models, it uses the model’s standard reasoning prompt with varying effort settings, including adaptive, none, minimal, low, medium, and high depending on the model family [2604.03374].

Two metrics are reported. The first is **Exact Match Accuracy**, based on normalization of lowercase, punctuation, and Unicode for Russian followed by direct comparison with the gold answer. The second is **LLM-as-a-Judge Accuracy**, using **GPT-4o** as judge. In the judge setup, GPT-4o sees the question, gold answer, comments, acceptable answers, and model answer, and is instructed to ignore typos, articles, capitalization, formatting differences, and even word-count mismatches if the meaning is correct. For factual-versus-creative comparisons, the paper uses **bootstrap sampling with 1,000 iterations** to balance sample sizes.

The evaluated model set spans both non-thinking and thinking systems. The **non-thinking models** are OLMo-2-32B-Instruct, C4AI-command-a, Llama-3.3-70B-Instruct, Mistral-Large-3-675B-Instruct, Qwen3-235B-A22B-Instruct, GPT-4.1-mini, and GPT-4.1. The **thinking models** are Qwen3.5-397B-A17B, Qwen3-235B-A22B-Thinking, DeepSeek-V3.2, GLM-5, GPT-5.4, Gemini-3-Flash, and Gemini-3.1-Pro.

A further structural distinction is the benchmark’s bilingual design. Because CresOWLve is released in both Russian and English, evaluation can probe not only overall creative problem-solving performance but also cross-lingual robustness, translation sensitivity, and possible contamination effects.

## 5. Quantitative results and failure modes

CresOWLve is reported to be highly challenging. Overall performance ranges from under **10%** to above **80%** depending on model, language, and metric, but the aggregate pattern is sharply stratified by model type: **non-thinking models stay below 30% overall**, whereas thinking models perform substantially better [2604.03374].

Among non-thinking systems, the strongest reported results are modest. **GPT-4.1** reaches **13.44 EM / 24.99 judge** on English and **17.03 EM / 26.06 judge** on Russian. **Mistral-Large-3-675B** reaches **11.26 EM / 17.61 judge** on English and **15.48 EM / 22.22 judge** on Russian. Among thinking models, **Gemini-3.1-Pro** is the strongest overall: on English it reaches **46.43 EM / 67.54 judge** at low effort, **49.10 / 72.97** at medium, and **51.29 / 76.03** at high; on Russian it reaches **64.10 / 80.49** at low effort, **65.79 / 83.60** at medium, and **67.78 / 85.74** at high. **Gemini-3-Flash** also performs strongly, reaching **35.08 EM / 52.16 judge** on English at medium effort and **51.82 / 67.25** on Russian. Increasing thinking effort generally improves results, especially for GPT-5.4 and Gemini models.

One of the paper’s central empirical findings is the **creative-versus-factual gap**. Under LLM-judge evaluation on **CresOWLve-En**, the drop from factual to creative questions ranges from **-6.08%** for Gemini-3.1-Pro at high effort to **-17.21%** for Mistral-Large-3-675B-Instruct. On **CresOWLve-Ru**, the gap is smaller for the best Gemini models, around **-2.14% to -2.73%**. The authors interpret this as evidence that the principal bottleneck is not factual retrieval alone, but the ability to form the creative connection.

Performance declines monotonically with difficulty level, and even strong thinking models lose **over 20%** from “Very Simple” to “Very Hard” under the judge metric. The benchmark therefore validates its own difficulty annotations empirically. At the same time, the paper reports no significant correlation between performance and simple complexity proxies, reinforcing the view that the latent difficulty variable is creative integration rather than superficial combinatorial load.

The failure analysis supports that diagnosis. Manual inspection of **150** errors from **Gemini-3-Flash (medium)** yields seven categories: **Missing creative connection**, **Overthinking**, **Hallucination / unsupported fabrication**, **Incorrect concept anchoring**, **Wrong hypothesis**, **Wrong reference**, and **Clue misinterpretation**. The most common errors are **Missing creative connection** and **Clue misinterpretation**. Hallucination rates and wrong-reference rates are described as often relatively low, which suggests that failure is not primarily ignorance but a breakdown in integrating retrieved knowledge along the intended associative path. The illustrative examples follow a common pattern: the model finds a semantically plausible explanation, but not the intended one. A puzzle whose answer is **“The American Farmer”** elicits an over-elaborated answer about ships and the Mona Lisa; a cinema puzzle whose answer is **close-up** produces **plan**; a Thomas Jefferson clue whose answer is **the white and black kings** instead elicits **chains and shackles**; and a Henry Moore puzzle whose answer is **curvy lines** produces **hollow spaces**.

## 6. Cross-lingual behavior, human comparison, and limitations

Cross-lingual behavior is mixed. Many non-thinking models do slightly worse on Russian, whereas some thinking models, especially proprietary ones, do better on Russian [2604.03374]. The paper suggests that this may reflect multilingual reasoning strengths at higher effort, but it also raises the possibility of **data contamination**. The appendix contamination test identifies this as plausible for **Gemini-3.1-Pro**: on **580** Russian questions that the model answered correctly in Russian but not in English, it achieved **72%**, **73%**, and **74%** accuracy across three contamination quiz runs, compared with a **25% random baseline**.

The benchmark does not include a formal human study under matched conditions. Instead, the paper treats the fact that the questions were originally solved correctly in game episodes by expert players as an implicit human upper bound. This matters for interpretation. A plausible implication is that CresOWLve should not be read as a general measure of all creativity, but as a measure of one specific and demanding slice of creativity: single-answer, clue-driven, real-world puzzle solving.

The limitations are explicitly stated. They include **subjectivity** in labeling questions as factual versus creative and in the error taxonomy; **potential data contamination** because some questions are publicly available; **cultural bias and translation issues** due to the Russian source material; and the fact that the benchmark is centered on **single-answer puzzles**, capturing only one slice of creativity rather than all forms. Human validation is described as strong, but it is not a full human-performance study across the same conditions as model evaluation.

A common misconception would be to treat strong factual retrieval as sufficient evidence of creative reasoning. CresOWLve argues against that inference directly: high factual knowledge does not guarantee the ability to combine clues under non-obvious constraints. Extended “thinking” helps, but even top models remain far from a general solution to this form of creative problem-solving.

## 7. Nomenclature and adjacent uses of the term

In the supplied sources, the name “CresOWLve” is not used uniformly. The OWLViz paper explicitly states that **“CresOWLve” is not a separate benchmark, task, or concept defined in the paper**, and that the paper never introduces a term spelled “CresOWLve” [2503.07631]. That clarification is relevant because OWLViz is itself an open-world visual question answering benchmark concerned with visual understanding, web exploration, and specialized tool usage, but it is conceptually distinct from CresOWLve’s focus on creative problem-solving over real-world knowledge.

A separate supplied description identifies CresOWLve as the proof-of-concept **Chromium-based “memento-aware” browser** presented in “A Chromium-based Memento-aware Web Browser” [2104.13361]. In that account, the system detects archived pages via the **`Memento-Datetime`** HTTP header, surfaces archival state in Chromium’s location bar and page-info UI, distinguishes among pure mementos, mixed archival content, and **“zombie” mementos**, and adds a **bookmark as archive** workflow supporting **Internet Archive**, **Archive.today**, and **Megalodon.jp**. That browser work addresses archival provenance and browser UI, not creative puzzle solving.

Within the benchmark literature proper, however, CresOWLve denotes the 2026 evaluation suite for creative problem-solving over real-world knowledge. Its substantive contribution is to operationalize a narrow but consequential question for LLM research: whether a model can retrieve relevant facts from multiple domains and then make the non-obvious creative leap required to convert those facts into a correct solution.

Source: https://www.emergentmind.com/topics/cresowlve