CresOWLve: Creative Problem-Solving Benchmark
- CresOWLve is a benchmark for creative problem-solving that integrates lateral thinking, analogy-making, and abstraction to solve real-world puzzles.
- It is built on 2,413 curated puzzles from 'What? Where? When?' and balances questions across difficulty levels to test diverse cognitive skills.
- Evaluation reveals a creative-versus-factual gap, highlighting the challenges models face in synthesizing non-obvious, interdisciplinary clues.
CresOWLve is a benchmark for creative problem-solving over real-world knowledge. It is designed to test whether LLMs can do more than retrieve facts or execute isolated reasoning primitives: the benchmark requires models to combine reasoning, lateral thinking, analogy-making, abstraction, commonsense reasoning, and language-based creativity in order to solve single-answer puzzles grounded in authentic knowledge (Ismayilzada et al., 3 Apr 2026). The benchmark is built from questions from the Russian intellectual game What? Where? When?, and its central premise is that the main difficulty lies not in raw factual access alone, but in making the non-obvious creative connection that links disparate clues into a correct solution.
1. Conceptual scope and benchmark gap
CresOWLve was introduced against the observation that prior LLM evaluations tend to isolate components of creative cognition rather than test the broader process of creative problem-solving. The motivating contrast is between benchmarks that focus on factual reasoning, math or logical inference, commonsense QA, lateral-thinking brainteasers, divergent-thinking tests, or analogy and abstraction tasks, and real-world puzzle solving, which may require vertical reasoning, lateral thinking, convergent and divergent thinking, analogy-making, and commonsense background knowledge within a single item (Ismayilzada et al., 3 Apr 2026).
A defining feature of CresOWLve is that it uses problems grounded in real-world knowledge rather than contrived setups. The source material comes from What? Where? When?, where expert players solve carefully crafted questions by combining facts across domains in surprising ways. This makes CresOWLve distinct from creativity benchmarks centered on open-ended ideation or creative writing. Each puzzle has a single answer, or a few acceptable answers, so the challenge is not unrestricted generation but arriving at the intended solution through a creative inferential path.
The benchmark is positioned relative to prior resources such as BRAINTEASER, RiddleSense, Briangle, SPLAT, LatEval, Columbus, Connections, DAT, McGyver, OnlyConnect, CREATE, analogical reasoning benchmarks, coding and math creativity benchmarks, and OMEGA. The paper distinguishes CresOWLve by combining convergent and divergent thinking, lateral thinking, analogy, commonsense reasoning, real-world knowledge, and broad domain coverage, whereas most earlier benchmarks cover only a subset of these dimensions.
2. Dataset construction and annotation
CresOWLve is built from 3,789 public questions from db.chgk.info, the archive of What? Where? When? questions spanning more than 50 years. Each source question includes a short answer, an explanation, and a human-assigned difficulty rating from 1 to 5. The construction process attempts to preserve difficulty balance by collecting roughly 700 questions per difficulty level (Ismayilzada et al., 3 Apr 2026).
The curation pipeline removes items that would not survive translation or would require extra materials beyond text. Questions requiring handouts, images, hidden text, or other external artifacts are filtered; Russian-specific or untranslatable questions are also excluded; the remaining questions are translated into English using GPT-4o; and three authors then conduct human validation to remove additional items that fail answerability, translation, or portability criteria.
| Aspect | Value | Notes |
|---|---|---|
| Initial pool | 3,789 | Public questions from db.chgk.info |
| Removed: external materials | 295 | Handouts, images, hidden text, related artifacts |
| Removed: Russian-specific / untranslatable | 799 | Wordplay or deeply culture-bound references |
| Removed after human validation | 282 | Three-author validation |
| Final dataset | 2,413 | Total after filtering |
| Core creative subset | 2,061 | Used for main benchmark evaluation |
| Factual subset | 352 | Separate comparison subset |
The resulting benchmark is released in two language versions, CresOWLve-Ru and CresOWLve-En. It is also annotated along several axes. Knowledge domains are labeled first with 541 fine-grained labels and then collapsed to 34 coarse domains. Additional annotations cover creative concepts or thinking strategies, including lateral thinking, analogy, abstraction, commonsense reasoning, jokes, puns, and metaphors, as well as cultures and demographics involved in the clues. The paper reports that most questions involve 2–4 knowledge domains and often multiple creativity concepts. This suggests that the benchmark is structured to capture compositionality across both epistemic content and cognitive strategy.
3. Cognitive skills and task structure
CresOWLve is explicitly designed to probe a bundle of cognitive abilities rather than a single capability. The benchmark targets reasoning, lateral thinking, analogy-making, abstraction, commonsense reasoning, convergent and divergent thinking, and language-based creativity such as puns, metaphors, idioms, jokes, and figurative language (Ismayilzada et al., 3 Apr 2026). Although divergent thinking is part of the conceptual framing, the benchmark is primarily convergent in form because puzzles usually have one correct answer.
This design matters because it separates creative problem-solving from generic fluency. A model can possess broad factual coverage and still fail if it does not reinterpret a clue, transfer structure across domains, or notice the intended associative bridge. The paper’s framing therefore treats creativity as a mode of constrained integration: the model must retrieve relevant knowledge, suppress plausible but incorrect semantic trajectories, and converge on the one answer compatible with the puzzle’s clue structure.
The benchmark’s complexity does not reduce to obvious structural proxies. The authors test three such proxies—number of domains involved, semantic distance between domains, and number of atomic facts needed—and find no significant correlation with either assigned difficulty or model performance. This is an important negative result. It indicates that CresOWLve’s difficulty is not simply an effect of “more facts” or “more domains,” but is tied to the need for a specifically creative synthesis.
4. Evaluation protocol and model classes
Evaluation is conducted in an open-ended QA setting. For non-thinking models, the paper uses Chain-of-Thought prompting. For thinking models, it uses the model’s standard reasoning prompt with varying effort settings, including adaptive, none, minimal, low, medium, and high depending on the model family (Ismayilzada et al., 3 Apr 2026).
Two metrics are reported. The first is Exact Match Accuracy, based on normalization of lowercase, punctuation, and Unicode for Russian followed by direct comparison with the gold answer. The second is LLM-as-a-Judge Accuracy, using GPT-4o as judge. In the judge setup, GPT-4o sees the question, gold answer, comments, acceptable answers, and model answer, and is instructed to ignore typos, articles, capitalization, formatting differences, and even word-count mismatches if the meaning is correct. For factual-versus-creative comparisons, the paper uses bootstrap sampling with 1,000 iterations to balance sample sizes.
The evaluated model set spans both non-thinking and thinking systems. The non-thinking models are OLMo-2-32B-Instruct, C4AI-command-a, Llama-3.3-70B-Instruct, Mistral-Large-3-675B-Instruct, Qwen3-235B-A22B-Instruct, GPT-4.1-mini, and GPT-4.1. The thinking models are Qwen3.5-397B-A17B, Qwen3-235B-A22B-Thinking, DeepSeek-V3.2, GLM-5, GPT-5.4, Gemini-3-Flash, and Gemini-3.1-Pro.
A further structural distinction is the benchmark’s bilingual design. Because CresOWLve is released in both Russian and English, evaluation can probe not only overall creative problem-solving performance but also cross-lingual robustness, translation sensitivity, and possible contamination effects.
5. Quantitative results and failure modes
CresOWLve is reported to be highly challenging. Overall performance ranges from under 10% to above 80% depending on model, language, and metric, but the aggregate pattern is sharply stratified by model type: non-thinking models stay below 30% overall, whereas thinking models perform substantially better (Ismayilzada et al., 3 Apr 2026).
Among non-thinking systems, the strongest reported results are modest. GPT-4.1 reaches 13.44 EM / 24.99 judge on English and 17.03 EM / 26.06 judge on Russian. Mistral-Large-3-675B reaches 11.26 EM / 17.61 judge on English and 15.48 EM / 22.22 judge on Russian. Among thinking models, Gemini-3.1-Pro is the strongest overall: on English it reaches 46.43 EM / 67.54 judge at low effort, 49.10 / 72.97 at medium, and 51.29 / 76.03 at high; on Russian it reaches 64.10 / 80.49 at low effort, 65.79 / 83.60 at medium, and 67.78 / 85.74 at high. Gemini-3-Flash also performs strongly, reaching 35.08 EM / 52.16 judge on English at medium effort and 51.82 / 67.25 on Russian. Increasing thinking effort generally improves results, especially for GPT-5.4 and Gemini models.
One of the paper’s central empirical findings is the creative-versus-factual gap. Under LLM-judge evaluation on CresOWLve-En, the drop from factual to creative questions ranges from -6.08% for Gemini-3.1-Pro at high effort to -17.21% for Mistral-Large-3-675B-Instruct. On CresOWLve-Ru, the gap is smaller for the best Gemini models, around -2.14% to -2.73%. The authors interpret this as evidence that the principal bottleneck is not factual retrieval alone, but the ability to form the creative connection.
Performance declines monotonically with difficulty level, and even strong thinking models lose over 20% from “Very Simple” to “Very Hard” under the judge metric. The benchmark therefore validates its own difficulty annotations empirically. At the same time, the paper reports no significant correlation between performance and simple complexity proxies, reinforcing the view that the latent difficulty variable is creative integration rather than superficial combinatorial load.
The failure analysis supports that diagnosis. Manual inspection of 150 errors from Gemini-3-Flash (medium) yields seven categories: Missing creative connection, Overthinking, Hallucination / unsupported fabrication, Incorrect concept anchoring, Wrong hypothesis, Wrong reference, and Clue misinterpretation. The most common errors are Missing creative connection and Clue misinterpretation. Hallucination rates and wrong-reference rates are described as often relatively low, which suggests that failure is not primarily ignorance but a breakdown in integrating retrieved knowledge along the intended associative path. The illustrative examples follow a common pattern: the model finds a semantically plausible explanation, but not the intended one. A puzzle whose answer is “The American Farmer” elicits an over-elaborated answer about ships and the Mona Lisa; a cinema puzzle whose answer is close-up produces plan; a Thomas Jefferson clue whose answer is the white and black kings instead elicits chains and shackles; and a Henry Moore puzzle whose answer is curvy lines produces hollow spaces.
6. Cross-lingual behavior, human comparison, and limitations
Cross-lingual behavior is mixed. Many non-thinking models do slightly worse on Russian, whereas some thinking models, especially proprietary ones, do better on Russian (Ismayilzada et al., 3 Apr 2026). The paper suggests that this may reflect multilingual reasoning strengths at higher effort, but it also raises the possibility of data contamination. The appendix contamination test identifies this as plausible for Gemini-3.1-Pro: on 580 Russian questions that the model answered correctly in Russian but not in English, it achieved 72%, 73%, and 74% accuracy across three contamination quiz runs, compared with a 25% random baseline.
The benchmark does not include a formal human study under matched conditions. Instead, the paper treats the fact that the questions were originally solved correctly in game episodes by expert players as an implicit human upper bound. This matters for interpretation. A plausible implication is that CresOWLve should not be read as a general measure of all creativity, but as a measure of one specific and demanding slice of creativity: single-answer, clue-driven, real-world puzzle solving.
The limitations are explicitly stated. They include subjectivity in labeling questions as factual versus creative and in the error taxonomy; potential data contamination because some questions are publicly available; cultural bias and translation issues due to the Russian source material; and the fact that the benchmark is centered on single-answer puzzles, capturing only one slice of creativity rather than all forms. Human validation is described as strong, but it is not a full human-performance study across the same conditions as model evaluation.
A common misconception would be to treat strong factual retrieval as sufficient evidence of creative reasoning. CresOWLve argues against that inference directly: high factual knowledge does not guarantee the ability to combine clues under non-obvious constraints. Extended “thinking” helps, but even top models remain far from a general solution to this form of creative problem-solving.
7. Nomenclature and adjacent uses of the term
In the supplied sources, the name “CresOWLve” is not used uniformly. The OWLViz paper explicitly states that “CresOWLve” is not a separate benchmark, task, or concept defined in the paper, and that the paper never introduces a term spelled “CresOWLve” (Nguyen et al., 4 Mar 2025). That clarification is relevant because OWLViz is itself an open-world visual question answering benchmark concerned with visual understanding, web exploration, and specialized tool usage, but it is conceptually distinct from CresOWLve’s focus on creative problem-solving over real-world knowledge.
A separate supplied description identifies CresOWLve as the proof-of-concept Chromium-based “memento-aware” browser presented in “A Chromium-based Memento-aware Web Browser” (Mabe, 2021). In that account, the system detects archived pages via the Memento-Datetime HTTP header, surfaces archival state in Chromium’s location bar and page-info UI, distinguishes among pure mementos, mixed archival content, and “zombie” mementos, and adds a bookmark as archive workflow supporting Internet Archive, Archive.today, and Megalodon.jp. That browser work addresses archival provenance and browser UI, not creative puzzle solving.
Within the benchmark literature proper, however, CresOWLve denotes the 2026 evaluation suite for creative problem-solving over real-world knowledge. Its substantive contribution is to operationalize a narrow but consequential question for LLM research: whether a model can retrieve relevant facts from multiple domains and then make the non-obvious creative leap required to convert those facts into a correct solution.
