---
title: 'General365: Evaluating Broad Reasoning in LLMs'
url: https://www.emergentmind.com/topics/general365
type: topic
---

# General365: Evaluating Broad Reasoning in LLMs

General365 is a benchmark for evaluating **general reasoning** in large language models under conditions that are intentionally difficult, diverse, and largely independent of specialized expertise. It was introduced to probe whether strong performance on mathematics, physics, programming, and expert-level question answering transfers to broader reasoning settings in which the required background knowledge is restricted to approximately **K–12 level**, but the inferential burden remains high because of complex constraints, nested logical branches, semantic interference, latent clues, planning, and uncertainty [2604.11778]. The benchmark contains **365 seed problems** and **1,095 variant problems**, spans **eight challenge categories**, and is designed to test whether models can reason robustly in ordinary-knowledge scenarios rather than relying on domain-specific pattern familiarity [2604.11778].

## 1. Concept and benchmark scope

General365 is framed around a distinction between **domain-specific reasoning** and **general reasoning**. In the benchmark’s formulation, domain-specific evaluations such as olympiad mathematics, physics, coding, graduate-domain question answering, or software engineering can conflate reasoning skill with exposure to specialized concepts, formulas, conventions, or corpora. General reasoning, by contrast, is defined through problems that depend less on expert knowledge and more on handling interacting constraints, compositional inference, misleading semantics, hidden rules, strategy, and uncertainty [2604.11778].

A central design principle is the restriction of background knowledge to **K–12 level**. The intent is to decouple reasoning from specialist recall, so that success reflects the ability to track logical structure rather than mastery of an expert discipline. The benchmark therefore targets problems that resemble broad “everyday” inferential challenges, including scheduling, spatial navigation, hidden-pattern discovery, strategic decision making, and reasoning under incomplete information [2604.11778].

The benchmark is also positioned as a response to perceived limitations in prior “general” reasoning benchmarks. General365 argues that existing resources such as **BIG-Bench Hard (BBH)**, **BIG-Bench Extra Hard (BBEH)**, **ARC-AGI**, and **KOR-Bench** may be too easy for modern models, too narrow in coverage, or too dependent on template-based construction. A plausible implication is that General365 is intended not merely as another hard benchmark, but as an attempt to measure a broader and less shortcut-prone notion of reasoning competence [2604.11778].

## 2. Composition and challenge taxonomy

The benchmark comprises **1,460 evaluation instances** in total, formed from **365 seed problems** and **1,095 variants**. The seed problems are the primary handcrafted units, while the variants alter surface semantics or constraints while preserving the core reasoning skill being tested. This seed-plus-variant structure is meant to probe robustness against superficial reformulation rather than memorization of a single wording [2604.11778].

Among the **365 seed problems**, the answer-type distribution is as follows.

| Component | Count |
|---|---:|
| Seed problems | 365 |
| Variant problems | 1,095 |
| Total instances | 1,460 |
| Numerical-answer seeds | 238 |
| Select-answer seeds | 46 |
| Text-answer seeds | 81 |

General365 organizes its tasks into **eight challenge categories**, and each seed problem is annotated with one or more of them. The paper reports that nearly **70%** of problems carry **two or more challenge tags**, indicating that the dataset is explicitly multi-skill rather than a collection of isolated single-ability tests [2604.11778].

The eight categories are:

- **Complex Constraints**: problems requiring simultaneous satisfaction of many interacting predicates.
- **Branching & Enumeration**: problems requiring systematic case analysis or exhaustive coverage of possibilities.
- **Spatial & Temporal Reasoning**: problems involving movement, orientation, ordering, or state evolution.
- **Recursive & Backtracking**: problems requiring iterative hypothesis testing, revision, or adversarial search.
- **Semantic Interference**: problems designed to trigger inappropriate reliance on familiar semantics or commonsense defaults.
- **Implicit Information Reasoning**: problems in which the solver must infer a latent pattern or hidden regularity.
- **Optimal Strategy**: problems requiring the best action sequence rather than merely a feasible one.
- **Probability & Uncertainty**: problems involving probabilistic or incomplete-information inference [2604.11778].

The benchmark notes that **Complex Constraints** is the most common challenge label, while **Probability & Uncertainty** is the least represented, though it still contains **more than 20 seed problems**. The latter is described as being comparable to category sizes in benchmarks such as BBH and BBEH [2604.11778].

## 3. Construction, expansion, and quality control

General365 was produced through a **multi-stage pipeline** with extensive manual review. The construction process begins with **seed data construction**, where problems were crowdsourced from diverse real-world domains to encourage broad initial diversity. Each seed problem was required to satisfy at least one of the eight challenge categories, to be **original and non-replicable via standard web searches**, and to be stored as a triplet consisting of **problem description**, **detailed reasoning trace**, and **ground-truth answer**. The final seed set then underwent manual inspection to verify consistency among statement, reasoning path, and answer [2604.11778].

A subsequent stage performed **difficulty filtering and diversity augmentation**. Problems that were too easy for top-tier models, or too semantically similar to existing benchmarks, were removed. Categories with insufficient coverage were manually expanded. This suggests that difficulty and diversity were treated as explicit curation targets rather than byproducts of collection [2604.11778].

The benchmark then underwent **data post-processing** to reduce ambiguity and improve evaluability. Ambiguous wording discovered through interactions with LLMs was revised, output-format instructions were appended to problem statements, and answers were normalized into **numerical or canonical formats** whenever possible. These steps were intended to make grading more robust and reduce evaluation noise [2604.11778].

For large-scale expansion, an LLM generated **10 candidate expansions** for each seed. Human experts reviewed these candidates and discarded invalid or illogical ones. If a seed had fewer than three valid expansions, humans manually supplemented the set. A further difficulty and diversity audit selected the final variants. The resulting benchmark therefore combines machine-assisted expansion with human validation at multiple stages [2604.11778].

Quality control is a central claim of the benchmark. Reported elements include manual challenge-category annotation, manual verification of general-reasoning scope, manual cross-checking of statements and answers, filtering of easy instances, filtering of semantically similar instances, human review of generated variants, and post-processing for ambiguity reduction and output standardization [2604.11778].

## 4. Evaluation protocol and grading framework

General365 uses an empirical accuracy-based evaluation protocol. The benchmark evaluates **26 leading LLMs** from major model families including OpenAI, Gemini, Anthropic, DeepSeek, Qwen, GLM, Moonshot/Kimi, Grok/xAI, and LongCat. It explicitly distinguishes **reasoning models** from **chat/non-reasoning models**, while also including strong non-reasoning baselines [2604.11778].

Inference settings are specified. The benchmark uses **temperature \(T = 1.0\)** for reasoning models and **temperature \(T = 0.7\)** for non-reasoning models, with **top-p = 1.0**, **top-k unconstrained**, maximum output length set to each model’s capacity, and the **highest available setting** for models with adjustable reasoning effort or test-time scaling [2604.11778].

Prompt templates enforce standardized answer formats. For numerical tasks, models are instructed to output:

```text
### The final answer is: \boxed{<your answer>}
```

For select and text answers, the benchmark requires a strict concluding format. Scoring is then performed with a **hybrid grading framework**. For the **238 numerical-answer** seed problems, the boxed final answer is parsed and checked for equivalence using **math-verify** tools; for approximation tasks, required decimal precision is manually specified. For the **46 select-answer** and **81 text-answer** seed problems, grading is performed by **GPT-4.1** under a prompt that asks whether the student answer is equivalent to the reference answer and requires the grader to conclude with:

```text
### Conclusion: Correct/Incorrect
```

The main reported metric is **accuracy (%)**, defined as the proportion of correctly solved tasks under this hybrid grader. The benchmark does not report alternative evaluation measures such as F1, pass@k, or calibrated probability [2604.11778].

The paper also examines sampling stability. For several representative models, it performs **four independent evaluations** and reports maximum deviation \(\Delta\): **1.7** for DeepSeek-V3.2-Thinking, **1.7** for Kimi-K2-Thinking, **2.0** for GLM-4.7-Thinking, **0.9** for LongCat-Flash-Thinking-2601, and **2.6** for LongCat-Flash-2512. All reported deviations are under **3%**, which is presented as evidence of reproducibility despite stochastic decoding [2604.11778].

Grading reliability is assessed through manual auditing of **1,460 grading instances** based on outputs from **o4-mini**, **Gemini-2.5-Pro**, **DeepSeek-V3.1-Thinking**, and **LongCat-Flash-Thinking**. The reported grading accuracy is **99.6%**, indicating that the hybrid rule/model grading procedure is intended to be highly reliable [2604.11778].

## 5. Empirical results and diagnostic findings

The headline empirical result is that the best-performing model, **Gemini-3-Pro**, achieves only **62.8%** accuracy on the full benchmark. Other top scores include **Gemini-3-Flash: 60.8%**, **GLM-5-Thinking: 59.9%**, **GPT-5-Thinking: 58.6%**, **GPT-5.1-Thinking: 58.2%**, **Qwen3.5-397B-A17B-Thinking: 57.7%**, **DeepSeek-V3.2-Speciale: 57.5%**, **GLM-4.7-Thinking: 57.4%**, and **Qwen3-Max-Thinking: 57.2%**. At the lower end, **Gemini-2.5-Flash** scores **39.6%** and **DeepSeek-V3.2-Chat** scores **37.6%** [2604.11778].

These results are interpreted as evidence that current LLM reasoning remains strongly **domain-dependent**. The benchmark contrasts these low-60s results with the near-saturated performance that frontier models often show on established mathematics and physics evaluations. This suggests that success on expert-domain reasoning does not imply equally strong competence on broader ordinary-knowledge reasoning tasks [2604.11778].

Several model-family patterns are reported. Reasoning models outperform chat models overall, but the gap is not absolute. The best open-source model, **GLM-5-Thinking (59.9%)**, comes within about **3 percentage points** of the top proprietary model. Some chat-style or non-reasoning systems are described as surprisingly competitive; for example, **Qwen3-Max-Instruct (48.5%)** and **LongCat-Flash** variants outperform some dedicated reasoning models such as **Gemini-2.5-Flash (39.6%)** [2604.11778].

The category breakdown for **Gemini-3-Pro** is:

- **Complex Constraints**: **65.3%**
- **Branching & Enumeration**: **64.3%**
- **Spatial & Temporal**: **57.6%**
- **Recursive & Backtracking**: **66.8%**
- **Semantic Interference**: **55.7%**
- **Implicit Information**: **68.8%**
- **Optimal Strategy**: **50.8%**
- **Probability & Uncertainty**: **54.5%**
- **Overall**: **62.8%** [2604.11778]

Across models, **Semantic Interference** and **Optimal Strategy** emerge as the most persistent bottlenecks, with category performance tending to be roughly **10 percentage points below** overall average. The benchmark also highlights a large reasoning-versus-chat gap in **Implicit Information**; one reported comparison is **Gemini-3-Pro: 68.8%** versus **DeepSeek-V3.2-Chat: 29.6%** [2604.11778].

The benchmark does not report a main-table breakdown comparing **seed** accuracy against **variant** accuracy. A plausible implication is that robustness to variants is treated primarily as a design objective and construction principle rather than as a separately quantified headline result in the reported evaluation [2604.11778].

## 6. Diversity validation, public release, and significance

General365 places unusual emphasis on validating both **semantic diversity** and **reasoning diversity**. For qualitative semantic-space analysis, the benchmark embeds task queries using **text-embedding-ada-002** into **1536-dimensional vectors**, projects them to 2D with **t-SNE (Perplexity = 30)**, and normalizes coordinates to \([-1,1]\). In this visualization, General365 is described as more uniformly distributed than BBH or BBEH, which show local collapse into dense clusters [2604.11778].

For quantitative reasoning-similarity analysis, the benchmark retrieves each problem’s nearest neighbor in embedding space and asks **Gemini-3-Pro** to score logical similarity on a **0 to 5** scale, considering the problem, chain-of-thought, and final answer. The reported mean similarity is \(\mu = 2.16\) for General365, compared with **BBH: \(\mu = 4.71\)** and **BBEH: \(\mu = 4.80\)**. The paper further reports that about **68.2%** of General365 instances fall in the **1–2** similarity range, whereas **77.8%** of BBH and **83.5%** of BBEH samples receive the maximum similarity score of **5**. This is presented as evidence that General365 has substantially lower logical redundancy and is less vulnerable to template-driven shortcut exploitation [2604.11778].

Difficulty is also validated by comparing accuracy and generated output length to BBH and BBEH. The paper reports that top models which solve BBH with fewer than about **2k tokens** may require over **13k tokens** on General365, and some models exceed **18k tokens** while still achieving lower accuracy. **Gemini-3-Pro** is noted for combining the best score (**62.8%**) with about **14k output tokens on average**, whereas other frontier models often use **25k–30k tokens** to reach comparable but usually lower performance [2604.11778].

The benchmark is publicly released with **dataset**, **code**, and **leaderboard** at **https://general365.github.io/**. It also includes a **public subset** containing **180 seed problems** and their variants, for a total of **720 instances**. Reported full-versus-public-subset scores include **Gemini-3-Pro: 62.8% vs 61.3%**, **Gemini-3-Flash: 60.8% vs 59.7%**, **GLM-5-Thinking: 59.9% vs 59.2%**, **GPT-5-Thinking: 58.6% vs 56.0%**, and **Qwen3.5-397B-A17B-Thinking: 57.7% vs 57.9%**. This is taken to indicate that the public subset is a close proxy for the full benchmark [2604.11778].

The benchmark also states several absences. It reports **no human performance comparison**, **no classic contamination study** beyond the seed originality criterion, **no ablation study** on construction steps or category removal, and **no detailed seed-vs-variant accuracy table** in the described results. These omissions are important for interpreting General365 as an empirical benchmark resource rather than a complete theory of general reasoning evaluation [2604.11778].

In aggregate, General365’s significance lies in making a specific claim testable: that contemporary LLM reasoning is still **heavily domain-dependent**. By combining **365 human-curated seed tasks**, **1,095 variants**, **eight reasoning categories**, strong manual quality control, hybrid grading, and explicit diversity analysis, the benchmark argues that broad ordinary-knowledge reasoning remains a distinct and unsolved capability. Its central result—that the best model reaches only **62.8%** accuracy—suggests substantial remaining headroom for progress toward robust general-purpose reasoning systems [2604.11778].

Source: https://www.emergentmind.com/topics/general365