General365: Evaluating Broad Reasoning in LLMs

• General365 is a benchmark for general reasoning that tests LLM capabilities using K–12 background knowledge against complex, multi-constraint problems.
• It comprises 365 seed problems and 1,095 variants across eight challenge categories, ensuring robustness against superficial reformulations.
• Empirical findings show the top model achieves only 62.8% accuracy, highlighting substantial headroom in bridging domain-specific and broad reasoning skills.

General365 is a benchmark for evaluating general reasoning in LLMs under conditions that are intentionally difficult, diverse, and largely independent of specialized expertise. It was introduced to probe whether strong performance on mathematics, physics, programming, and expert-level question answering transfers to broader reasoning settings in which the required background knowledge is restricted to approximately K–12 level, but the inferential burden remains high because of complex constraints, nested logical branches, semantic interference, latent clues, planning, and uncertainty (Liu et al., 13 Apr 2026). The benchmark contains 365 seed problems and 1,095 variant problems, spans eight challenge categories, and is designed to test whether models can reason robustly in ordinary-knowledge scenarios rather than relying on domain-specific pattern familiarity (Liu et al., 13 Apr 2026).

1. Concept and benchmark scope

General365 is framed around a distinction between domain-specific reasoning and general reasoning. In the benchmark’s formulation, domain-specific evaluations such as olympiad mathematics, physics, coding, graduate-domain question answering, or software engineering can conflate reasoning skill with exposure to specialized concepts, formulas, conventions, or corpora. General reasoning, by contrast, is defined through problems that depend less on expert knowledge and more on handling interacting constraints, compositional inference, misleading semantics, hidden rules, strategy, and uncertainty (Liu et al., 13 Apr 2026).

A central design principle is the restriction of background knowledge to K–12 level. The intent is to decouple reasoning from specialist recall, so that success reflects the ability to track logical structure rather than mastery of an expert discipline. The benchmark therefore targets problems that resemble broad “everyday” inferential challenges, including scheduling, spatial navigation, hidden-pattern discovery, strategic decision making, and reasoning under incomplete information (Liu et al., 13 Apr 2026).

The benchmark is also positioned as a response to perceived limitations in prior “general” reasoning benchmarks. General365 argues that existing resources such as BIG-Bench Hard (BBH), BIG-Bench Extra Hard (BBEH), ARC-AGI, and KOR-Bench may be too easy for modern models, too narrow in coverage, or too dependent on template-based construction. A plausible implication is that General365 is intended not merely as another hard benchmark, but as an attempt to measure a broader and less shortcut-prone notion of reasoning competence (Liu et al., 13 Apr 2026).

2. Composition and challenge taxonomy

The benchmark comprises 1,460 evaluation instances in total, formed from 365 seed problems and 1,095 variants. The seed problems are the primary handcrafted units, while the variants alter surface semantics or constraints while preserving the core reasoning skill being tested. This seed-plus-variant structure is meant to probe robustness against superficial reformulation rather than memorization of a single wording (Liu et al., 13 Apr 2026).

Among the 365 seed problems, the answer-type distribution is as follows.

Component	Count
Seed problems	365
Variant problems	1,095
Total instances	1,460
Numerical-answer seeds	238
Select-answer seeds	46
Text-answer seeds	81

General365 organizes its tasks into eight challenge categories, and each seed problem is annotated with one or more of them. The paper reports that nearly 70% of problems carry two or more challenge tags, indicating that the dataset is explicitly multi-skill rather than a collection of isolated single-ability tests (Liu et al., 13 Apr 2026).

The eight categories are:

• Complex Constraints: problems requiring simultaneous satisfaction of many interacting predicates.
• Branching & Enumeration: problems requiring systematic case analysis or exhaustive coverage of possibilities.
• Spatial & Temporal Reasoning: problems involving movement, orientation, ordering, or state evolution.
• Recursive & Backtracking: problems requiring iterative hypothesis testing, revision, or adversarial search.
• Semantic Interference: problems designed to trigger inappropriate reliance on familiar semantics or commonsense defaults.
• Implicit Information Reasoning: problems in which the solver must infer a latent pattern or hidden regularity.
• Optimal Strategy: problems requiring the best action sequence rather than merely a feasible one.
• Probability & Uncertainty: problems involving probabilistic or incomplete-information inference (Liu et al., 13 Apr 2026).

The benchmark notes that Complex Constraints is the most common challenge label, while Probability & Uncertainty is the least represented, though it still contains more than 20 seed problems. The latter is described as being comparable to category sizes in benchmarks such as BBH and BBEH (Liu et al., 13 Apr 2026).

3. Construction, expansion, and quality control

General365 was produced through a multi-stage pipeline with extensive manual review. The construction process begins with seed data construction, where problems were crowdsourced from diverse real-world domains to encourage broad initial diversity. Each seed problem was required to satisfy at least one of the eight challenge categories, to be original and non-replicable via standard web searches, and to be stored as a triplet consisting of problem description, detailed reasoning trace, and ground-truth answer. The final seed set then underwent manual inspection to verify consistency among statement, reasoning path, and answer (Liu et al., 13 Apr 2026).

A subsequent stage performed difficulty filtering and diversity augmentation. Problems that were too easy for top-tier models, or too semantically similar to existing benchmarks, were removed. Categories with insufficient coverage were manually expanded. This suggests that difficulty and diversity were treated as explicit curation targets rather than byproducts of collection (Liu et al., 13 Apr 2026).

The benchmark then underwent data post-processing to reduce ambiguity and improve evaluability. Ambiguous wording discovered through interactions with LLMs was revised, output-format instructions were appended to problem statements, and answers were normalized into numerical or canonical formats whenever possible. These steps were intended to make grading more robust and reduce evaluation noise (Liu et al., 13 Apr 2026).

For large-scale expansion, an LLM generated 10 candidate expansions for each seed. Human experts reviewed these candidates and discarded invalid or illogical ones. If a seed had fewer than three valid expansions, humans manually supplemented the set. A further difficulty and diversity audit selected the final variants. The resulting benchmark therefore combines machine-assisted expansion with human validation at multiple stages (Liu et al., 13 Apr 2026).

Quality control is a central claim of the benchmark. Reported elements include manual challenge-category annotation, manual verification of general-reasoning scope, manual cross-checking of statements and answers, filtering of easy instances, filtering of semantically similar instances, human review of generated variants, and post-processing for ambiguity reduction and output standardization (Liu et al., 13 Apr 2026).

4. Evaluation protocol and grading framework

General365 uses an empirical accuracy-based evaluation protocol. The benchmark evaluates 26 leading LLMs from major model families including OpenAI, Gemini, Anthropic, DeepSeek, Qwen, GLM, Moonshot/Kimi, Grok/xAI, and LongCat. It explicitly distinguishes reasoning models from chat/non-reasoning models, while also including strong non-reasoning baselines (Liu et al., 13 Apr 2026).

Inference settings are specified. The benchmark uses temperature $T = 1.0$ for reasoning models and temperature $T = 0.7$ for non-reasoning models, with top-p = 1.0, top-k unconstrained, maximum output length set to each model’s capacity, and the highest available setting for models with adjustable reasoning effort or test-time scaling (Liu et al., 13 Apr 2026).

Prompt templates enforce standardized answer formats. For numerical tasks, models are instructed to output:

### The final answer is: \boxed{<your answer>}

For select and text answers, the benchmark requires a strict concluding format. Scoring is then performed with a hybrid grading framework. For the 238 numerical-answer seed problems, the boxed final answer is parsed and checked for equivalence using math-verify tools; for approximation tasks, required decimal precision is manually specified. For the 46 select-answer and 81 text-answer seed problems, grading is performed by GPT-4.1 under a prompt that asks whether the student answer is equivalent to the reference answer and requires the grader to conclude with:

### Conclusion: Correct/Incorrect

The main reported metric is accuracy (%), defined as the proportion of correctly solved tasks under this hybrid grader. The benchmark does not report alternative evaluation measures such as F1, pass@k, or calibrated probability (Liu et al., 13 Apr 2026).

The paper also examines sampling stability. For several representative models, it performs four independent evaluations and reports maximum deviation $\Delta$: 1.7 for DeepSeek-V3.2-Thinking, 1.7 for Kimi-K2-Thinking, 2.0 for GLM-4.7-Thinking, 0.9 for LongCat-Flash-Thinking-2601, and 2.6 for LongCat-Flash-2512. All reported deviations are under 3%, which is presented as evidence of reproducibility despite stochastic decoding (Liu et al., 13 Apr 2026).

Grading reliability is assessed through manual auditing of 1,460 grading instances based on outputs from o4-mini, Gemini-2.5-Pro, DeepSeek-V3.1-Thinking, and LongCat-Flash-Thinking. The reported grading accuracy is 99.6%, indicating that the hybrid rule/model grading procedure is intended to be highly reliable (Liu et al., 13 Apr 2026).

5. Empirical results and diagnostic findings

The headline empirical result is that the best-performing model, Gemini-3-Pro, achieves only 62.8% accuracy on the full benchmark. Other top scores include Gemini-3-Flash: 60.8%, GLM-5-Thinking: 59.9%, GPT-5-Thinking: 58.6%, GPT-5.1-Thinking: 58.2%, Qwen3.5-397B-A17B-Thinking: 57.7%, DeepSeek-V3.2-Speciale: 57.5%, GLM-4.7-Thinking: 57.4%, and Qwen3-Max-Thinking: 57.2%. At the lower end, Gemini-2.5-Flash scores 39.6% and DeepSeek-V3.2-Chat scores 37.6% (Liu et al., 13 Apr 2026).

These results are interpreted as evidence that current LLM reasoning remains strongly domain-dependent. The benchmark contrasts these low-60s results with the near-saturated performance that frontier models often show on established mathematics and physics evaluations. This suggests that success on expert-domain reasoning does not imply equally strong competence on broader ordinary-knowledge reasoning tasks (Liu et al., 13 Apr 2026).

Several model-family patterns are reported. Reasoning models outperform chat models overall, but the gap is not absolute. The best open-source model, GLM-5-Thinking (59.9%), comes within about 3 percentage points of the top proprietary model. Some chat-style or non-reasoning systems are described as surprisingly competitive; for example, Qwen3-Max-Instruct (48.5%) and LongCat-Flash variants outperform some dedicated reasoning models such as Gemini-2.5-Flash (39.6%) (Liu et al., 13 Apr 2026).

The category breakdown for Gemini-3-Pro is:

• Complex Constraints: 65.3%
• Branching & Enumeration: 64.3%
• Spatial & Temporal: 57.6%
• Recursive & Backtracking: 66.8%
• Semantic Interference: 55.7%
• Implicit Information: 68.8%
• Optimal Strategy: 50.8%
• Probability & Uncertainty: 54.5%
• Overall: 62.8% (Liu et al., 13 Apr 2026)

Across models, Semantic Interference and Optimal Strategy emerge as the most persistent bottlenecks, with category performance tending to be roughly 10 percentage points below overall average. The benchmark also highlights a large reasoning-versus-chat gap in Implicit Information; one reported comparison is Gemini-3-Pro: 68.8% versus DeepSeek-V3.2-Chat: 29.6% (Liu et al., 13 Apr 2026).

The benchmark does not report a main-table breakdown comparing seed accuracy against variant accuracy. A plausible implication is that robustness to variants is treated primarily as a design objective and construction principle rather than as a separately quantified headline result in the reported evaluation (Liu et al., 13 Apr 2026).

6. Diversity validation, public release, and significance

General365 places unusual emphasis on validating both semantic diversity and reasoning diversity. For qualitative semantic-space analysis, the benchmark embeds task queries using text-embedding-ada-002 into 1536-dimensional vectors, projects them to 2D with t-SNE (Perplexity = 30), and normalizes coordinates to $[-1,1]$. In this visualization, General365 is described as more uniformly distributed than BBH or BBEH, which show local collapse into dense clusters (Liu et al., 13 Apr 2026).

For quantitative reasoning-similarity analysis, the benchmark retrieves each problem’s nearest neighbor in embedding space and asks Gemini-3-Pro to score logical similarity on a 0 to 5 scale, considering the problem, chain-of-thought, and final answer. The reported mean similarity is $\mu = 2.16$ for General365, compared with BBH: $\mu = 4.71$ and BBEH: $\mu = 4.80$. The paper further reports that about 68.2% of General365 instances fall in the 1–2 similarity range, whereas 77.8% of BBH and 83.5% of BBEH samples receive the maximum similarity score of 5. This is presented as evidence that General365 has substantially lower logical redundancy and is less vulnerable to template-driven shortcut exploitation (Liu et al., 13 Apr 2026).

Difficulty is also validated by comparing accuracy and generated output length to BBH and BBEH. The paper reports that top models which solve BBH with fewer than about 2k tokens may require over 13k tokens on General365, and some models exceed 18k tokens while still achieving lower accuracy. Gemini-3-Pro is noted for combining the best score (62.8%) with about 14k output tokens on average, whereas other frontier models often use 25k–30k tokens to reach comparable but usually lower performance (Liu et al., 13 Apr 2026).

The benchmark is publicly released with dataset, code, and leaderboard at https://general365.github.io/. It also includes a public subset containing 180 seed problems and their variants, for a total of 720 instances. Reported full-versus-public-subset scores include Gemini-3-Pro: 62.8% vs 61.3%, Gemini-3-Flash: 60.8% vs 59.7%, GLM-5-Thinking: 59.9% vs 59.2%, GPT-5-Thinking: 58.6% vs 56.0%, and Qwen3.5-397B-A17B-Thinking: 57.7% vs 57.9%. This is taken to indicate that the public subset is a close proxy for the full benchmark (Liu et al., 13 Apr 2026).

The benchmark also states several absences. It reports no human performance comparison, no classic contamination study beyond the seed originality criterion, no ablation study on construction steps or category removal, and no detailed seed-vs-variant accuracy table in the described results. These omissions are important for interpreting General365 as an empirical benchmark resource rather than a complete theory of general reasoning evaluation (Liu et al., 13 Apr 2026).

In aggregate, General365’s significance lies in making a specific claim testable: that contemporary LLM reasoning is still heavily domain-dependent. By combining 365 human-curated seed tasks, 1,095 variants, eight reasoning categories, strong manual quality control, hybrid grading, and explicit diversity analysis, the benchmark argues that broad ordinary-knowledge reasoning remains a distinct and unsolved capability. Its central result—that the best model reaches only 62.8% accuracy—suggests substantial remaining headroom for progress toward robust general-purpose reasoning systems (Liu et al., 13 Apr 2026).