KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models

Abstract: Many real-world questions appear deceptively simple yet implicitly demand two capabilities: (i) systematic coverage of a bounded knowledge universe and (ii) compositional set-based reasoning over that universe, a phenomenon we term "the tip of the iceberg." We formalize this challenge through two orthogonal dimensions: knowledge width, the cardinality of the required universe, and reasoning depth, the number of compositional set operations. We introduce KnowledgeBerg, a benchmark of 4,800 multiple-choice questions derived from 1,183 enumeration seeds spanning 10 domains and 17 languages, with universes grounded in authoritative sources to ensure reproducibility. Representative open-source LLMs demonstrate severe limitations, achieving only 5.26-36.88 F1 on universe enumeration and 16.00-44.19 accuracy on knowledge-grounded reasoning. Diagnostic analyses reveal three stages of failure: completeness, or missing knowledge; awareness, or failure to identify requirements; and application, or incorrect reasoning execution. This pattern persists across languages and model scales. Although test-time compute and retrieval augmentation yield measurable gains -- up to 4.35 and 3.78 points, respectively -- substantial gaps remain, exposing limitations in how current LLMs organize structured knowledge and execute compositional reasoning over bounded domains. The dataset is available at https://huggingface.co/datasets/2npc/KnowledgeBerg

• The paper introduces KnowledgeBerg, a benchmark designed to evaluate LLMs' ability to systematically enumerate bounded knowledge universes and execute compositional set-based reasoning.
• The paper reports low performance metrics—with Universe F1 scores from 5.26 to 36.88 and KRQ accuracy from 16.00 to 44.19—highlighting significant reasoning gaps.
• The paper demonstrates that combining explicit universe prompts with chain-of-thought yields meaningful but limited improvements across multiple languages.

KnowledgeBerg: A Benchmark for Systematic Knowledge Coverage and Compositional Reasoning in LLMs

Motivation and Problem Formalization

KnowledgeBerg addresses a deficiency in existing evaluation paradigms for LLMs: the inability to diagnose whether models can systematically enumerate bounded knowledge universes and perform compositional set-based reasoning—requirements prevalent in many high-stakes and technical domains. The authors formalize this as the “tip-of-the-iceberg phenomenon,” wherein surface-simple questions mask substantial latent requirements: complete domain enumeration and chained set operations. These are operationalized through the axes of knowledge width (cardinality of the relevant universe) and reasoning depth (steps in the compositional set-based program).

Figure 1: Illustration of the tip-of-the-iceberg phenomenon: a surface question on constellations requires enumeration, classification, aggregation, counting, and comparison across the full set of IAU-recognized constellations.

This diagnostic challenge—distinguishing simple retrieval from systematic coverage and robust composition—has previously lacked standardized resources. Existing benchmarks underemphasize such latent complexity, focusing instead on single-fact recall or linear multi-hop inference, without pressure on universe exhaustivity or compositionality.

Benchmark Construction and Iceberg Gap Metric

KnowledgeBerg is constructed over 10 domains—spanning science, health, humanities, geography, economy, time, sports, environment, transport, and law—using authoritative data sources (e.g., Gene Ontology, official institutional records, curated Wikipedia lists). The dataset comprises 4,800 multiple-choice knowledge-grounded reasoning questions (KRQs) generated from 1,183 enumeration seeds (EQs). The questions are translated into 17 languages to facilitate multilingual analysis.

Each KRQ is grounded in an implicit, fully specified universe and constructed by applying a compositional set program (e.g., filtering, aggregation, partitioning, etc.), with program length dictating reasoning depth. This design enables precise manipulation of the two core axes: knowledge width and reasoning depth.

To quantify hidden complexity, the paper introduces the Iceberg Gap (IG): the geometric mean of normalized surface simplicity, knowledge width, and reasoning depth. Benchmarks with high IG feature surface-concise questions with large, compositional latent requirements.

Figure 2: Iceberg Gap across multiple benchmarks, evidencing that KnowledgeBerg drives the highest gap between prompt simplicity and latent knowledge/reasoning requirements.

KnowledgeBerg exhibits the highest IG among evaluated benchmarks ($0.225$), reflecting its unique diagnostic utility.

Empirical Evaluation and Model Diagnostics

Open LLMs (Qwen, Llama, Mistral, Phi, Gemma families) were systematically evaluated on both enumeration and KRQ tasks. Universe F1 (set-level matching) for enumeration ranged from 5.26 to 36.88, and KRQ accuracy from 16.00 to 44.19, underscoring the consistently low completeness and limited compositional reasoning skills even for large-scale models.

A crucial finding is the observed decoupling between knowledge enumeration and compositional accuracy. Models with higher Universe F1 do not necessarily achieve higher KRQ accuracy; at the instance level, Universe F1 and KRQ accuracy are uncorrelated ($\rho=0.0023$; $\tau=0.0020$).

Figure 3: KRQ accuracy binned by enumeration quality, revealing near-zero correlation between knowledge set completeness and compositional task success.

A three-stage diagnostic framework is introduced:

1. Completeness — Does the model retrieve the necessary universe elements?
2. Awareness — Does it recognize the need for full coverage and the required structured program?
3. Application — Can it execute the compositional reasoning steps correctly?

Systematic analysis reveals that errors arise at all three levels. Accuracy degrades strongly with increasing knowledge width, reasoning depth, and especially with higher Iceberg Gap, highlighting persistent bottlenecks across all model classes.

Figure 4: KRQ accuracy decomposed by knowledge width, reasoning depth, and Iceberg Gap, for each evaluated model.

Mitigation Strategies and Intervention Effects

Prompt engineering and inference-time compute interventions were evaluated:

• Providing explicit universe knowledge (Univ): marginal gains (~$+2.8$ points)
• Forcing candidate enumeration (Enum): negligible effect (~$+1.8$ points)
• Chain-of-thought (CoT): limited effect (~$+3.1$ points)
• Combinations (Enum+CoT, Univ+CoT): Only Univ+CoT yielded substantially higher gains (up to $+13.6$ across languages), though far from perfect.

  Figure 5: Accuracy impact of prompt variants; sizable improvement only when explicit universe plus structured reasoning is combined.

Among inference strategies, self-consistency is the most effective, providing monotonic improvements as sampling increases. Self-verification (proposal-selection decoupling) is also beneficial, whereas iterative self-refinement often amplifies mistakes if initial premises are incorrect. Retrieval-augmented generation provides additional (but smaller) gains ($+2$–$4$ points), especially when retrieval quality is optimized.

Even with advanced methods, ceiling effects remain, indicating that missing knowledge is only one part of the problem—robust reasoning over explicit, structured sets is not achieved by scale or prompt engineering alone.

Cross-Lingual Robustness

Models exhibit consistent weaknesses across all 17 evaluated languages. Aggregate ranking is stable; loss in accuracy from high to low-resource languages is present but not universally severe. These results suggest that the observed limitations are inherent to LLM architectures and training rather than language-specific pretraining exposure.

Practical and Theoretical Implications

KnowledgeBerg exposes a categorical failure of LLMs to handle questions requiring both systematic, set-level coverage and compositional set-based reasoning even under idealized bounded-universe conditions. This is critical in domains such as scientific discovery, education, and legal or medical expert systems, where both exhaustivity and composition are required for safety and rigor.

The contrast between apparent progress on conventional benchmarks and consistent failures under iceberg conditions demonstrates that further advances in model pretraining, retrieval augmentation, or mere parameter scaling are insufficient. Instead, new architectures or training schemes that explicitly represent, enumerate, and manipulate bounded universes via compositional program induction are likely required.

The persistent decoupling between knowledge completeness and reasoning reliability points to fundamental limitations: current LLMs cannot reliably organize their knowledge parametrically for set-based manipulations, nor can they reliably externalize implicit requirements through prompting alone.

Future Directions

Directions prompted by this research include:

• Explicit symbolic or neuro-symbolic methods for set representation and manipulation within LLMs
• Differentiable program induction over bounded, latent universes
• Training protocols that directly supervise compositional set-based reasoning and universe enumeration
• Mechanisms for error localization and targeted correction in completeness, awareness, and application

KnowledgeBerg serves as both a discriminative benchmark and a practical diagnostic tool for future LLM architectures and reasoning methods.

Conclusion

KnowledgeBerg systematically demonstrates critical failure modes in current LLMs on tasks requiring systematic coverage and compositional reasoning over bounded universes, validated across model families and languages. The benchmark’s structure and analytic toolkit provide a principled foundation for diagnosing, isolating, and eventually overcoming these fundamental reasoning gaps (2604.17621).