AdamBench Biographical Evaluation
- AdamBench is a biographical reasoning benchmark derived from a large multilingual database, designed to evaluate models on factual recall and higher-order reasoning.
- It organizes evaluation using Bloom’s taxonomy, spanning tasks from simple fact recall to creative synthesis, and incorporates popularity stratification and multimodal inputs.
- Empirical findings indicate a marked performance gap between closed-source and open-source models, with retrieval augmentation significantly improving accuracy.
AdamBench is the evaluation component of the ADAM framework, introduced as a multilingual, partially multimodal benchmark for biographical reasoning by LLMs and multimodal LLMs. It is generated from AdamDB, a large biographical database, and organizes evaluation according to Bloom’s taxonomy, spanning six reasoning levels from factual recall to creation. In the reported formulation, AdamBench is a multiple-choice benchmark over biographies, designed to probe not only factual recall but also higher-order reasoning, multilingual generalization, popularity effects, and the interaction between textual and face-image inputs (Cekinmez et al., 26 Sep 2025).
1. Definition, scope, and naming
AdamBench is defined in the source paper as the benchmarking layer of ADAM, where ADAM comprises three parts: AdamDB as the underlying knowledge base, AdamBench as the evaluation suite, and AdamRAG as the retrieval-augmented system used to improve performance on the benchmark. AdamBench is therefore not a standalone corpus in isolation, but an evaluation instrument systematically generated from AdamDB for a selected set of individuals (Cekinmez et al., 26 Sep 2025).
Its stated task format is multiple-choice question answering over biographies. The benchmark is intended to measure how well models reason about people rather than merely reproduce isolated facts. The paper frames biography as a “critical yet underexplored dimension of factual knowledge,” and uses AdamBench to expose hallucination, English-centric bias, poor performance on lesser-known individuals, and limited gains from multimodal input when such input is not properly grounded (Cekinmez et al., 26 Sep 2025).
A naming ambiguity is relevant. “AdamBench” in this context refers to the biographical reasoning benchmark introduced within ADAM (Cekinmez et al., 26 Sep 2025). It is distinct from “ATOM-Bench,” which is a real-world robotics benchmark for atomic skills and compositional generalization in manipulation policies and is not presented as an alternate name for AdamBench (Wu et al., 15 Jun 2026). This suggests that references to “AdamBench” should not be conflated with the robotics benchmark.
2. Position within ADAM and benchmark construction
AdamBench is constructed from AdamDB, which the paper describes as a multilingual and multimodal repository of about 4,016,647 unique individuals. AdamDB is built from WikiDBS relational tables through a multi-stage pipeline involving heuristic identification of person-related columns, named entity recognition to retain PERSON-dominated columns, row-wise extraction of structured records, name-based merging, and subsequent revalidation with NER. Records are aligned with Wikidata Q-IDs for deduplication and standardization, and duplicates are consolidated by taking modal values for biography, nationality, birth date, and birthplace (Cekinmez et al., 26 Sep 2025).
The retained AdamDB set includes only individuals with non-null biography, birth date, nationality, and birthplace. Popularity is measured using 2024 annual page views of the person’s English Wikipedia entry, and entries with zero page views are discarded. To ensure global coverage, the construction enforces a minimum of 10 individuals per country or territory. AdamDB itself spans over 200 countries and all continents, though the appendix reports a strong continental imbalance: 52.5% Europe, 27.4% North America, 8.4% Asia, 6.0% South America, 3.9% Oceania, and 1.9% Africa (Cekinmez et al., 26 Sep 2025).
The benchmark subset is not drawn uniformly from AdamDB. Instead, individuals are grouped by country, and a base cluster count is computed as
The paper states that this guarantees each country is represented by at least one cluster. Within each country, individuals are stratified into three popularity tiers: high, medium, and low. Then -means clustering is applied independently within each tier, using a feature vector combining birth date and biography, where birth date is quantized to the nearest 50 years and biography is encoded as a normalized BERT embedding after removing explicit mentions of age or nationality. From each cluster, one individual is selected according to tier-specific rules. This yields a final benchmark person set of approximately 1,650 individuals (Cekinmez et al., 26 Sep 2025).
For each selected individual, the authors compile multilingual names and a Wikipedia summary, then supply this package to an LLM that synthesizes a concise biography and generates multilingual and multimodal questions grounded in Bloom’s taxonomy. The paper does not specify which LLM was used for this authoring stage, nor does it report a fully described human validation protocol, inter-annotator agreement, or exact train/dev/test split for AdamBench itself (Cekinmez et al., 26 Sep 2025).
3. Cognitive structure, multilinguality, and modality
AdamBench is explicitly organized around all six levels of Bloom’s taxonomy. The paper lists the levels as Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating. These are operationalized as benchmark categories intended to move beyond factual lookup: lower levels focus on recall and comprehension, whereas higher levels probe abstraction, comparison, critique, and synthesis (Cekinmez et al., 26 Sep 2025).
The benchmark is also multilingual. Questions are presented in English and in the subject’s “original language” or native language, where the operational definition of that language is based on city of birth. In the reported results tables, these conditions are denoted “En” and “Org.” The paper states that zero-shot performance shows a modest but consistent advantage for Org over English, especially in Remembering and Understanding, while this language gap largely disappears once retrieval is introduced (Cekinmez et al., 26 Sep 2025).
AdamBench is only partially multimodal. Some questions incorporate face images, and the corresponding evaluation condition is reported as “Face Image.” The paper does not claim that all questions are multimodal. Rather, multimodal evaluation tests whether a model can use a face image alongside text prompts to identify or reason about the correct person. The reported conclusion is that multimodal input yields smaller and less consistent gains than retrieval, and can even degrade performance in some cases when it is not grounded by external evidence (Cekinmez et al., 26 Sep 2025).
The benchmark also stratifies individuals by popularity. This is central to its difficulty model: accuracy rises consistently from low-popularity individuals to high-popularity individuals, which the paper interprets as evidence that pretraining exposure strongly mediates performance. Retrieval reduces, but does not eliminate, this disparity (Cekinmez et al., 26 Sep 2025).
4. Evaluation format and reporting protocol
AdamBench is evaluated primarily through accuracy on multiple-choice questions. The paper identifies accuracy as the principal metric because it directly measures whether the correct option is selected among distractors. It does not define additional benchmark metrics such as calibration error, Brier score, F1, or a separate hallucination-specific formula, even though hallucination is a major motivation for the benchmark (Cekinmez et al., 26 Sep 2025).
Automatic evaluation is conducted using EleutherAI’s lm-eval-harness and the Khayyam Challenge platform. The latter is used to support multilingual and multimodal presentation. The paper does not describe a human scoring protocol for benchmark evaluation, and it does not specify the exact number of answer options per question, exact distractor-generation methodology, or a formal benchmark schema in the main text (Cekinmez et al., 26 Sep 2025).
Several construction and reporting details are explicitly left unspecified. The paper does not state the total number of questions per person or per Bloom level, does not report balancing across Bloom levels beyond stating that all six are included, and does not define a held-out split between benchmark-construction examples and evaluation examples. It also does not quantify how many multimodal questions use crawled images versus images linked from Wikipedia (Cekinmez et al., 26 Sep 2025).
These omissions matter for interpretation. A plausible implication is that AdamBench is strongest as a structured evaluation framework for model comparison and stratified analysis, while some benchmark-engineering details remain underspecified in the paper text.
5. Empirical findings and diagnostic value
The reported experiments compare closed-source models—Gemini Flash 2.5 and GPT-4—and open-source models—Gemma3-12b-it and Qwen2.5-7b—under zero-shot, retrieval-augmented, and some image-conditioned conditions. The paper’s central empirical result is a strong gradient across Bloom levels: Remembering and Understanding are the easiest categories, while Evaluating and Creating are the hardest (Cekinmez et al., 26 Sep 2025).
The benchmark reveals a large open-versus-closed performance gap. Closed-source models are described as frequently achieving above 85–95% across many conditions, especially with retrieval, whereas open-source models are much weaker in zero-shot settings and are especially poor on less popular individuals. Among the named systems, Gemini Flash 2.5 is described as the most consistent overall, GPT-4 as slightly more balanced on higher-order reasoning, and Gemma3-12b-it as usually stronger than Qwen2.5-7b among the open-source models (Cekinmez et al., 26 Sep 2025).
Popularity effects are one of AdamBench’s strongest findings. The paper gives specific examples: GPT-4 in zero-shot rises from about 65% on low-popularity people to above 90% on highly popular ones; Gemini Flash 2.5 gains 15–20 percentage points across Remembering and Applying as popularity increases; and Qwen2.5-7b barely exceeds 20% on low-popularity individuals while reaching roughly 50–60% on highly popular ones. This is interpreted as evidence that benchmark performance is heavily mediated by prior exposure during pretraining (Cekinmez et al., 26 Sep 2025).
The multilingual findings are more nuanced. Zero-shot performance often slightly favors the subject’s original language over English, especially for closed-source models on lower Bloom levels. Once AdamRAG is enabled, the difference between English and original language mostly disappears. The paper interprets this as evidence that retrieval compensates for multilingual knowledge gaps rather than merely amplifying them (Cekinmez et al., 26 Sep 2025).
The multimodal findings are explicitly mixed. Gemini Flash 2.5 remains strong with or without image input, but GPT-4 sometimes declines on Remembering tasks when given face images without retrieval. Open-source models gain only modestly from face-image input, and those gains are inconsistent. The paper’s broader conclusion is that images are not nearly as helpful as retrieval for AdamBench (Cekinmez et al., 26 Sep 2025).
6. AdamRAG, limitations, and significance
AdamRAG is the retrieval-augmented component associated with AdamBench. For text-based disambiguation, it first attempts exact matching in AdamDB; if that is ambiguous, it uses Language-Agnostic BERT Sentence Embeddings (LaBSE) to retrieve semantically similar candidates, then filters candidates sequentially by nationality and by birth date within a -year window. Final selection is based on cosine similarity between biography embeddings and query context. For image-based retrieval, face embeddings are used to retrieve the top 100 similar entries, which are then filtered by nationality and birth date to leave up to five candidates (Cekinmez et al., 26 Sep 2025).
The impact of AdamRAG on AdamBench is substantial, especially for open-source models and lower-order reasoning. The paper states that retrieval helps most on Remembering and Understanding, pushes many closed-source scores above 95%, and produces especially large gains for open-source models. For example, Qwen2.5-7b is reported to rise from below 40% to well above 70% in many cases, while Gemma3-12b-it exceeds 80% on some Applying and Analyzing conditions with retrieval. At the same time, retrieval helps less on Evaluating and Creating, where the limiting factor is not only access to facts but also synthesis and judgment (Cekinmez et al., 26 Sep 2025).
The benchmark has several stated or implied limitations. It inherits coverage biases from Wikipedia and Wikidata; its popularity definition is based on English Wikipedia page views from 2024; the benchmark’s “native/original language” is operationalized through city of birth, which may not coincide with a person’s true native language; and the question-generation process is LLM-assisted without a reported large-scale human validation workflow. Geographic diversity is global but not balanced, and multimodal grounding remains weaker than textual retrieval (Cekinmez et al., 26 Sep 2025).
The release plan is also explicit. The paper states that the full ADAM release, including AdamDB and AdamBench, will be made public on Hugging Face, with a representative sample included in supplementary material during review. Evaluation is intended to be reproducible through lm-eval-harness and the Khayyam Challenge platform, while copyrighted images are not redistributed directly; instead, the dataset stores public URLs to original hosts (Cekinmez et al., 26 Sep 2025).
AdamBench’s significance lies less in raw benchmark size than in evaluation design. It introduces a biographical reasoning benchmark that is simultaneously cognitively stratified, multilingual, popularity-aware, and partially multimodal. Its main substantive result is that current models are strongest on famous individuals and lower-order tasks, much weaker on lesser-known individuals and higher-order reasoning, and improved far more by retrieval than by face-image input. In that sense, AdamBench functions as a diagnostic benchmark for factual grounding and reasoning depth in biography-centered evaluation (Cekinmez et al., 26 Sep 2025).
