---
title: Ambiguous Biographies Dataset
url: https://www.emergentmind.com/topics/ambiguous-biographies-dataset
type: topic
---

# Ambiguous Biographies Dataset

The Ambiguous Biographies Dataset refers to a class of evaluation corpora designed to probe and measure the ability of computational and information retrieval systems to reason about or disambiguate multiple distinct real-world entities sharing the same surface name across documents. Two prominent instantiations in the literature are the “Ambiguous Biographies” corpus introduced for entity-oriented web search evaluation and the AmbigDocs benchmark for multi-entity question answering over ambiguous names. Both are constructed via controlled extraction and annotation of web or Wikipedia content for names with multiple associated referents, drawing ground-truth from human annotation and leveraging Wikipedia disambiguation pages.

## 1. Motivation and Core Use Cases

The central challenge motivating ambiguous biographies datasets is the prevalence of homonymous entity mentions in web search, information retrieval, and natural language question answering. In settings where queries or questions refer to an ambiguous name (e.g., “Michael Jordan”), the system must identify—from among the set of potentially relevant documents—which mention or characterize which real-world entity, cluster results accordingly, and (in QA) associate extracted facts with the correct referent. State-of-the-art search engines and language modeling systems routinely conflate entities, favor popular referents, or merge conflicting facts unless explicitly evaluated and trained for this capability [1312.1897, 2404.12447].

## 2. Dataset Construction Methodologies

### Ambiguous Biographies (Web Corpus)

- **Source Selection:** All English Wikipedia Human Name Disambiguation Pages (HDPs) as of July 2012, totaling ≈38,692 HDPs covering ≈213,564 person articles, were extracted. HDPs with fewer than two links were discarded.
- **Query Sampling:** A stratified-random sample of 50 HDPs was drawn to balance frequency: “head” (≥24 links), “mid-frequency” (10–23 links), and “tail” (<10 links) names.
- **Web-page Retrieval:** For each ambiguous name $x$, the top-100 Google search results (excluding Wikipedia) were retrieved, filtering out non-HTML, empty, or error pages.
- **Manual Annotation:** Each of ≈4,683 URLs was labeled by trained annotators as describing a single Wikipedia person entity ($e \in E_x$) or assigned to the special “#noise” class for pages about other persons, multiple persons, or no person in $E_x$. Over 85,000 candidate page-entity pairs were evaluated [1312.1897].

### AmbigDocs (Wikipedia QA Benchmark)

- **Surface Name Extraction:** All Wikipedia disambiguation pages from December 2018 were crawled. Each page defines a surface name (sn) and enumerates up to 10 entities (DE₁…DEₖ).
- **Document Preparation:** Wikipedia articles were segmented into 100-word passages assigned to their respective entities. Surface names yielding more than 10 entities or fewer than 20 passages per entity were excluded.
- **Confusion Pair Generation:** For question synthesis, entity-pair passages with “confusing” (3 < shared n-gram < 10 tokens) overlap were prioritized, totaling 67,294 candidate seed pairs.
- **QA Synthesis:** LLM prompts resulted in ambiguous-name-containing questions and distinct entity-answers. Output was filtered for ambiguity, answer overlap, and NLI-consistency using a T5-11B NLI model, yielding a final set of 36,098 multi-answer QA instances [2404.12447].

## 3. Dataset Structure, Feature Schema, and Labeling

### Ambiguous Biographies Corpus

- **Instances:** 50 ambiguous person name queries, 4,683 web pages (per-query range 74–100).
- **Fields:**
   - URL
   - Raw text (HTML-stripped)
   - Unigram set $F_d$ with term frequencies $tf(f,d)$
   - Document length $|d| = \sum_f tf(f,d)$
   - Feature document frequencies $df(f)$
- **Labels:** Each page $d$ is mapped to a Wikipedia entity $e$ or “#noise” (≈75% of all pages due to coverage bias).
- **Feature Complexity:** Only unigram lexical features were derived; no semantic metadata, titles, or structured snippets are indexed.

### AmbigDocs

- **Instances:** 36,098 questions, 102,624 distinct entities.
- **Per-instance structure:** Surface name, up to 10 Wikipedia entities with 100-word passages, question containing sn, and 2–10 gold answers (average 2.92).
- **Label Ontology:** Each model prediction is scored as “Complete,” “Partial,” “Ambiguous,” “Merged,” or “No answer,” based on answer-entity association completeness.

| Dataset        | #Queries/#QAs | Entities covered | Annotation        | Feature set      |
|----------------|--------------|------------------|-------------------|------------------|
| Ambig Biog [1312.1897] | 50 / 4,683  | Wikipedia persons | Manual, per-URL   | Unigrams         |
| AmbigDocs [2404.12447]  | 36,098       | 102,624           | LLM+human, QA pairs| 100-word passages|

## 4. Evaluation Metrics and Scoring Frameworks

### Ambiguous Biographies Clustering Evaluation

System output is a clustering $\{\omega_1, ..., \omega_K\}$ of web pages. Let $N$ denote total items, $C = \{c_1, ..., c_M\}$ be the gold-standard entities plus noise.

- **Micro Precision/Recall:**
  - $P_\mu = \frac{\sum_tp}{\sum_tp+\sum_fp}$
  - $R_\mu = \frac{\sum_tp}{\sum_tp+\sum_fn}$
  - $F_{1\mu} = 2\cdot P_\mu \cdot R_\mu / (P_\mu+R_\mu)$
- **Macro F1:** $F_{1\text{macro}} = \frac{1}{M}\sum_{j=1}^M F_{1j}$
- **Purity:** $\text{Purity}(\Omega,C) = \frac{1}{N} \sum_k \max_j |\omega_k \cap c_j|$
- **Normalized Mutual Information (NMI):**
  $$
  \text{NMI}(\Omega,C) = \frac{I(\Omega;C)}{(H(\Omega)+H(C))/2}
  $$

### AmbigDocs Answer Scoring

Given $m$ answers $\{(DE_i^*,a_i^*)\}$ and a model prediction $y$:
- **Token recall:** $R(s,y) = \frac{|T_s \cap T_y|}{|T_s|}$
- **Answer Recall:** $(1/m) \sum_i R(a_i^*, y)$
- **Entity-Answer Recall (EAR):** $(1/m) \sum_i R(DE_i^*, y) \cdot R(a_i^*, y)$
- **Disambig-F1 (DF1):** For each $i$, a RoBERTa-SQuAD2 model extracts a short answer $\hat{y}_i$ from $y$ and $DF1(y) = (1/m)\sum_i F1(\hat{y}_i, a_i^*)$
- **K-Precision:** Proportion of tokens in $y$ found in context.

Predictions are automatically classified by answer-type (Complete, Partial, Ambiguous, Merged, No Answer) using a threshold-based heuristic, achieving high agreement with human annotation ($\kappa \approx 0.83$) [2404.12447].

## 5. Empirical Findings, System Performance, and Limitations

### Baseline System Results

- In clustering (Ambiguous Biographies), clusters defined solely by unigrams perform moderately, limited by the lack of semantic or relational features [1312.1897].
- In QA (AmbigDocs), top LLMs (GPT-4, Mistral-7B) reached EAR ≈0.47 and DF1 ≈0.29 (gold passages only). No system exceeded 0.5 EAR, reflecting consistent challenges in attaching all answer spans to the correct entity.
- High K-Precision values (≥0.84) indicate models copy input text but frequently misassociate facts to entities.
- GPT-4 achieved only 38.9% “Complete” answer rates; partial, ambiguous, and merged output types are common.
- Few-shot in-context prompting substantially lifts metrics (e.g., GPT-4’s EAR +0.174 to 0.647).

### Limitations

- **Coverage bias:** Wikipedia disambiguation pages cover mostly notable entities; ≈75% of retrieved web pages in the Ambiguous Biographies dataset could not be mapped to a Wikipedia person and were labeled “noise.”
- **Manual annotation:** Resource-intensive and no formal inter-annotator agreement was reported in the original Ambiguous Biographies; cross-review substituted.
- **Feature expressivity:** Both datasets are limited by simple lexical features (unigrams or 100-word context); improvements may require more structured or relational signals (named entities, graph structure, etc.).
- **Noise:** Both corpora suffer from query drift, non-biographical content, and ambiguous labeling granularity.

## 6. Research Impact and Future Directions

Ambiguous biographies datasets operationalize multi-entity disambiguation evaluation at both the document and answer levels. They have established baselines for clustering, retrieval, and LLM-based reasoning over ambiguous names and motivated progress in:

- Multi-document, multi-entity reasoning for retrieval and QA
- Synthetic data generation filtered for annotator/LLM/entailment consistency
- Taxonomy of common LLM errors (partial, merged, ambiguous, and no-answer responses)
- Heuristic and statistical answer typing with benchmarks for recall and faithfulness [2404.12447]

The literature suggests future work should expand query and entity scope beyond notable individuals, introduce additional web and social profiles for richer bootstrapping, incorporate dual labeling for reliability, and extend feature schemas to multi-word, semantic, and link-graph derived signals [1312.1897, 2404.12447].

Source: https://www.emergentmind.com/topics/ambiguous-biographies-dataset