Source Attribution Evaluation Framework

• Source Attribution Evaluation Framework is a formal system that defines and measures how generated content is linked to supporting sources using unified taxonomies.
• It organizes evaluation across dimensions such as attribution, citation, correctness, and retrieval, drawing on methodologies like AIS, CiteEval, and MAVIS.
• The framework integrates multimodal benchmarks and diverse pipeline protocols to address consistency, bias, and computational trade-offs in evidence-based text generation.

A source attribution evaluation framework is a formal apparatus for assessing whether generated content can be traced to supporting evidence or to an originating source. In evidence-based text generation, a survey of 134 papers characterizes the field as fragmented due to inconsistent terminology, isolated evaluation practices, and a lack of unified benchmarks, and organizes it through a unified taxonomy and seven key evaluation dimensions (Schreieder et al., 21 Aug 2025). In multimodal question answering, the same problem extends beyond text: MAVIS introduces the first benchmark designed to evaluate multimodal source attribution systems that understand user intent behind visual questions, retrieve multimodal evidence, and generate long-form answers with citations; its dataset contains 157K visual QA instances with fact-level citations to multimodal documents (Song et al., 15 Nov 2025).

1. Conceptual scope and formal definitions

Within the recent literature, attribution is the broadest notion: it links generated content back to one or more evidence sources. Citation is a specific form of attribution in which the model inserts explicit reference markers such as “[1]”, and quotation is a narrower form in which the model reproduces verbatim excerpts from its sources (Schreieder et al., 21 Aug 2025). The same survey further distinguishes attribution approaches by whether they are parametric or non-parametric, by how retrieval is integrated into generation, and by citation characteristics such as modality, granularity, style, visibility, and frequency (Schreieder et al., 21 Aug 2025).

A foundational formalization is AIS, “Attributable to Identified Sources.” AIS treats attribution as a property of a sentence $s$ in context $c$ relative to a provided set of source parts $P \subset K$. The framework requires that $s$ be interpretable in context, that its explicature be a stand-alone proposition, and that a generic hearer would affirm “According to $P$, $e$,” where $e$ is the explicature of $s$ at time $t$ (Rashkin et al., 2021). This formulation separates interpretability from source support and makes attribution a human-evaluable property rather than merely a retrieval score.

The literature also uses “source attribution” in a provenance sense. In the watermarking framework WASA, the task is to identify which provider $p \in \mathcal{P}$ was responsible for the segments of training data that most influenced a synthetic text $c$0; the extraction function is defined as $c$1 (Wang et al., 2023). TRACE gives a related but embedding-based definition for text provenance, with an attribution function $c$2 that maps a generated response to the subset of sources most responsible for it (Wang et al., 2024). The literature therefore suggests two broad lineages: evaluation of evidence-grounded citations in generated answers, and evaluation of source provenance for synthetic outputs.

2. Evaluation dimensions and metric families

A general framework for evidence-based generation evaluates more than supportiveness alone. The seven dimensions synthesized in the survey are Attribution, Citation, Correctness, Linguistic Quality, Preservation, Relevance, and Retrieval (Schreieder et al., 21 Aug 2025). Later frameworks specialize these dimensions in different operational settings.

Framework	Setting	Core evaluation signals
AIS (Rashkin et al., 2021)	Human evaluation of grounded NLG	Interpretability rate, AIS rate
CiteEval (Xu et al., 2 Jun 2025)	Fine-grained citation evaluation in RAG	5-point Likert citation rating conditioned on cited passages, full retrieval set, response, and query
Deep research parser (Onweller et al., 7 May 2026)	Markdown reports with inline citations	Link Works, Relevant Content, Fact Check
MAVIS (Song et al., 15 Nov 2025)	Multimodal long-form visual QA	Informativeness, groundedness, fluency
AttriBench (Berman et al., 6 Apr 2026)	Quote attribution fairness	Attribution Accuracy, Omission Suppression, Evidence-conditioned Suppression

AIS collapses evaluation to two binary judgments. Over a test set of $c$3 outputs, the interpretability rate is

$c$4

and the AIS rate is

$c$5

This design is deliberately coarse-grained and human-centered (Rashkin et al., 2021).

CiteEval replaces binary NLI-style judgment with a rating function

$c$6

which conditions not only on the cited passages $c$7 but also on the entire retrieval set $c$8, the generated response $c$9, and the user query $P \subset K$0 (Xu et al., 2 Jun 2025). It is explicitly principle-driven: evaluation against full retrieval sources, attribution of statements beyond retrieval context, and fine-grained criteria covering support, redundancy, and credibility. CiteEval also distinguishes a “Full” scenario, where all citable statements are evaluated and uncited statements are penalized, from a “Cited” scenario, where only cited statements are rated (Xu et al., 2 Jun 2025).

For long cited reports, the framework in “Cited but Not Verified” operationalizes citation quality along three increasingly rigorous dimensions. “Link Works” checks URL accessibility; “Relevant Content” measures topical alignment between a claim and the retrieved page excerpt; and “Fact Check” verifies each factual assertion in the claim against the cited content (Onweller et al., 7 May 2026). MAVIS extends metric design into multimodality by defining automatic metrics along informativeness, groundedness, and fluency, and reports strong correlation with human judgments (Song et al., 15 Nov 2025). AttriBench introduces a distinct fairness-oriented axis through omission suppression: a model may fail not by naming the wrong author, but by omitting attribution entirely (Berman et al., 6 Apr 2026).

3. Pipeline structure and evaluation protocols

Many frameworks decompose source attribution into a sequence of retrieval, generation, parsing, and verification steps. In attributed information retrieval, one framework compares three LLM-centric architectures: Generate, where the model answers without retrieval; Retrieve-then-Generate (RTG), where relevant passages are retrieved and passed to the LLM; and Generate-then-Retrieve (GTR), where the model first produces an answer and citations are added post hoc by retrieving for each generated sentence (Djeddal et al., 2024). The same work evaluates answer correctness with BLEU, ROUGE-L, and BERTScore, and citation quality with AutoAIS, NLI-based precision and recall, and citation overlap (Djeddal et al., 2024).

A different design principle appears in the Markdown AST parser for deep research agents. The pipeline canonicalizes Markdown, strips fenced code blocks, builds an abstract syntax tree, extracts citation-like nodes, deduplicates and normalizes citation URLs, segments paragraphs into sentences, and attaches citation identifiers to claims using sentence-level and backward attribution rules (Onweller et al., 7 May 2026). The framework then “closes the loop” by re-fetching the exact cited page rather than verifying the claim in isolation, so that evaluation is performed on the same URL the agent cited (Onweller et al., 7 May 2026).

Localized attribution introduces another protocol. LAQuer defines user-highlighted output spans $P \subset K$1 and seeks source spans $P \subset K$2 such that $P \subset K$3, where $P \subset K$4 denotes the decontextualized meaning of the highlighted spans in context (Hirsch et al., 1 Jun 2025). Its two-stage pipeline first decontextualizes the highlighted span into a self-contained statement and then performs query-focused attribution either by prompting an LLM to copy minimal supporting spans or by aligning output and source tokens using internal hidden states (Hirsch et al., 1 Jun 2025). This shifts evaluation from sentence-level attribution to user-directed sub-sentence verification.

Multimodal frameworks preserve the same overall structure but replace text passages with richer evidence objects. VISA retrieves document screenshots, forms a three-image candidate set, and uses Qwen2-VL to emit answer text, the evidence-document index, and a bounding box over the supporting region (Ma et al., 2024). MAVIS evaluates systems that must understand user intent behind visual questions, retrieve multimodal evidence, and generate long-form answers with citations (Song et al., 15 Nov 2025). A plausible implication is that evaluation protocols increasingly treat attribution not as a single scalar but as an end-to-end property of retrieval, citation placement, evidence localization, and factual verification.

4. Benchmarks and modality-specific instantiations

Benchmark design varies sharply with modality and task. MAVIS is a benchmark for multimodal source attribution in long-form visual question answering. Its dataset comprises 157K visual QA instances, and each answer is annotated with fact-level citations referring to multimodal documents (Song et al., 15 Nov 2025). The benchmark is explicitly intended to evaluate systems that retrieve multimodal evidence and generate cited long-form answers rather than short factual responses (Song et al., 15 Nov 2025).

VISA evaluates visual source attribution in retrieval-augmented generation through region localization on screenshots. Wiki-VISA is based on Natural Questions long answers aligned to rendered Wikipedia pages and has 87K train and 3K test examples; Paper-VISA is derived from PubLayNet and has 100K train and 2,160 test examples (Ma et al., 2024). Answer accuracy uses a relaxed exact match criterion, while box localization uses Intersection over Union,

$P \subset K$5

with a correct box defined by $P \subset K$6 (Ma et al., 2024). Zero-shot Qwen2-VL-72B yields bounding-box accuracy of 1.5 on both Wiki and Paper, while the fine-tuned 7B model reaches 54.2 on Wiki and 68.2 on Paper in the single-candidate setting (Ma et al., 2024).

Synthetic-media provenance benchmarks use a different notion of source. For AI-generated images, the resynthesis framework defines 10 core sources, 100 unique character prompts, and one-shot, few-shot, and zero-shot attribution protocols; the training-free resynthesis method attributes a test image to the model whose resynthesis is closest in CLIP Large ViT-L/14 feature space under Euclidean distance (Bongini et al., 28 Oct 2025). On the 10-class setting, resynthesis achieves approximately 0.66 accuracy and outperforms all baselines up to $P \subset K$7 shots, while fine-tuned CLIP+MLP and CLIP-LoRA overtake beyond approximately $P \subset K$8 to $P \subset K$9 (Bongini et al., 28 Oct 2025).

Video and 3D benchmarks introduce yet another evaluation vocabulary. SAGA defines five attribution levels—authenticity, generation task, model version, development team, and precise generator—and reports in-domain accuracy of 99.94% for authenticity and 94.99% for precise generator attribution using only 0.5% source-labeled data in its two-stage setting (Kundu et al., 16 Nov 2025). FAKEPCD evaluates close-world and open-world source attribution for synthetic point clouds across single-shape and multi-shape scenarios; in open-world single-shape evaluation, known-source accuracy ranges from 0.82 to 0.98 and unknown-source accuracy from 0.73 to 1.00, depending on the backbone and setting (Qu et al., 2023). These provenance-oriented benchmarks evaluate attribution as classification over generators rather than supportiveness of citations.

5. Bias, confounding, and failure modes

A central theme in recent work is that attribution quality is not exhausted by average accuracy. AttriBench is explicitly fame- and demographically-balanced and introduces quote attribution as a benchmark for representational fairness in LLMs (Berman et al., 6 Apr 2026). On the Intersectional subset, the best models under no-evidence direct prompting reach only approximately 25–27% accuracy, indirect prompting roughly halves these scores, and White male authors enjoy about 10 percentage points higher accuracy than any other intersectional group in every model with $s$0 (Berman et al., 6 Apr 2026). The framework also identifies suppression as a distinct failure mode: in no-evidence indirect settings, White male or White authors have the lowest omission suppression, while Black female, Latino, and Asian authors are 10–15 percentage points more likely to receive no author mention at all (Berman et al., 6 Apr 2026).

Bias can also be induced by metadata rather than by content. The counterfactual RAG framework defines attribution sensitivity as the average absolute change in attribution precision or recall when moving from a Vanilla setting to an Authorship-Informed setting, and attribution bias as the signed difference between the Informed and Counterfactual-Authorship-Informed settings (Abolghasemi et al., 2024). Across Mistral-7B, Meta-Llama3-8B, and GPT-4, CAB is consistently positive, indicating a bias toward documents labeled as “[Human],” and overall attribution quality can shift by 3% to 18% simply by adding or swapping authorship labels (Abolghasemi et al., 2024).

Other failure modes arise from evaluation methodology itself. ROAD shows that pixel-perturbation evaluation of attribution methods is strongly affected by information leakage through the shape of the removed pixels rather than their actual values, and introduces Noisy Linear Imputation to reduce this confounder while avoiding retraining (Rong et al., 2022). On CIFAR-10 with ResNet-18 and eight removal fractions, ROAD reduces evaluation time from approximately 3,900 seconds for ROAR to 33 seconds, or 0.85% of ROAR time (Rong et al., 2022). The threshold-free AUC-IoU framework identifies another artifact: single-threshold binarization can reverse method rankings by over 200 percentage points, whereas integrating IoU over the full threshold range removes threshold-selection bias (Aksoy, 3 Sep 2025).

Multimodal evaluation has its own bias terms. MAVIS reports that LVLMs with multimodal RAG generate more informative and fluent answers than unimodal RAG, but exhibit weaker groundedness for image documents than for text documents, and that this gap is amplified in multimodal settings (Song et al., 15 Nov 2025). The same paper states that mitigating contextual bias in interpreting image documents is a crucial direction for future research (Song et al., 15 Nov 2025). This suggests that multimodal source attribution inherits not only retrieval and citation problems but also perception-specific interpretive biases.

6. Computational trade-offs and open research directions

A recurring difficulty is that principled attribution can be computationally expensive. In document-level attribution for RAG, one framework defines the utility of a subset of retrieved documents $s$1 as the conditional log-likelihood of the target response:

$s$2

Exact Shapley attribution then averages the marginal contribution of each document over all subsets, but requires $s$3 LLM calls; with $s$4, that is 1,024 utility evaluations (Nematov et al., 6 Jul 2025). Kernel-SHAP and ContextCite reach above 90% correlation with exact Shapley using far fewer calls, while Leave-One-Out performs worst (Nematov et al., 6 Jul 2025). The same study also shows that pure utility-based attribution has systematic biases under redundancy and synergy scenarios, including position bias and under-crediting of bridge documents in multi-hop settings (Nematov et al., 6 Jul 2025).

Automated citation evaluation itself increasingly combines larger context windows with model-based judges. CiteEval-Auto uses a context attribution model and citation-rating models based on iterative chains of edits and edit-distance regression; on the metric test set, its ensemble reaches statement-level Pearson $s$5 and response-level Pearson $s$6, substantially above AutoAIS-Recall (Xu et al., 2 Jun 2025). In long-form deep research reports, however, more retrieval does not guarantee more reliable citations: across GPT-5.4 and Claude Opus 4.6, Fact Check accuracy drops by approximately 42% on average as tool calls increase from 2 to 150, even while Link Works and Relevant Content stay high (Onweller et al., 7 May 2026).

The survey literature identifies several field-wide open problems: fragmented evaluation practices, reliance on heterogeneous human judgments, underexplored parametric attribution, and limited explainable citation reasoning (Schreieder et al., 21 Aug 2025). A plausible implication is that future frameworks will need to combine multiple elements already present in separate strands of the literature: the explicature-centered discipline of AIS, the context-sensitive rating principles of CiteEval, the end-to-end loop closure of citation parsers for research agents, the user-directed localization of LAQuer, and modality-aware metrics such as the groundedness and region-localization criteria used in MAVIS and VISA.