Davidsonian Scene Graph (DSG) Score

• DSG Score is a formal evaluation metric that decomposes text prompts into a DAG of atomic propositions and applies logical entailment.
• It employs a three-step pipeline that extracts tuples, generates yes/no questions, and enforces dependency consistency to avoid hallucinations.
• The metric integrates coverage and consistency scores, validated through the DSG-1k benchmark, to reliably assess text-to-image alignment.

The Davidsonian Scene Graph (DSG) Score is a formal, empirically-grounded evaluation metric for the fine-grained assessment of text-to-image alignment in generative models. The DSG framework decomposes textual prompts into a Directed Acyclic Graph (DAG) of atomic propositions, organizes these into question-answer pairs, enforces logical entailment via dependency structure, and aggregates visual question answering (VQA) outputs into an interpretable coverage score. This methodology addresses reliability challenges in prior Question Generation/Answering (QG/A) approaches by ensuring atomicity, uniqueness, coverage, and dependency consistency. The open-source DSG-1k benchmark provides high-resolution human and model-based evaluations across diverse categories (Cho et al., 2023).

1. Formal Definition of the Davidsonian Scene Graph

At its core, the Davidsonian Scene Graph is a DAG that encapsulates the full decompositional semantics of a text prompt by mapping it to a set of atomic propositions (the nodes) and entailment dependencies (the edges).

• Atomic Propositions (nodes) fall into four well-defined types:
  • Entities: Single-item tuples, e.g., (“motorcycle”)
  • Attributes: Pairwise tuples of the form (attribute, entity), e.g., (“blue”, “motorcycle”)
  • Relationships: Triplet tuples (relation, subject, object), e.g., (“next to”, “motorcycle”, “door”)
  • Globals: Single-item scene-wide properties, e.g., (“bright lighting”)
• Dependency Edges (directed) encode entailment: Child node truth can be evaluated only if the parent node is true. For example, the truth of (“blue”, “motorcycle”) is conditioned on the existence of (“motorcycle”). Mathematically, for prompt $P$,
  • The set of atomic tuples is $T = \{t_1,\dots, t_M\}$ ($t_i$ of arity 1, 2, or 3).
  • The set of dependencies is $E \subseteq \{(t_i \to t_j)\}$, such that if $(t_i \to t_j) \in E$, $\mathrm{truth}(t_j)$ entails $\mathrm{truth}(t_i)$.

2. Question Generation Pipeline

The QG pipeline extracts and realizes the semantic structure of the prompt in three sequential LLM stages:

1. Tuple Extraction The prompt $P$ is parsed to produce a complete, non-hallucinated set of atomic tuples $\{(i, t_i)\}$, with instructions to strictly avoid fact invention.
2. Question Realization Each tuple $(i, t_i)$ is mapped to a unique, atomic yes/no question $T = \{t_1,\dots, t_M\}$0 reflecting the truth of $T = \{t_1,\dots, t_M\}$1. For example, ("blue", "motorcycle") yields “Is the motorcycle blue?” Each question targets only a single atomic fact.
3. Dependency Extraction For every tuple $T = \{t_1,\dots, t_M\}$2, the set of parent IDs $T = \{t_1,\dots, t_M\}$3 is determined, capturing logical entailment structure.

This three-step process guarantees atomicity (one fact per question), uniqueness (no duplications), hallucination-freeness (constrained to prompt content), and comprehensive semantic coverage (all ground-truth tuples enumerated). The entire method is codified by:

$E \subseteq \{(t_i \to t_j)\}$0

3. Visual Question Answering and Consistency Enforcement

Given the set of DSG questions $T = \{t_1,\dots, t_M\}$4 and a rendered image $T = \{t_1,\dots, t_M\}$5 (from any text-to-image model), each $T = \{t_1,\dots, t_M\}$6 pair is answered via a pretrained VQA model (e.g., PaLI, mPLUG-large, Instruct-BLIP), yielding $T = \{t_1,\dots, t_M\}$7.

To maintain logical strictness, dependency consistency is enforced: For any child question $T = \{t_1,\dots, t_M\}$8 whose parent $T = \{t_1,\dots, t_M\}$9 receives a "No," $t_i$0 is forcibly answered "No," preempting invalid queries (e.g., questioning "Is the motorcycle blue?" when the motorcycle is absent).

Enforcement algorithm:

$E \subseteq \{(t_i \to t_j)\}$1

4. Mathematical Formulation of the DSG Score

The DSG Score is mathematically defined as a composite of coverage and consistency components:

• Coverage Score: Measures agreement of VQA answers $t_i$1 with ground-truth $t_i$2 over all questions $t_i$3:

$t_i$4

• Consistency Score: Quantifies dependency violations:

$t_i$5

i.e., one minus the fraction of entailment violations.

• Composite DSG Score:

$t_i$6

with $t_i$7 balancing strict coverage vs. dependency consistency. Empirically, $t_i$8 was used due to the hard enforcement of dependency consistency.

Algorithmic summary:

$E \subseteq \{(t_i \to t_j)\}$2

5. DSG-1k Benchmark: Structure and Evaluation

DSG-1k is an open-sourced benchmark assembled to guarantee broad and challenging semantic coverage and to rigorously validate DSG scoring reliability. It consists of 1,060 human-written prompts, sampled uniformly from ten diverse sources, including TIFA-160, Stanford Paragraphs, Localized Narratives, CountBench, VRD relations, DiffusionDB, Midjourney, PoseScript, Whoops! commonsense-defying examples, and DrawText.

Distribution of question types:

Category	Approximate Count	Examples
Entity questions	~3,400	(“motorcycle”)
Attribute questions	~2,250	(“blue”, “motorcycle”)
Relation questions	~2,080	(“next to”, ...)
Global questions	~530	(“bright lighting”)

Evaluation on DSG-1k comprised both VQA model outputs and human answers for each question over three state-of-the-art generators (Stable Diffusion v2.1, Imagen*, MUSE*), measuring:

• Per-question VQA vs. human matching accuracy (73.8% for PaLI)
• Per-item Spearman correlation with human 1–5 Likert ratings ($t_i$9)
• Detailed error analysis by semantic category (counting, text-rendering remain hardest)

These results confirm that DSG-1k prompts and corresponding DSG questions are atomic, unique, and hallucination-free, and that the DSGScore provides reliable fine-grained text-to-image alignment measurement.

6. Reliability and Significance of DSG Scoring

The DSG framework addresses prior QG/A shortcomings by eliminating hallucinations, duplications, and omissions at tuple extraction; enforcing atomicity and uniqueness of questions; and maintaining logical entailment consistency. Integrated human and model-based assessments in DSG-1k empirically validate both coverage and reliability across a wide semantic range. This suggests DSGScore can serve as a standard for evaluating prompt-image faithfulness in generative modeling pipelines when fine-grained metric rigor is required (Cho et al., 2023).