OverLayScore: Overlap Difficulty Metric

• OverLayScore is a metric that quantifies layout difficulty by combining geometric overlap (IoU) with semantic similarity from CLIP embeddings.
• It computes a score by summing the products of overlapping IoU measures and cosine similarities between instance captions.
• The metric is used in OverLayBench to characterize dense overlaps, stratify datasets, and diagnose failure modes in layout-to-image systems.

Searching arXiv for the primary paper and closely related layout-to-image generation work to ground the article. arXiv search query: "OverLayBench OverLayScore layout-to-image overlap"

OverLayScore is a difficulty metric for layout-to-image generation under overlap, introduced in the benchmark paper "OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps" (Li et al., 23 Sep 2025). It does not measure image quality directly. Instead, it quantifies how hard a layout is expected to be for a layout-to-image model when multiple bounding boxes overlap and the associated instance descriptions are semantically similar. The metric is designed to capture what the paper calls the two primary challenges of dense-overlap generation: large overlapping regions and overlapping instances with minimal semantic distinction. In the paper’s formulation, OverLayScore serves four closely related functions: dataset characterization, pre-generation difficulty estimation, benchmark stratification, and diagnosis of a failure mode that existing layout-to-image benchmarks underrepresent.

1. Definition and mathematical form

The paper defines OverLayScore on a full layout with $K$ objects, where $p_k$ denotes the instance caption and $B_k$ the normalized bounding box for object $k$. The metric is

$$\mathtt{OverLayScore} = \sum_{(i, j):\;\text{IoU}(B_i, B_j) > 0} \text{IoU}(B_i, B_j) \cdot \cos \big(\langle p_i, p_j \rangle \big).$$

Here, $\text{IoU}(B_i,B_j)$ is the intersection-over-union of two boxes, and $\cos(\langle p_i,p_j\rangle)$ is the CLIP-based cosine similarity between the text embeddings of the two instance captions. Only overlapping pairs contribute, since the summation domain is $(i,j): \text{IoU}(B_i,B_j) > 0$. The paper characterizes this sum as capturing the spatial-semantic entanglement of overlapping object pairs in a layout, and states that a higher OverLayScore indicates greater expected difficulty for a layout-to-image model to generate an image that conforms to both layout and instance-level semantics (Li et al., 23 Sep 2025).

Several properties follow directly from the definition. OverLayScore is fundamentally pairwise at the contribution level, because each unordered overlapping pair contributes a term of the form $$\text{IoU}(B_i,B_j)\cdot \cos(\langle p_i,p_j\rangle)$$. It is simultaneously layout-level at the final output, because those pairwise terms are accumulated over the whole scene. The metric has no explicit normalization by the number of objects or by the number of overlapping pairs, so layouts with more overlapping interactions can naturally receive higher scores. The only threshold appearing in the formal definition is $\text{IoU}(B_i,B_j)>0$, meaning that non-overlapping pairs are ignored.

The notation $p_k$0 is slightly informal, because $p_k$1 and $p_k$2 are captions rather than vectors. The paper resolves the ambiguity in prose by specifying that the quantity is cosine similarity in CLIP embedding space. This suggests that OverLayScore should be read as a metric over localized semantic descriptions and normalized geometry, rather than as a purely geometric statistic.

2. Conceptual basis: spatial overlap and semantic overlap

The rationale for OverLayScore is that overlap difficulty is not only geometric. The paper argues that current layout-to-image systems often fail not merely when boxes overlap, but when they overlap in ways that induce ambiguity between semantically similar instances. Two layouts can therefore have similar overlap geometry while differing substantially in generation difficulty. A pair of objects with modest overlap but sharply different semantics may remain manageable, whereas highly overlapping boxes with semantically close captions are much harder because the generator must preserve separate identity for visually or conceptually similar objects in nearly the same spatial support (Li et al., 23 Sep 2025).

This is why the formula multiplies an overlap term by a semantic-similarity term rather than relying on a crude overlap count or a purely geometric aggregate. The paper explicitly argues that IoU alone cannot distinguish between highly overlapping but semantically dissimilar objects and highly overlapping, semantically similar objects. In the authors’ toy two-object analysis, image quality deteriorates as IoU increases, and at fixed IoU it deteriorates further as semantic similarity increases. OverLayScore is meant to encode exactly that interaction.

The metric is therefore best understood as a difficulty surrogate for dense-overlap conditions. It is not intended to replace post-generation quality measures such as mIoU or O-mIoU. Instead, it estimates ex ante how challenging a layout is likely to be, before an image is generated. This suggests a division of labor between metrics: OverLayScore characterizes input difficulty, while downstream metrics assess output fidelity.

3. Motivation in dense-overlap failure analysis

The paper motivates OverLayScore through both qualitative failure cases and quantitative analysis. The qualitative examples emphasize that layout-to-image models degrade when boxes overlap substantially, when the overlapped objects are visually or semantically similar, and when the generator must preserve separate identity despite occlusion. The failure modes named in the paper include object blending/fusion, spatial ambiguity, and visual distortion. The appendix expands this into a more explicit error taxonomy: incorrect object number, object fusion, object distortion, incorrect category, and bounding-box misalignment (Li et al., 23 Sep 2025).

These errors are treated as manifestations of the same underlying regime. When two or more instances compete for the same region and their descriptions are minimally distinct, a model may merge them, miscount them, deform them, misclassify them, or place them inaccurately. OverLayScore was introduced to make this regime measurable rather than anecdotal.

A second part of the motivation concerns benchmark design. The paper argues that common layout-to-image benchmarks, including COCO-derived setups, HiCo-7k, and LayoutSAM, are strongly biased toward layouts with low OverLayScore. As a consequence, they overrepresent simpler overlap configurations and underrepresent the cases where current models fail most visibly. This creates an evaluation blind spot: a method can appear strong on average while remaining weak on overlap-heavy, semantically confusable layouts. OverLayScore is intended to expose that bias and make difficulty stratification explicit.

4. Computation and practical use

In practical terms, OverLayScore requires two inputs per object: a normalized bounding box $p_k$3 and an instance caption $p_k$4. A faithful operational reading of the paper is: gather all objects in the layout, encode each caption with CLIP’s text encoder, compute IoU for each unordered pair, discard non-overlapping pairs, compute cosine similarity for the remaining caption pairs, multiply the two terms for each pair, and sum the results to obtain a scalar layout score (Li et al., 23 Sep 2025).

The paper also clarifies an important distinction between metric definition and dataset curation. OverLayScore itself includes only pairs with $p_k$5. By contrast, OverLayBench uses stricter thresholds during data filtering, retaining only images with one to ten valid overlapping bounding-box pairs where IoU is greater than $p_k$6 and intersection area exceeds $p_k$7 of total image area. These thresholds are not part of the metric; they are part of the benchmark-construction pipeline.

At dataset level, the authors aggregate per-layout OverLayScore values empirically rather than defining a new dataset-level formula. The score distributions are used to plot histograms, compare datasets, create difficulty bins, and balance benchmark splits. This is central to the paper’s use of OverLayScore: the metric is computed per layout, but its aggregate distribution determines whether a dataset adequately covers simple, regular, and complex overlap regimes.

The semantic component also introduces a dependency on annotation quality. OverLayScore assumes reliable instance captions rather than only coarse category labels. In OverLayBench, the captions are produced through Qwen-based captioning and grounding followed by human verification, precisely because noisy local descriptions would distort the CLIP-based semantic term. This makes OverLayScore simultaneously a metric and an annotation-sensitive instrument.

5. Role in OverLayBench and empirical findings

OverLayScore is the organizing abstraction behind OverLayBench. The benchmark is built through a curation pipeline that generates reference images with Flux from captions derived from real-world COCO images, uses Qwen models for image caption refinement, instance grounding, and relationship extraction, and then applies human curation for annotation quality control and difficulty balancing. After validation, the authors compute OverLayScore for each example and retain 2,052 simple, 1,000 regular, and 1,000 complex layouts. The resulting benchmark is described as featuring high-quality annotations, detailed image and dense instance captions, pairwise relationship descriptions, improved semantic grounding, and a more balanced difficulty distribution (Li et al., 23 Sep 2025).

The paper uses OverLayScore empirically in several ways. On a COCO subset with layouts containing 2 to 10 objects, the authors evaluate GLIGEN, InstanceDiffusion, and CreatiLayout, divide layouts into simple, regular, and complex categories based on OverLayScore, and sample 100 layouts from each category. The reported pattern is monotonic degradation: model performance declines as OverLayScore increases, with mIoU used in the figure caption for this analysis. This is presented as evidence that the metric reflects generation difficulty.

The same logic extends to the benchmark tables. Performance consistently drops from OverLayBench-Simple to OverLayBench-Regular to OverLayBench-Complex, especially on overlap-sensitive metrics such as O-mIoU. For example, CreatiLayout-FLUX declines from mIoU $p_k$8 and O-mIoU $p_k$9; EliGen declines from mIoU $B_k$0 and O-mIoU $B_k$1; and InstanceDiff declines from mIoU $B_k$2 and O-mIoU $B_k$3. The paper emphasizes O-mIoU because it isolates overlap regions and is therefore particularly sensitive to the phenomena that OverLayScore is meant to characterize.

OverLayScore is also central to the paper’s framing of CreatiLayout-AM. It is not part of the training loss. Rather, it identifies the regime where current models are weak and defines the difficulty splits on which the amodal-mask baseline is tested. In this framework, OverLayScore plays a diagnostic and benchmarking role, while CreatiLayout-AM is an initial model-side response to the overlap-heavy regime the score exposes. The reported O-mIoU gains of CreatiLayout-AM over CreatiLayout are $B_k$4 on Simple, $B_k$5 on Regular, and $B_k$6 on Complex, suggesting that the hardest high-score regime remains difficult (Li et al., 23 Sep 2025).

6. Interpretation, caveats, and scope

Several limitations are implied by the paper’s design choices. First, OverLayScore depends directly on caption quality. Because semantic similarity is computed from instance captions, bad local descriptions can distort the score. This is why OverLayBench emphasizes better grounding with Qwen2.5-VL-32B than GroundingDINO and includes human auditing of boxes, captions, and relationship validity (Li et al., 23 Sep 2025).

Second, the semantic term is a CLIP-based proxy for confusability rather than a perfect model of visual ambiguity. The paper uses CLIP cosine similarity because it is practical and scalable, but this does not guarantee exact alignment with the true difficulty of rendering two overlapping objects distinctly. A plausible implication is that OverLayScore is most reliable when the caption space and the intended visual distinctions are well aligned.

Third, the metric conflates severity of individual overlaps with the number of overlapping pairs, because it is an unnormalized sum. The paper does not treat this as an error; it may be desirable, since more overlapping interactions often do increase difficulty. Even so, the score does not isolate whether a layout is hard because of one extreme pair or because of many moderate interactions.

Fourth, OverLayScore is deliberately narrow in scope. Only overlapping pairs contribute, so non-overlap-based layout difficulty is ignored. This is consistent with the benchmark’s purpose: the metric is not a universal hardness estimator for all layout-to-image problems, but a targeted abstraction for dense overlaps with semantic confusability.

Finally, the paper text provided does not specify the exact numeric boundaries used to divide OverLayScore into simple, regular, and complex bins. The split concept is explicit and central, but fully reproducible binning depends on additional project materials rather than on the excerpt alone. This suggests that OverLayScore is already a useful analytical primitive at the scalar level, while some benchmark-specific operational details remain external to the formal definition.