ParaDLC-Bench: Parallel Localized Captioning Benchmark

• ParaDLC-Bench extends single-region captioning to simultaneous multi-mask evaluation, testing models' ability to maintain per-region fidelity while avoiding cross-mask interference.
• The benchmark leverages segmentation datasets and curated multiple-choice questions to assess both caption quality and inference efficiency under parallel decoding.
• Its evaluation framework utilizes LLM judges to score attribute inclusion and exclusion, offering actionable insights into localized visual perception.

Searching arXiv for the primary and contextual papers to ground the article. Parallel Detailed Localized Captioning Benchmark (ParaDLC-Bench) is a pure evaluation benchmark for parallel detailed localized captioning in which multiple region masks per image are captioned jointly. It was introduced with PerceptionDLM to test whether diffusion LLMs can exploit parallel decoding for multi-region perception while preserving region-specific fidelity. The benchmark extends single-region detailed localized captioning to concurrent multi-region evaluation, explicitly measuring both caption quality and inference efficiency, with particular emphasis on cross-region interference and attribute entanglement under multi-mask conditions (Sun et al., 17 Jun 2026).

1. Conceptual definition and task formulation

ParaDLC-Bench is motivated by the limitations of DLC-Bench, which evaluates single-region detailed localized captioning with a reference-free LLM-judge protocol focused on intra-region detail accuracy and the avoidance of hallucinated but absent attributes. ParaDLC-Bench preserves the localized, reference-free philosophy but moves from one target region to a set of target regions within the same image, so that interference between regions and the efficiency cost of describing many regions become first-class evaluation targets (Lian et al., 22 Apr 2025).

The benchmark is aligned with the task formulation used in PerceptionDLM. Given an image $I$ and a set of $N$ regions $\{R_i\}_{i=1}^N$, each represented by a binary mask, the objective is to generate corresponding textual descriptions $\{y_i\}_{i=1}^N$. In the sequential autoregressive setting, captioning is expressed as

$y_i = f(I, R_i),$

so the cost grows roughly linearly in $N$. In the parallel diffusion-language-model setting, captioning is expressed as

$\{y_i\}_{i=1}^N = f\big(I, \{R_i\}_{i=1}^N\big),$

so all captions are generated simultaneously within one diffusion process. ParaDLC-Bench is structured around this second formulation: each evaluation instance is an image with multiple masks, and the output is a set of region-wise detailed descriptions.

Conceptually, the benchmark converts the single-mask question of detailed localized captioning into a multi-mask question: whether a model can maintain per-region fidelity without interference while remaining efficient as the number of masks increases.

2. Dataset composition and construction

ParaDLC-Bench is built from segmentation datasets rather than directly from DLC-Bench data. Its sources are the Objects365 V2 validation set and the DaTaSeg Objects365 instance segmentation dataset. The released benchmark contains 100 images and 299 region masks in total (Sun et al., 17 Jun 2026).

Subset	Images	Mask instances
Objects365 V2 subset	54	178
DaTaSeg subset	46	121
Total	100	299

The benchmark is intentionally multi-instance. Annotators manually select multi-instance combinations that are spatially adjacent, semantically similar or confusing, and likely to cause feature entanglement. This design differs from DLC-Bench’s single challenging region per image: the point is not only local detail, but also the tendency of models to mix attributes across nearby or similar objects.

The mask statistics are explicitly designed to stress dense, fine-grained perception. Every image has at least 2 masks. The majority have 2–4 masks per image, and some “extremely challenging samples” contain up to 8 masks. The benchmark also retains many micro-detail targets: the average mask area ratio is 0.07, and the mask-area distribution includes many small, fine-grained regions. This design tests both localized caption quality and robustness to small-object perception in cluttered scenes.

ParaDLC-Bench is treated as a pure evaluation benchmark. It does not define train/validation/test splits; all 100 images and 299 masks are used for testing. The paper also states that strict data deduplication and isolation are enforced so that no benchmark image appears in any training sets of the evaluated models.

3. Annotation schema and question-bank design

ParaDLC-Bench does not store ground-truth captions. Instead, it stores region masks and a bank of positive and negative multiple-choice questions derived from visually grounded attributes. This continues the reference-free evaluation logic of DLC-Bench, but adapts it to multi-region settings in which the crucial errors often involve borrowing details from another concurrent mask (Sun et al., 17 Jun 2026).

The annotation pipeline has three stages. First, attribute extraction is performed with GPT-5.2 using both a cropped original image and a segmented image showing only the target instance. For positive attributes, GPT-5.2 enumerates visible parts and attributes such as color, shape, texture, material, and size, outputting tuples of the form

$$(\text{object name}, \text{part name}, \text{property name}, \text{property value}).$$

For negative attributes, the pipeline uses two categories. The first is Negative Parts, meaning typical parts of the object category that are not visible in the mask. The second is Salient Negatives / Mislocalization Targets, meaning attributes taken from other objects in the same image, especially other concurrent masks. This second category is the central multi-target innovation: it creates distractors that mimic cross-region hallucinations.

Second, expert annotators perform human curation and refinement. They cross-verify and correct GPT-5.2 outputs, add missing salient attributes, remove ambiguous or noisy attributes, enforce spatial alignment, and strengthen deceptive negative distractors. Third, GPT-5.2 converts the curated attribute lists into mutually exclusive multiple-choice questions, including questions about the presence of positive attributes and the absence of negative or mislocalized features.

The final question bank contains 2,345 high-quality multiple-choice questions. Because the benchmark stores question sets rather than reference captions, it evaluates whether a model captures the right facts and avoids the wrong ones, rather than whether it reproduces a canonical wording.

4. Scoring protocol, anti-interference design, and efficiency metrics

The evaluation protocol has three steps: generation, judge-based scoring, and aggregation. For each image, the input is the RGB image $I$ and a set of binary masks $\{R_i\}$. A model must generate one detailed description per mask,

$N$0

Parallel-capable models can produce all captions in a single pass; sequential models generate one caption per mask, one after another (Sun et al., 17 Jun 2026).

Scoring is performed by a text-only LLM judge, GPT-5.2 by default with temperature $N$1. For positive questions, the judge checks whether the caption correctly and explicitly includes a target attribute. The score is +1 for correct inclusion, 0 for omission, and a penalty for incorrect or contradictory detail. For negative questions, the judge checks whether the caption avoids hallucinating absent or mislocalized features. The score is +1 if such attributes are avoided and a penalty if they are hallucinated or borrowed from another region. To prevent empty or off-topic captions from scoring well, the negative point is awarded only if the caption correctly recognizes the target object.

ParaDLC-Bench changes the aggregation rule used in DLC-Bench. Instead of averaging global positive and negative scores over the dataset, it computes the final score by averaging the individual scores across all masks, ensuring equal weighting for each region. The benchmark therefore reports Pos (%), Neg (%), and Avg (%) as averages over masks.

A second design axis is efficiency. The benchmark reports Time (s), defined as wall-clock time for completing the entire ParaDLC-Bench under standard settings, and TPF (Tokens Per Forward), defined informally as the average number of generated tokens per forward pass. Sequential autoregressive models and baseline diffusion VLMs have TPF = 1, whereas PerceptionDLM reaches TPF = 2.9 on ParaDLC-Bench. The paper also reports TPS and latency figures: PerceptionDLM shows near-linear TPS growth with stable per-image latency of about 2.9 s as mask count increases, whereas GAR-8B has nearly constant TPS and latency that grows roughly linearly with the number of masks. Under a constant-workload setting with four masks per image, increasing the degree of parallelism improves throughput by 3.44× and reduces latency from 10.04 s to 2.92 s.

The anti-interference design is inseparable from these metrics. Every image has at least two masks; many negative questions directly probe cross-region leakage; and small masks prevent models from relying on coarse scene cues. This makes ParaDLC-Bench a benchmark not only of descriptive detail but of strict mask-level disentanglement.

5. Experimental role in PerceptionDLM and diagnostic sensitivity

In the PerceptionDLM study, ParaDLC-Bench is used to compare four groups of systems: general VLMs, autoregressive region-specific VLMs, baseline diffusion VLMs, and PerceptionDLM itself. On ParaDLC-Bench, the paper reports Pos (%), Neg (%), Avg (%), TPF, and Time (s); on DLC-Bench, it reports the single-region Pos (%), Neg (%), and Avg (%) for comparison (Sun et al., 17 Jun 2026).

The benchmark shows a specific pattern. Among diffusion VLMs, PerceptionDLM is much stronger than LLaDA-V, SDAR-VL, and Dream-VL on both ParaDLC-Bench and DLC-Bench. Against autoregressive region models, however, the comparison is mixed: on ParaDLC-Bench, DAM-3B reaches 69.2% Avg in 326 s, GAR-8B reaches 69.5% Avg in 479 s, and PerceptionDLM-8B reaches 62.4% Avg in 276 s with TPF = 2.9. On the single-region DLC-Bench, the autoregressive region models remain stronger in accuracy, and PerceptionDLM’s advantage is primarily in the multi-region setting where parallel generation becomes operative.

The benchmark also functions as a diagnostic instrument for model architecture. In the component ablation reported on ParaDLC-Bench, removing region prompting collapses performance to 1.1% Avg, indicating that the model loses spatial binding and produces nearly identical generic captions across masks. Replacing structured attention with full attention reduces Avg from 53.7% to 47.5%, consistent with increased cross-region attribute entanglement. Removing RoI-aligned feature replay reduces Avg from 53.7% to 51.3%, showing that localized visual features matter for fine-grained mask-level perception.

The benchmark is also sensitive to training-data scaling. With DAM only, the reported ParaDLC-Bench Avg is 53.7%; with DAM + COCONut, it becomes 57.7%; and with DAM + COCONut + SA-1B, it becomes 62.4%. Judge sensitivity experiments further show that replacing GPT-5.2 with Gemini-3.1-Pro or Qwen3.5-27B preserves the ranking pattern: PerceptionDLM remains the top diffusion VLM and stays competitive with autoregressive region models.

A plausible implication is that ParaDLC-Bench is not merely a leaderboard instrument. It is structured to expose whether a model’s localized captioning mechanism genuinely binds attributes to the correct mask under concurrent decoding, rather than simply producing locally plausible descriptions.

6. Position within the benchmark landscape, limitations, and prospective extensions

ParaDLC-Bench inherits the reference-free LLM-judge paradigm of DLC-Bench but extends it from single-region to multi-region evaluation, adding multi-mask images, cross-region hallucination probes, and explicit efficiency measurement under parallel versus sequential decoding (Lian et al., 22 Apr 2025). Relative to global detailed-captioning benchmarks such as CapArena, which uses pairwise caption battles and Bradley–Terry ranking for whole-image captions, ParaDLC-Bench remains localized and mask-conditioned rather than arena-based and image-global (Cheng et al., 16 Mar 2025).

In a broader landscape, MotionAtlas-Bench applies a related region-aware, question-based strategy to motion-centric videos with spatiotemporal masks and motion fact MCQs, but it is a single-target video benchmark rather than a concurrent multi-region image benchmark (Liu et al., 28 Jun 2026). Omni-Cloze, by contrast, evaluates detailed audio, visual, and audio-visual captions through cloze-style blanks with explicit Not Given handling; this suggests an alternative fact-structured evaluation regime, but not one centered on multi-mask localized captioning (Ma et al., 14 Oct 2025). These comparisons indicate that ParaDLC-Bench occupies a distinct niche: multi-region detailed localized image captioning with simultaneous attention to quality, anti-interference, and inference efficiency.

The paper does not present a standalone limitations section devoted exclusively to ParaDLC-Bench, but several constraints are explicit or directly inferable from the benchmark description. The benchmark has 100 images and 299 masks, so it is heavily curated rather than large-scale. Its visual coverage is tied to Objects365 V2 and DaTaSeg, which limits domain breadth relative to more open-ended captioning corpora. Its question sets target attributes, parts, and mislocalization, not complex multi-step reasoning. Its evaluation depends on LLM judges, which introduces judge dependence even though sensitivity analysis is reported. It also uses multiple-choice questions rather than free-form human assessment.

The released resources include code, models, and datasets, with evaluation scripts and dataset access through the PerceptionDLM repository and model collection. A plausible extension, suggested by the paper’s own discussion, is to combine ParaDLC-Bench-style multi-region localization with broader reasoning-oriented benchmarks or to expand its density and domain coverage while preserving its central anti-interference principle. Within current multimodal benchmarking, however, ParaDLC-Bench is best understood as a compact, explicitly multi-mask benchmark for testing whether localized captioning systems can describe several regions at once without collapsing into cross-region confusion, and whether they can do so with measurable gains in parallel efficiency (Sun et al., 17 Jun 2026).