Layout-HF100k: Human Feedback Dataset

• Layout-HF100k is a large-scale human feedback dataset consisting of 100k layouts with binary 'qualified' or 'unqualified' labels reflecting human design judgments.
• It supports training multimodal evaluators and aligning layout generators using preference-based reinforcement learning, notably with the DMPO method.
• The dataset spans diverse design tasks through a unified taxonomy, covering various canvas sizes, element types, and semantic constraints.

Layout-HF100k is a large-scale human feedback dataset introduced in the context of the Uni-Layout framework for layout generation and evaluation. It comprises 100,000 expertly annotated layouts and is designed to address a specific deficiency in prior layout research: existing automatic metrics such as overlap, alignment, and IoU often diverge from human judgments, so layouts with better numeric scores can look worse to people. The dataset provides binary supervision in the form of “qualified” versus “unqualified” labels and is intended to support three tightly coupled functions: training a multimodal evaluator that mimics human reasoning, benchmarking layout quality, and aligning a unified layout generator to human preferences through preference-based reinforcement learning, specifically Dynamic-Margin Preference Optimization (DMPO) (Lu et al., 4 Aug 2025).

1. Motivation and problem setting

Layout-HF100k was created to fill the absence of a large-scale human feedback dataset for layout evaluation. The underlying premise is explicit: without such data, it is impossible to train an evaluator that truly reflects human perception. In the formulation associated with Uni-Layout, the problem is not limited to generation quality alone; it also concerns the mismatch between human-perceptual assessment and the metrics commonly used for automatic scoring.

The dataset is therefore positioned at the interface between generation and evaluation. Its labels enable a human-mimicking evaluator trained with dual-branch vision and geometric inputs plus a Chain-of-Thought mechanism. The same data also support preference-based optimization of a unified generator. This arrangement makes Layout-HF100k a core component of a closed-loop system in which generation, evaluation, and alignment are explicitly coupled.

A common misconception in layout research is that geometric metrics are sufficient proxies for perceptual quality. Layout-HF100k was constructed precisely because the cited work reports that overlap, alignment, and IoU can rank visually inferior layouts above layouts that people prefer. In that sense, the dataset operationalizes human acceptability directly rather than assuming that low-level geometric heuristics are adequate substitutes.

2. Taxonomy and task coverage

Layout-HF100k follows the Unified Layout Taxonomy, a $2 \times 2$ grid that organizes layout tasks according to whether the background and the element contents are free or constrained. The four task types are:

Task type	Description	Source
BFEF	Background Free / Element Free	PubLayNet
BCEF	Background Constrained / Element Free	CGL-Dataset
BFEC	Background Free / Element Constrained	EP-Layout
BCEC	Background Constrained / Element Constrained	CGL-Dataset V2

This taxonomy supports universal generation by treating heterogeneous layout problems within one scheme. Samples for BFEF are drawn from PubLayNet and correspond to document layouts. BCEF samples come from CGL-Dataset and represent poster design with a given background. BFEC uses newly collected EP-Layout and covers e-commerce posters on a blank canvas. BCEC uses CGL-Dataset V2 and corresponds to posters with both image and selling-point text.

The task coverage is structurally important because the dataset is not restricted to a single design regime. Canvas sizes vary, with examples including $513 \times 750$ for e-commerce posters and A4 for documents. Element semantics also differ by domain: posters contain four element types—logo, text, underlay, and embellishment—whereas documents contain five types—text, title, list, table, and figure. This suggests that the dataset was designed to span both semantic and geometric heterogeneity rather than a narrow benchmark slice.

3. Annotation protocol and qualification criteria

The annotation process uses professional graphic designers with at least five years’ experience. Labels are assigned through a three-stage workflow.

First, in primary annotation, each layout is shown together with its background, if any, its element contents, if any, and its predicted bounding-box layout consisting of type plus $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$. Annotators then assign either “qualified” or “unqualified” according to detailed, task-specific rubrics.

Second, a quality inspection stage is performed by a second team, which verifies each label against the original data and guidelines.

Third, a statistical sampling audit re-checks at least $10\%$ of each batch using senior auditors, and overall labeling accuracy must be at least $98\%$.

The criteria for “unqualified” labels are concrete and task dependent. Examples include disorder, defined as no clear grouping or spacing such as randomly scattered elements; misaligned layouts, defined as missing intra-group or inter-group alignment or overall perceptual imbalance; empty layouts, defined as large blank regions or drastically uneven element density; and overlap, defined as any two boxes overlapping unless an underlay box is intended. For BCEF and BCEC, an invalid underlay is one in which an underlay box contains no nested element. For BCEC, extreme-sized elements include cases where a box covers more than one third of the canvas or becomes too small, specifically area less than 1,000 pixels or height less than 30 pixels.

The paper provides illustrative annotations. One qualified BCEF example uses a $1024 \times 1024$ canvas with a beach photo background, a logo box at $[30,30,200,80]$, title text at $[300,50,900,150]$, and an underlay at $[280,40,920,160]$, with the annotator note: “Underlay neatly frames title; no overlaps; balanced spacing.” One unqualified BFEC example uses a blank white $800 \times 600$ canvas with a product image at $513 \times 750$0 and a text box at $513 \times 750$1, labeled unqualified with the note: “Text box overlaps critical product region; misaligned and unreadable.”

4. Dataset composition and schema

Layout-HF100k contains 100,000 layouts, partitioned into 96,000 training samples and 4,000 test samples. The test split contains 1,000 samples per task, each with roughly a $513 \times 750$2 ratio of qualified to unqualified layouts.

The training distribution is as follows:

Split component	Count
BCEF	31,000
BFEF	19,000
BFEC	19,000
BCEC	27,000
Train total	96,000
Test total	4,000

The dataset is distributed in JSON format, with one file per split containing an array of samples. Each sample includes the following fields:

• id: unique sample identifier
• task_type: one of $513 \times 750$3
• canvas_size: $513 \times 750$4
• background_image: file path or URL, which may be null for BFEF and BFEC
• element_contents: array of image paths or text strings, empty for BCEF and BFEF
• layout: a list of elements, each with:
  • type: for example “text”, “logo”, or “underlay”
  • bbox: $513 \times 750$5
• label: $513 \times 750$6 for qualified and $513 \times 750$7 for unqualified
• confidence_estimate: optional placeholder for evaluator output during the reinforcement-learning stage

The access API is described at the implementation level as Python code in the GitHub repository that loads JSON, fetches images, and converts bounding boxes to tensors for model input. Figure 1 in the paper presents visual examples of good and bad layouts, with red dashed lines marking defects.

5. Human feedback signal, evaluator design, and DMPO integration

Layout-HF100k provides binary label supervision, but the associated framework derives richer optimization signals from those labels. The evaluator includes a confidence head that operates on the final hidden token $513 \times 750$8:

$513 \times 750$9

Here, $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$0 is the probability of “qualified,” and this quantity serves as the Layout Reward (LR) (Lu et al., 4 Aug 2025).

The evaluator is described as human-mimicking because it integrates visual and geometric information and uses a Chain-of-Thought mechanism to conduct qualitative assessments together with a confidence estimation module to yield quantitative measurements. In the intended usage, CoT reasoning is trained with auxiliary GPT-4o plus DeepSeek-R1 generated rationale and final human conclusion. The implementation guidance specifies dual-branch preprocessing: a colored overlay for the visualization branch and normalized bounding boxes for the geometry branch.

For preference-based reinforcement learning, the framework compares a preferred layout $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$1 and a dispreferred layout $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$2. The pairwise preference margin is defined as

$[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$3

and then amplified nonlinearly by

$[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$4

The DMPO objective fine-tunes the generator $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$5 against a frozen reference $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$6:

$[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$7

During fine-tuning, the generator proposes two layouts per prompt, the evaluator selects the winner $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$8, computes $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$9, and $10\%$0 updates the generator via LoRA. The reported outcome is that the resulting layouts better match human preferences, specifically showing higher Human Pass Rate and LR.

For evaluator benchmarking, the accuracy metric is

$10\%$1

The intended use is therefore not merely classification. The same signal can function as a pass/fail target, a scalar reward, and a pairwise margin for alignment.

6. Intended uses and research significance

The dataset is intended for three primary uses. First, it supports training the human-mimicking evaluator by supervising the classification head on binary labels. Second, it supports benchmarking layout generators by using the frozen evaluator to score held-out Layout-HF100k test layouts and report accuracy and LR. Third, it supports RLHF-style alignment via DMPO.

These intended uses are tightly connected. Layout-HF100k is not simply an archive of accepted and rejected layouts; it is structured so that the same annotation substrate can serve evaluation and optimization. A plausible implication is that the dataset functions as an interface layer between discrete human judgments and continuous training objectives.

Another common misunderstanding is to treat the dataset as if it were exclusively for post hoc scoring. The design described in Uni-Layout is broader: it explicitly uses the evaluator’s softmax output $10\%$2 as a reward signal and as an ingredient in the pairwise preference margin used for preference optimization. The integration tips make this operational: load the dataset via the provided JSON loader, preprocess images into dual-branch form, and plug the evaluator’s softmax $10\%$3 into the reward signal or pairwise margin $10\%$4.

Because the supervision is binary—“qualified” versus “unqualified”—the dataset encodes a thresholded notion of human acceptability rather than an explicit fine-grained ranking scale. This suggests that its primary strength lies in acceptability modeling and preference alignment. It also explains why the associated framework augments the binary labels with confidence estimation and pairwise comparison machinery.

In the Uni-Layout formulation, Layout-HF100k is therefore both a dataset and a mechanism for re-centering layout research around human judgment. By combining expert annotation, task coverage across the four taxonomy cells, a clear JSON schema, and compatibility with evaluator training and DMPO-based alignment, it provides the human feedback substrate for unified generation and evaluation.