FIRM-Gen-293K: Checklist-Supervised Critic Dataset

• FIRM-Gen-293K is a dataset that uses a plan-then-score pipeline to decompose complex prompts and reduce hallucination in text-to-image RL systems.
• It is compiled from diverse high-quality sources and multiple generation models to ensure robust, generator-agnostic critic training.
• The dataset integrates within the FIRM framework, enhancing RL outcomes with Quality-Modulated Alignment by improving instruction-following and visual fidelity.

FIRM-Gen-293K is a 292,792-sample, checklist-supervised dataset for training faithful text-to-image generation critics, introduced within the FIRM (Faithful Image Reward Modeling) framework in "Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation" (Zhao et al., 12 Mar 2026). Its purpose is to reduce hallucination and noisy scoring in reward models used for reinforcement learning (RL) over text-to-image systems, particularly on compositional prompts containing multiple entities, attributes, spatial relations, and style constraints. Within FIRM, the dataset supervises FIRM-Gen-8B, supplies the Instruction Following component used in Quality-Modulated Alignment (QMA), and is explicitly separated from evaluation, which is conducted on the human-annotated FIRM-Bench-Gen benchmark (Zhao et al., 12 Mar 2026).

1. Position within the FIRM framework

FIRM-Gen-293K addresses a specific failure mode in RL for text-to-image generation: the critic, or reward model, often hallucinates, misses fine-grained constraints, and produces noisy, hard-to-trust scores. According to the paper, these problems are especially acute for prompts that are compositionally complex, where faithful assessment requires simultaneous tracking of subjects, attributes, relations, style, and negative constraints (Zhao et al., 12 Mar 2026).

Within the broader FIRM framework, FIRM-Gen-293K is the generation-side scoring dataset. It is used to supervise FIRM-Gen-8B through supervised fine-tuning starting from Qwen3-VL-8B-Instruct. It also provides the critic signal that becomes the Instruction Following term in the generation reward-shaping scheme, QMA. By design, it is not the evaluation set for critic reliability; evaluation is instead performed on the disjoint and fully human-annotated FIRM-Bench-Gen, whose size is reported as $n=250$.

This separation between training and evaluation is central to the dataset’s role. FIRM-Gen-293K is a training resource for critic construction, whereas FIRM-Bench-Gen measures critic-human alignment through Mean Absolute Error (MAE). A plausible implication is that the authors intended to prevent circular evaluation of the critic on its own synthetic supervision source.

2. Plan-then-score annotation pipeline

The defining methodological feature of FIRM-Gen-293K is a tailored “plan-then-score” pipeline for generation. The pipeline has two stages and is explicitly designed to reduce critic hallucination by forcing structured verification of prompt requirements (Zhao et al., 12 Mar 2026).

In Stage 1: Explicit Criteria Planning, a text-only LLM acts as planner. Given a user prompt, Qwen3-32B produces a structured “Analysis Plan,” effectively a checklist that decomposes the prompt into verifiable criteria. The paper states that this plan typically covers main subjects or elements, attributes, actions, and relationships, style and composition, and optional negative constraints such as forbidden content.

In Stage 2: Structured Analytical Scoring, a vision-language MLLM acts as evaluator. Qwen3-VL-235B-A22B receives the generated image, the original prompt, and the Analysis Plan, then performs step-by-step verification against the checklist and outputs a final Instruction Following score on a 1–5 scale together with textual reasoning. The paper characterizes this as a mechanism that “forces attention to each requirement,” thereby mitigating hallucinations and “attention dilution.”

The criterion being labeled is instruction following, operationalized as a scalar Likert score from 1 to 5, where 5 denotes perfect alignment and 1 denotes irrelevance. The labels consist of these scalar scores plus rationale text. No pairwise preferences are reported.

The paper attributes several quality-control effects to this pipeline. Structural decomposition ensures explicit checking of requirements before score aggregation. Generator diversity reduces overfitting to the artifacts of any single text-to-image system. Human verification for FIRM-Gen-293K itself, however, is not reported, and inter-annotator agreement is likewise not reported because the annotations are produced by a planner–evaluator LLM/MLLM pipeline rather than multiple human annotators.

3. Data sources, composition, and schema

The prompts in FIRM-Gen-293K are sampled from three sources identified as high-quality and diverse: OpenGPT-4o-Image, ShareGPT-4o-Image, and BLIP3o-60k (Zhao et al., 12 Mar 2026). To generate images for scoring, the framework uses five different image generation models: Ovis-Image, Z-Image-turbo, Flux.1-dev, SDXL, and SD1.5. The paper states that this model pool is intended to diversify style and quality and help the critic learn generator-agnostic signals. Image resolution and generation details beyond model names are not specified.

The dataset schema is only partially enumerated. Each entry includes at least a prompt, a generated image, planner output in the form of Analysis Plan text, and evaluator output consisting of step-by-step analysis plus the final Instruction Following score. The paper also notes that metadata such as generator identity is likely present, but does not explicitly enumerate fields beyond prompt, plan, image, score, and reasoning. A train/validation/test split is not specified.

The reported score distribution is as follows:

Instruction Following score	Count
5	116,333
4	41,833
3	56,157
2	59,010
1	19,459

The paper does not provide finer-grained distributional statistics by prompt category, objects, actions, styles, or aesthetics.

An illustrative entry structure is included in the paper. It contains a prompt such as “A watercolor painting of two golden retriever puppies playing on a beach at sunset, soft pastel palette, gentle waves, no people or text,” a generator model such as “SDXL,” an image path or URI, a plan text that enumerates criteria for subjects, scene, style, and negatives, evaluator reasoning, and an Instruction Following score of 4. The paper labels this structure illustrative but consistent with the pipeline.

4. Reward modeling and the connection to QMA

FIRM-Gen-293K is used to train FIRM-Gen-8B, an 8B-parameter multimodal critic built from Qwen3-VL-8B-Instruct (Zhao et al., 12 Mar 2026). The critic takes image and text inputs, specifically the original prompt and Analysis Plan, and outputs reasoning together with a scalar Instruction Following score from 1 to 5. The paper does not specify multimodal fusion details, head design, or exact parameterization beyond the Qwen3-VL-8B base.

Training is described as supervised fine-tuning conducted with LLaMA-Factory on $8 \times$ H200 GPUs. The exact SFT hyperparameters are not detailed, and the paper does not specify whether the scalar prediction is implemented through regression, classification, or a more instruction-tuning-style decoding format. It also explicitly does not report a pairwise ranking objective such as

$L = -\log \sigma(s(x^+) - s(x^-)).$

The critic learned from FIRM-Gen-293K becomes the Instruction Following component in the generation reward used for RL. The paper introduces Quality-Modulated Alignment (QMA) to address a documented reward-hacking problem: using Instruction Following alone as reward can produce outputs such as “black shadow” silhouettes that satisfy object-category constraints while lacking visual fidelity. The QMA reward is given as

$$R_{QMA} = \text{InsFollowing} \cdot \left(w_1 + w_2 \cdot \text{Quality}\right),$$

with $w_1 = 0.4$ and $w_2 = 0.6$.

In this formulation, InsFollowing is the alignment score predicted by the generation critic trained on FIRM-Gen-293K, and Quality is a separate signal that provides a bonus when visual quality is high. The multiplicative structure is described as penalizing cases with high alignment but low quality, and vice versa to a lesser extent. The source of the Quality term is not specified. Normalization ranges for generation scores are also not specified, although normalization to $[0,1]$ is discussed elsewhere in the paper for editing rewards.

The paper further notes that the surrounding RL framework uses DiffusionNFT, an online RL method defined over the forward diffusion or flow-matching process. This objective is presented as orthogonal to FIRM-Gen-293K itself; the dataset’s function is to provide a reliable InsFollowing critic.

5. Benchmarking and empirical performance

Critic evaluation is performed on FIRM-Bench, a separate, fully human-annotated benchmark with $n=807$ total examples and a generation subset of $n=250$, with uniform score distributions across the 1–5 scale (Zhao et al., 12 Mar 2026). For generation, the benchmark is split into Easy and Hard subsets by prompt complexity. Reliability is measured using MAE between critic predictions and human-labeled ground truth scores.

The reported generation-critic comparisons include proprietary MLLMs—GPT-4o, GPT-4.1, GPT-5, and Gemini-3-Pro—and open-source MLLMs—Qwen3-VL-8B, Qwen3-VL-32B, Qwen3-VL-235B, InternVL3.5-8B, InternVL3.5-38B, and InternVL3.5-241B. The paper does not report comparisons to CLIPScore, ImageReward, PickScore, HPS v2, or aesthetic-only scorers.

For FIRM-Bench-Gen, the reported MAE values indicate the following: Gemini-3-Pro achieves 0.40 overall, GPT-5 achieves 0.52 overall, Qwen3-VL-32B achieves 0.54 overall as the best open-source baseline, and FIRM-Gen-8B achieves 0.51 overall, with 0.45 on Easy and 0.57 on Hard. The paper states that this surpasses GPT-5 and all evaluated open-source baselines. Statistical significance tests are not reported.

The downstream RL outcomes are reported for FIRM-SD3.5, which uses SD3.5-Medium as the base generator and is trained with QMA using FIRM-Gen-8B as critic. The paper reports the following gains over the base model:

Benchmark	Base	FIRM-SD3.5
GenEval	0.52	0.77
DPGBench	84.08	87.16
TIIF (test-mini-short)	70.17	77.12
UniGenBench++ Short	60.71	69.56
UniGenBench++ Long	64.67	76.22

These improvements are also reported as $+0.25$, $8 \times$0, $8 \times$1, $8 \times$2, and $8 \times$3, respectively. The paper further states that, relative to using alternative Qwen3-VL reward models, FIRM-Gen-8B consistently drives larger gains, especially on complex prompts such as those emphasized by DPGBench, TIIF, and UniGenBench++. This suggests that checklist-supervised critic training on FIRM-Gen-293K improves reward attribution under compositional complexity.

6. Access, limitations, and interpretive significance

The paper states that all datasets, models, and code are publicly available, and provides three access points: the project page at https://firm-reward.github.io/, the code repository at https://github.com/VisionXLab/FIRM-Reward, and the Hugging Face collection at https://huggingface.co/collections/VisionXLab/firm-reward (Zhao et al., 12 Mar 2026). It does not provide licensing terms, usage restrictions, or detailed download and preprocessing instructions.

Several limitations are also explicit in the description. Labels for FIRM-Gen-293K are produced by planner–evaluator LLMs rather than human annotators. Human verification for the dataset itself is not reported. Inter-annotator agreement is not applicable or not reported. Exact schema fields beyond prompt, image, plan, reasoning, and score are not specified. Image resolutions are not specified. The precise critic-training loss is not specified. The source or evaluator used to compute the Quality term in QMA is not specified. Specific bias analysis and ethical considerations for FIRM-Gen-293K are likewise not detailed.

These limitations constrain how the dataset should be interpreted. It is not a human-annotated gold standard; rather, it is a large-scale synthetic supervision resource designed to make reward models less hallucination-prone through explicit criterion decomposition. A plausible implication is that its main contribution lies less in raw label authority than in annotation structure: the plan-then-score protocol operationalizes instruction following as checklist-grounded verification, and this structured supervision appears to transfer to improved critic-human alignment and improved RL outcomes. In that sense, FIRM-Gen-293K functions as a methodological dataset as much as a corpus: its central innovation is the coupling of explicit planning, multimodal verification, and downstream reward shaping for faithful text-to-image generation.