Counterfactual Size Text-Image Dataset

• Counterfactual size text-image datasets are constructed to intentionally invert natural size distributions, enabling precise evaluation of multimodal models.
• The framework utilizes automated prompt rewriting, advanced ranking strategies, and enhanced segmentation-based evaluators, achieving metrics like 0.88 F1 and 114% improvement in image segmentation.
• Applications span robustness diagnostics, creative visual generation, cognitive research, and medical imaging, offering actionable insights into model fairness and attribute control.

A counterfactual size text-image dataset consists of paired images and text where object sizes intentionally contradict commonsense or natural distributions, such as depicting a tiny walrus next to a giant button. Counterfactuality in this context refers to the generation or annotation of image–text examples that do not naturally occur or are highly improbable, demanding deliberate interventions—both semantically and visually—to construct meaningful, challenging data. The development of such datasets is driven by the goal of evaluating and enhancing the controllability, reasoning, and robustness of multimodal models with respect to attributes like size, beyond their learned priors.

1. Definition and Motivation

Counterfactual size text-image datasets are constructed to address the scarcity of naturally occurring examples where attribute values (e.g., object sizes) are intentionally inverted or assigned implausible relationships not captured in real-world data. This supports research in creative generation, robustness diagnostics, causal reasoning, and the disentanglement of spurious correlations in multimodal models. Applications include controlled benchmarking, augmentation for vision–language representation learning, and psychological or perception studies requiring precise attribute manipulation.

Unlike standard datasets where objects appear with typical real-world sizes (e.g., a large car, a small coin), counterfactual size datasets explicitly guide the generative or editing process to produce “anti-physics” scenarios, such as a car on a palm or a building tiny compared to a pencil (Jelaca et al., 23 Sep 2025).

2. Frameworks and Methodologies for Dataset Construction

The construction of counterfactual size datasets involves several algorithmic components:

• Prompt and Image Generation Pipeline: An automatic prompt engineering framework is deployed, consisting of (i) a prompt rewriter for modifying seed prompts (via LLMs) to encourage counterfactuality, (ii) a ranking model trained with DPO (Direct Preference Optimization) to select prompts that maximally induce the desired size inversion, and (iii) a segmentation-based image evaluator to validate whether the generated image actually reflects the prescribed counterfactual size relationship (Jelaca et al., 23 Sep 2025).
• Image Evaluator: An enhanced evaluator leverages Grounded SAM, complemented by refinements such as exclusive mask assignment, label verification (using CLIP embeddings), tiny-region filtering, and adaptive thresholds, to accurately compute the area ratios of segmented objects and verify whether a prompt-induced image expresses, for instance, “the small object is larger than the big object.”
• Dataset Assembly: Objects are chosen from curated sets partitioned by natural size (e.g., “big” and “small” categories). Pairwise base prompts (e.g., “Big [small object] and small [big object]. The [big object] is much smaller than the [small object].”) are rewritten and filtered using the above pipeline. Successful instances, as labeled by the evaluator, are included as positive counterfactual pairs.

3. Technical Innovations and Benchmarking

A summary of the main technical innovations and their empirical outcomes is presented below:

Component	Core Function	Key Result/Metric
Image Evaluator (Refined Grounded SAM)	Segmentation, mask assignment, label verification	114% improvement in F1 over vanilla SAM (Jelaca et al., 23 Sep 2025)
Prompt Rewriter + DPO Ranker	Prompt transformation and selection	30.3% accuracy in faithful generation; 3x baseline
Final Triplet Dataset	Counterfactual base and revised prompts with images	7304 validated (base, positive, negative) triplets

The framework achieves an F1 score of 0.88 in recognizing correct counterfactual size assignments, with prompt generation accuracy (as assessed by the evaluator) of 30.3%—a significant margin above both hand-crafted base prompts and strong LLM baselines.

4. Integration with Generative and Editing Approaches

Several parallel or complementary generative methodologies can inform the design and exploitation of counterfactual size datasets:

• Editing-based Counterfactuals: Techniques such as text-driven latent code manipulation in StyleGAN (CF-CLIP), using a novel CLIP-NCE loss and explicit text embedding mapping, enable modification of properties—potentially including size—against the prior distributions of the generator (Yu et al., 2022).
• Inference-based Methods: Doubly abductive counterfactual inference decomposes image editing into two exogenous factors (image content and semantic change), employing LoRA modules to disentangle semantic edits (e.g., size modifications) from overall fidelity. The process formalizes counterfactuality as abduction–action–prediction steps (Song et al., 5 Mar 2024).
• Diffusion Model Integration: Adapters injecting causal attribute conditions into frozen diffusion models have demonstrated state-of-the-art fidelity and precision in manipulating attributes such as object size, propagating the semantic effect across causal descendants in a structural causal model (Tong et al., 29 Sep 2025).

Such methodologies facilitate the synthesis of paired data where only the size attribute is systematically intervened upon, enabling both large-scale construction and strict control over confounding variables.

5. Evaluation Metrics and Automated Assessment

Robust evaluation of generated or edited counterfactual size data demands both automated and human metrics:

• Area Ratio Score: Defined as $S = \min(\tau_R, A_s/A_b)$ if the size relation is correctly inverted, with penalty factors if one or both objects are missing or misidentified (Jelaca et al., 23 Sep 2025).
• Perceptual Metrics: LPIPS and FID are used to assess the fidelity of edited images relative to originals and real images.
• Downstream Model Performance: Datasets are evaluated by their effect on out-of-domain (OOD) generalization, image–text matching, or compositional reasoning tasks. COCO-Counterfactuals, for instance, leads to performance drops of up to 50% on retrieval when models are tested out-of-domain, highlighting the challenge and value of these counterfactual examples (Le et al., 2023).

6. Applications and Implications

Counterfactual size text-image datasets serve as critical resources in multiple contexts:

• Robustness and Bias Mitigation: By exposing models to deliberate inversions of commonsense size relations, these datasets help models avoid overfitting to spurious correlations and improve causal feature reasoning.
• Creative Visual Content Generation: Enables artistic and design workflows where size inversion or implausible scenes are required.
• Scientific and Cognitive Research: Supports the construction of stimuli for perception and reasoning studies requiring precise manipulation of semantic attributes.
• Medical and Scientific Imaging: Allows for controlled simulation of attribute variations such as tumor or organ size progression, supporting clinical decision-making, diagnosis, and fairness analysis (Tong et al., 29 Sep 2025).

7. Limitations and Future Directions

The construction and utilization of counterfactual size datasets faces several constraints:

• Expressivity vs. Realism Trade-off: Ensuring that counterfactual images are plausible (visually convincing) despite their implausible semantics remains a challenge. The underlying generative or editing models may not generalize well for drastic size reversals or extreme attribute values (Yu et al., 2022, Tong et al., 29 Sep 2025).
• Automated Evaluation gappiness: While advanced evaluators (such as refined Grounded SAM augmented by CLIP label verification) reduce annotation costs, rare labeling errors and semantic mismatches still occur, especially when segmentation fails due to occlusion or style variation.
• Scaling and Domain Transfer: Most current frameworks focus on a pairwise or limited set of objects due to segmentation or prompt design constraints. Extending to more complex or densely populated scenes, or to specialized domains (e.g., medical, satellite), may require additional advances in both dataset construction and generative model controllability.
• Attribute Entanglement: Maintaining minimal unintended changes in non-intervened attributes is an ongoing concern. Techniques like Causal-Adapter’s prompt-aligned injection and contrastive losses help but may not fully eliminate attribute leakage.

Future research is likely to pursue broader attribute controllability (beyond size), larger and more diverse curated prompt sets for unnatural attribute combinations, deeper integration of semantic and compositional reasoning, and the development of more comprehensive evaluation and scoring schemes. Efforts to leverage counterfactual size datasets for targeted data augmentation, fairness diagnostics, or explainability in multimodal learning are plausible emergent directions.

References to Notable Works and Resources

• “Automated Prompt Generation for Creative and Counterfactual Text-to-image Synthesis” (Jelaca et al., 23 Sep 2025)
• “Towards Counterfactual Image Manipulation via CLIP” (Yu et al., 2022)
• “COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs” (Le et al., 2023)
• “Doubly Abductive Counterfactual Inference for Text-based Image Editing” (Song et al., 5 Mar 2024)
• “Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generation” (Tong et al., 29 Sep 2025)

These papers collectively establish the theoretical, algorithmic, and practical groundwork for constructing, leveraging, and evaluating counterfactual size text-image datasets, advancing both the methodological rigor and creative boundaries in multimodal AI research.