Image-Oriented Self-Adaptive Dataset Construction

• Image-oriented self-adaptive dataset construction is a process that leverages image signals to dynamically regulate selection, filtering, synthesis, and validation for high-quality datasets.
• It employs adaptive mechanisms such as ensemble consensus, foreground occupancy thresholds, and diffusion-based generation to ensure robust and diverse data collection across fields like sentiment analysis and medical imaging.
• The approach integrates formal decision functions and iterative feedback loops, enhancing dataset transferability, domain robustness, and reducing annotation costs.

Image-oriented self-adaptive dataset construction denotes dataset-building procedures in which images, image-derived features, or image-conditioned signals are the primary control variables, and the dataset is shaped by adaptive selection, filtering, synthesis, or validation rather than by a fixed one-pass collection. Across the recent literature, this pattern appears in visual sentiment analysis, dataset distillation, subject-driven text-to-image generation, task-oriented dialogue with image persona, multimodal safety, medical imaging, self-supervised augmentation, and text-rich image generation, using mechanisms such as ensemble consensus, foreground occupancy thresholds, cross-attention or DINO-based conditioning, retrieval-and-review loops, complementary transformation distributions, and differentiable camera pipelines (Blanchini et al., 16 Dec 2025, Li et al., 6 Jan 2026, Zhou et al., 2024, Lee et al., 24 Apr 2025, Qu et al., 4 Sep 2025, Roth et al., 2024, Yang et al., 2021, Yan et al., 30 Jun 2026, Oala et al., 2022).

1. Definitional scope and lineage

The cited literature exhibits a recurring pattern: the image is not merely the final payload of a dataset, but the organizing object around which selection, labeling, and refinement are performed. In "Enhancing Visual Sentiment Analysis via Semiotic Isotopy-Guided Dataset Construction" (Blanchini et al., 16 Dec 2025), the image pool is filtered by ensemble agreement so that accepted samples are those whose emotional meaning is robust across dataset-specific biases. In "Foreground-Aware Dataset Distillation via Dynamic Patch Selection" (Li et al., 6 Jan 2026), the adaptive variable is foreground occupancy extracted from Grounded SAM2. In "Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven Text-to-Image Generation" (Zhou et al., 2024), subject identity is captured from the image itself through DINOv2 embeddings, and the dataset is then expanded without subject-level fine-tuning.

Earlier web-image construction work already established two principles that remain central. First, multiple visual subpopulations must be preserved rather than collapsed into a single canonical distribution. "Exploiting Web Images for Dataset Construction: A Domain Robust Approach" (Yao et al., 2016) addresses this by query expansion, constrained positive bags, and multi-instance learning; "Towards Automatic Construction of Diverse, High-quality Image Dataset" (Yao et al., 2017) similarly uses multiple textual queries, multi-view learning, and bag-then-instance filtering. These papers are not image-first in the stricter sense used by later work, but they formalize diversity preservation, noise removal, and domain robustness as construction objectives.

More recent work extends the same logic from selection to fully stateful construction engines. "Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarios" (Qu et al., 4 Sep 2025) starts from safe, real photographs and constructs matched safe texts and guidance responses. "DataEvolver: Let Your Data Build and Improve Itself via Goal-Driven Loop Agents" (Zhang et al., 3 May 2026) and "DataEvolver: Self-Evolving Multi-Agent Data Construction for Text-Rich Image Generation" (Yan et al., 30 Jun 2026) turn dataset construction into an explicit loop with goals, artifacts, review traces, corrective actions, and acceptance decisions. "Data Models for Dataset Drift Controls in Machine Learning With Optical Images" (Oala et al., 2022) pushes the notion further by modeling image generation itself as a differentiable physical process.

2. Adaptive mechanisms

Selection-based adaptation is the most common form. Selecta pools two million images from ImageNet, COCO, and OpenImages, predicts them with three CLIP+MLP classifiers trained separately on VippSent, Flickr, and Instagram, and retains only 3-out-of-3 consensus cases, yielding approximately 400,000 automatically labeled images (Blanchini et al., 16 Dec 2025). Foreground-aware distillation computes per-image foreground occupancy with Grounded SAM2, derives a class-specific threshold $\tau_c$, and then either selects the most informative patch from random crops or resizes the full image when the foreground dominates (Li et al., 6 Jan 2026). ADC similarly combines LLM-based class design, web harvesting, feature-based label-noise detection, and early-learning filtering, while Dataset Factory externalizes adaptive signals as immutable metadata columns—embeddings, scores, distances, and classifier outputs—so that dataset versions can be re-queried and re-saved without moving the raw samples (Liu et al., 2024, Kharitonov et al., 2023).

A second family is generation-based adaptation. Toffee pre-trains only two reusable diffusion models—a Refiner and a View Generator—and then constructs Toffee-5M by conditioning on DINOv2-Giant image embeddings, CLIP text alignment, DINO identity filtering, and Grounded-SAM masks, producing 4.8M image pairs after filtering, including 1.6M editing pairs with masks (Zhou et al., 2024). DiffuGen uses Stable Diffusion, prompt templating, textual inversion, and either unsupervised DAAM-style cross-attention labeling or supervised YOLOv8-seg labeling to create labeled synthetic image data (Shenoda et al., 2023). In multimodal safety, RMS augments image-first patterns discovered from COCO through LAION retrieval and reviewer filtering, rather than by predefining a narrow risk ontology and then synthesizing images (Qu et al., 4 Sep 2025).

A third family is distributional or physically grounded adaptation. "Distribution Estimation to Automate Transformation Policies for Self-Supervision" (Yang et al., 2021) estimates the transformation distribution present in a dataset and then samples from its complement to create pretext pairs that are absent or rare in the source distribution. "Your Image is My Video" (Casarin et al., 2024) uses Differentiable Augmentation Search to convert an image into a pseudo-video whose temporal fusion reshapes the spatial receptive field. "Domain-Adaptive Pre-training of Self-Supervised Foundation Models for Medical Image Classification in Gastrointestinal Endoscopy" (Roth et al., 2024) adapts the label space rather than only the image pool, using dynamic class mappings so that ten public and private datasets can be harmonized into EndoExtend24 with over 226,000 labeled images and up to 123 pathological findings. "Data Models for Dataset Drift Controls in Machine Learning With Optical Images" (Oala et al., 2022) treats the ISP itself as a differentiable, adjustable component.

3. Representative construction paradigms

The literature differs in task, source pool, adaptive signal, and output artifact, but the core structures are sufficiently recurrent to permit a compact typology.

Representative method	Domain	Primary adaptive signal
Selecta (Blanchini et al., 16 Dec 2025)	Visual sentiment analysis	3-out-of-3 ensemble consensus
Foreground-aware dynamic patch selection (Li et al., 6 Jan 2026)	Dataset distillation	$f(I)$ and $\tau_c = Q_{0.3}(\mathcal S_c)$
Toffee (Zhou et al., 2024)	Subject-driven editing and generation	DINOv2 embeddings, CLIP/DINO filtering
PicPersona-TOD (Lee et al., 24 Apr 2025)	Task-oriented dialogue	Image persona, first impressions, PS/PV filtering
RMS (Qu et al., 4 Sep 2025)	Multimodal safety	Image-first latent-risk discovery and reviewer filtering
EndoExtend24 (Roth et al., 2024)	Gastrointestinal endoscopy	Dynamic class mappings and patient-aware splits
ATP (Yang et al., 2021)	Self-supervised learning	Complementary transformation distribution
DataEvolver (Zhang et al., 3 May 2026, Yan et al., 30 Jun 2026)	Goal-driven and text-rich generation	Review traces, gates, feedback memory

The table also indicates that “dataset construction” no longer denotes only image collection and annotation. In PicPersona-TOD, the output is a dialogue dataset in which images instantiate persona factors such as age, gender, formality, and coarse emotion, and the adaptive filters are embedding-based measures of personalization strength and personalization direction (Lee et al., 24 Apr 2025). In RMS, each sample stores image, safe text, risk category, safe guidance, and unsafe guidance (Qu et al., 4 Sep 2025). In DataEvolver, the persistent record may include RGB renders, masks, depth maps, normals, meshes, poses, trajectories, action programs, review signals, and verdict metadata (Zhang et al., 3 May 2026).

This suggests that the field is increasingly organized around construction-time control rather than only post hoc curation. Dataset Factory makes this explicit at infrastructure level by representing datasets as immutable named tables, with rows pointing to archived samples and columns holding metadata and learned signals (Kharitonov et al., 2023). DataEvolver makes it explicit at agent level by preserving construction traces and allowing later rounds to depend on earlier rejections (Yan et al., 30 Jun 2026).

4. Formal criteria and mathematical structure

Several papers formalize self-adaptation through explicit decision functions rather than through informal heuristics. Selecta models images in simulation as multisets of symbols and assigns a label by majority rule,

$$\ell(X)= \left\{ \begin{array}{ll} nt & \text{if } \#nt > \frac{k_X}{2} \ p & \text{if } \#nt \le \frac{k_X}{2} \ \cap \ \#p > \#n \ n & \text{otherwise} \end{array} \right.$$

with multiset IoU similarity

$$\text{IoU}(X,Y)=\frac{|\text{set}(X)\cap\text{set}(Y)|}{|\text{set}(X)\cup\text{set}(Y)|}.$$

For real images, the method does not define an explicit isotopy score $S(I)$; instead, 3-out-of-3 classifier agreement is used as a practical isotopy proxy. Foreground-aware distillation defines per-image foreground occupancy

$$f(I)=R_{\text{object}}(I)=\frac{1}{H\cdot W}\sum_{m=1}^{H}\sum_{n=1}^{W}F(m,n),$$

and a class-specific threshold

$\tau_c=Q_{0.3}(\mathcal S_c),$

so that the selected patch is

$$P^*(I)= \begin{cases} \arg\max_{P\in Crop(I,k)} S(P), & \text{if } f(I)<\tau_c \ Resize(I,s_{\text{patch}}), & \text{otherwise.} \end{cases}$$

These formulations capture two distinct adaptive logics: agreement as latent-pattern detection and thresholded occupancy as content-aware routing (Blanchini et al., 16 Dec 2025, Li et al., 6 Jan 2026).

Other work adapts through distributional estimation. ATP estimates a transformation histogram and constructs a complementary distribution

$$q_{\theta}(b)=\frac{H_{\max}-H(b)}{\sum_{b'}(H_{\max}-H(b'))},$$

then samples from $f(I)$0 via inverse transform sampling to build effective pretext pairs (Yang et al., 2021). In raw2logit, the processed view is a differentiable composition

$f(I)$1

which allows gradients to flow from task loss to data-generation parameters (Oala et al., 2022). In "Your Image is My Video", DAS relaxes operator choice with a Softmax over candidate transforms,

$f(I)$2

and optimizes augmentation parameters jointly with network weights (Casarin et al., 2024).

Goal-driven systems encode adaptation through gates and verdicts. DataEvolver defines a hybrid reviewer score

$f(I)$3

and, when multi-view evidence exists,

$f(I)$4

with hard caps applied when object integrity, composition, or semantic render quality are below required levels (Zhang et al., 3 May 2026). Text-rich DataEvolver similarly summarizes round-level construction by a vector $f(I)$5 and updates the next-round retrieval and prompt policy from semantic feedback memory (Yan et al., 30 Jun 2026). In multimodal dialogue, PicPersona-TOD defines personalization strength

$f(I)$6

and personalization direction via

$f(I)$7

thereby turning stylistic adaptation into a filterable geometric criterion (Lee et al., 24 Apr 2025).

5. Empirical performance across domains

The most direct evidence comes from cross-dataset and cross-architecture evaluations. In VSA, Selecta-trained CLIP+MLP reaches 89.34% on IAPS, 88.52% on GAPED, 68.61% on NAPS, 78.65% on OASIS, 82.91% on FI, 56.82% on VSO, 81.85% on Emotion6, and 82.45% on Emotic, outperforming the union of VippSent, Flickr, and Instagram on all reported test sets; the 3/3 consensus ablation also outperforms 2/3 consensus on all psychological datasets despite yielding fewer images (Blanchini et al., 16 Dec 2025). In dataset distillation, the foreground-aware method improves over RDED on ImageNette, ImageWoof, CIFAR-10, and CIFAR-100 across IPC settings, while the ablations show that $f(I)$8 is the best threshold quantile and $f(I)$9 patches per distilled image gives the best trade-off between information richness and spatial fidelity (Li et al., 6 Jan 2026). DAS produces consistent gains over baseline augmentation, dilated convolutions, larger kernels, and AutoAugment/RandAugment under a fixed 24-hour budget on ImageNet, CIFAR-10, CIFAR-100, Tiny-ImageNet, Pascal VOC 2012, and Cityscapes (Casarin et al., 2024).

In generative construction, Toffee-5M reaches 4.8M image pairs after filtering, includes 1.6M editing pairs with masks, and keeps pre-training cost under 3,000 GPU hours, compared with SuTI’s ~83,000 TPU hours for 1M subjects and CAFE’s ~10,000 GPU hours. On DreamBench, ToffeeNet reports DINO 0.728, CLIP-I 0.817, and CLIP-T 0.306, remaining competitive with methods that require test-time tuning (Zhou et al., 2024). Self-Distilled StyleGAN shows that generative self-filtering improves FID on LSUN-Bicycle from 5.42 to 3.66, on LSUN-Horse from 4.05 to 2.81, on Internet Lions from 5.15 to 3.43, and on Internet Dogs from 19.49 to 15.42; multimodal truncation is also preferred to global-mean truncation in human studies at matched FID (Mokady et al., 2022).

Image-first adaptive construction has also been validated in safety and text-rich generation. RMS produces a 35,610-pair dataset across 12 categories and 39 fine-grained scenarios, and the judge trained on RMS transfers better than judges trained on VLGuard or C$\tau_c = Q_{0.3}(\mathcal S_c)$0Ef on most reported targets: 29.59% on C$\tau_c = Q_{0.3}(\mathcal S_c)$1Ef, 26.25% on VLGuard, 21.56% on SIUO, and 30.25% on MSSbench (Qu et al., 4 Sep 2025). In text-rich image generation, DataEvolver improves OCR-F1 over the strongest baseline by 85.3% on TextScenesHQ and 35.3% on LongTextBench at the 0.75M scale on PixArt-α, and the gains transfer to Show-o2 (Yan et al., 30 Jun 2026). In medical imaging, EndoExtend24 coupled with domain-adaptive pre-training secures third place in the Capsule Endoscopy 2024 Challenge with macro AUC 0.762 and balanced accuracy 37.1% on the test set, compared with a best baseline ResNet50V2 at macro AUC 0.542 and BA 17.7% (Roth et al., 2024).

Older web-image literature provides a complementary validation criterion: domain robustness. DRID-20 achieves the highest average cross-dataset accuracy among the compared automatically constructed datasets and reaches average AP 19.74 in object detection on PASCAL VOC 2007, outperforming IDC-MTM, WSVCL, SPM-VID, and WSL (Yao et al., 2016). WSID-100 reports average classification accuracy 53.88 over 14 categories on VOC 2007 and DPM mAP 21.32 in object detection, again surpassing prior weakly supervised and web-supervised baselines (Yao et al., 2017). The continuity between these results and the later image-first systems is substantive: both generations of work improve not merely in-domain fitting but transfer across domains, architectures, or tasks.

6. Constraints, misconceptions, and future directions

A common misconception is that “self-adaptive” necessarily denotes an active-learning loop with repeated retraining. Selecta explicitly states that no iterative loop is reported; its adaptiveness comes from consensus-based filtering and diversification of the unlabeled pool, not from active learning or iterative retraining (Blanchini et al., 16 Dec 2025). By contrast, both DataEvolver variants implement explicit multi-round feedback loops in which rejection causes, review traces, or downstream weaknesses alter the next construction round (Zhang et al., 3 May 2026, Yan et al., 30 Jun 2026). The term therefore covers both one-shot adaptive selection and fully closed-loop construction.

Another recurrent limitation is that adaptive signals are often coarse proxies for semantic validity. Foreground occupancy does not encode dispersion, shape complexity, or instance count, and inaccurate or incomplete Grounded SAM2 masks can mislead patch selection (Li et al., 6 Jan 2026). In Selecta, no manual inter-annotator agreement is reported, and emotional responses are acknowledged as culturally and contextually variable, with future work suggesting an explicit ambiguous class (Blanchini et al., 16 Dec 2025). PicPersona-TOD is skewed toward adult and casual contexts and uses coarse positive/neutral/negative emotions, while RMS explicitly does not cover some risks that typically require unsafe content in at least one modality, such as gambling, drugs, or misinformation (Lee et al., 24 Apr 2025, Qu et al., 4 Sep 2025).

The data dependencies are equally important. Toffee’s View Generator can degrade on underrepresented categories because MVImageNet has limited coverage for those classes (Zhou et al., 2024). EndoExtend24 inherits label heterogeneity because only ERS is fully aligned to MST 3.0 terminology, and overlap control depends on filename matching where identifiers exist (Roth et al., 2024). raw2logit requires RAW data for full physical fidelity and currently focuses on ISP changes rather than a complete optics model (Oala et al., 2022). Text-rich DataEvolver depends on the reliability of OCR, perceptual hash deduplication, CLIP similarity, and the Critic’s ability to transform rejection causes into useful semantic feedback (Yan et al., 30 Jun 2026).

The most consistent future direction is multimodal and multi-artifact expansion. Selecta explicitly highlights multimodal extension of isotopy analysis across image and text (Blanchini et al., 16 Dec 2025). Foreground-aware distillation proposes adaptive layouts, diversity controls, and hybridization with optimization-based distillation (Li et al., 6 Jan 2026). RMS identifies broader inspiration datasets, audio and video, and richer evaluation beyond accuracy as future work (Qu et al., 4 Sep 2025). DataEvolver proposes image plus mask/depth/normal exports, then smooth rotation video, translation video, multi-object relation data, and compositional video editing data with multi-step edit sequences (Zhang et al., 3 May 2026). A plausible implication is that the field is moving from dataset construction as static acquisition toward dataset construction as controllable, inspectable, and continuously revisable infrastructure.