RealGen: Realistic Generative Modeling

• RealGen is a family of frameworks that generate photorealistic images, controllable traffic scenarios, and synthetic tabular data using advanced architectures.
• It integrates detectors and retrieval-augmented methods with reinforcement learning to enforce explicit realism constraints across domains.
• RealGen demonstrates significant improvements in metrics like mADE and LD, ensuring high fidelity, control, and privacy in synthetic data generation.

RealGen refers to a family of frameworks and methodologies in generative modeling focused on achieving realism (“indistinguishable-from-reality”) and robust controllability across a range of domains, including photorealistic image synthesis, controllable traffic scenario generation, and realistic synthetic tabular/relational data. The common thread across RealGen instantiations is the integration of advanced generative architectures (typically transformers or diffusion models) with explicit realism constraints—often via retrieval, detector-driven rewards, or both—to overcome the limitations of traditional overfit or artifact-prone generative approaches.

1. RealGen for Photorealistic Text-to-Image Generation

The RealGen text-to-image framework introduces detector-guided reinforcement learning directly into both LLM-driven prompt optimization and diffusion-based image generation. The core system is decomposed into two modules: a prompt optimization LLM (Qwen-3 4B with LoRA adapters) generating rich candidates from short prompts, and a latent-diffusion image generator (FLUX.1-dev, also adapted via LoRA). Training proceeds via initial supervised fine-tuning followed by generalized reinforcement policy optimization (GRPO) using a composite “Detector Reward.”

The Detector Reward is multi-objective, computed by:

• Semantic-level detectors (e.g., Forensic-Chat), outputting logits for “fake” versus “real” on image artifacts, such as plastic textures or unnatural lighting.
• Feature-level detectors (OmniAID), providing a probability that an image is “fake” at the feature representation level.
• Text-image alignment (Long-CLIP cosine similarity), ensuring semantic fidelity to the prompt.

The normalized advantage is computed per-sample using Z-normalization across batch rewards, and this signal is used to optimize policy parameters (LLM or diffusion) through GRPO, which clips and regularizes the policy update using KL divergence to a reference policy. All reward and optimization steps are automated and human-free, enabling fully scalable training cycles (Ye et al., 29 Nov 2025).

2. Retrieval-Augmented Scenario Generation in Autonomous Vehicle Simulation

In the traffic domain, RealGen advances scenario synthesis for simulation-based development of AVs. The framework dispenses with naive dataset-memorization, which is prone to overlook long-tail or critical events, by applying a retrieval-based in-context generation paradigm. At inference, K template scenarios are retrieved from a database using an autoencoder embedding and a minimum 2-Wasserstein (Sinkhorn) distance, with optional manual tags for behavior composition (e.g., “left-turn,” “crash”).

The retrieved behavior embeddings, user-specified initial agent poses, and a structured map context are combined using multi-head cross-attention layers (“combiner network”). Decoding produces novel, realistic agent trajectories. This retrieval-augmented architecture enables:

• Plug-in combinatorial behavior composition (e.g., U-turn + overtake)
• Fine-grained controllability through manipulation of templates, tags, and map/pose specification
• Critical scenario amplification, since edge-case templates guide retrieval toward rare but safety-critical events

Experimentally, RealGen achieves 13–14× lower mADE on nuScenes compared to KNN-based baselines and significantly reduces generated scene collision/off-road rates (Ding et al., 2023).

3. RealGen Principles in Relational and Tabular Data Synthesis

Within tabular and relational synthetic data, the “RealGen” principle is operationalized by REaLTabFormer, which jointly models parent tables (GPT-2 style autoregressive decoding) and child relational tables (Seq2Seq, parent-conditional).

Distinctive for its anti-memorization strategy, REaLTabFormer:

• Implements per-token target masking during training, substituting original tokens with MASK at random with a fixed probability, preventing trivial record copying
• Introduces the $Q_{\delta}$ statistic, leveraging the distance-to-closest-record (DCR) distribution to measure overfitting, calibrated by bootstrap early stopping; training halts if generated samples approach training data “too closely,” according to an empirically determined $Q'_{\delta}$ threshold

This produces synthetic tabular/relational datasets with high machine learning utility and strong privacy guarantees, as evidenced by low discriminator accuracy (near 50%) and high logistic detection fidelity (LD) across several benchmarks (Solatorio et al., 2023).

4. Methodological Comparison Across Domains

RealGen Variant	Domain	Realism Strategy	Control/Structure
Photorealistic T2I RealGen	Image	Detector-guided RL (semantic/feature)	LLM prompt optimization
Traffic Scenario RealGen	Multi-agent Simulation	Retrieval-augmented in-context synthesis	Templates, map/pose tags
REaLTabFormer	Tabular/Relational	Mem. avoidance (masking, $Q_{\delta}$)	Column vocabulary, seq2seq

Each variant employs explicit statistical testing or detector feedback to enforce a form of statistical realism beyond what is achievable with conventional MLE or diffusion losses alone.

5. Quantitative and Qualitative Evaluation

The photorealistic RealGen is benchmarked using RealBench, a novel detector- and LLM-based automatic framework, reporting results across Forensic-Chat, OmniAID, Effort, and GPT5-Prompt detectors. RealGen, both in its baseline and LLM-augmented forms, achieves the highest detector confidence for “realism” and tops win-rate comparisons with both synthetic and actual real samples. Notably, RealGen’s detector-guided reward boosts realism and detail without degrading human-level aesthetics, as assessed by HPS metrics.

In the traffic generation setting, realism is quantified using mADE, mFDE, scene-collision, and off-road rates, consistently showing substantial improvements over AE-KNN and related baselines. For relational tabular data, performance is validated via downstream ML metrics (R², macro-F1), discriminator accuracy near the random (ideal), and LD, all indicating high synthetic fidelity (Ye et al., 29 Nov 2025, Ding et al., 2023, Solatorio et al., 2023).

6. Limitations and Prospects

While RealGen architectures introduce significant advances in realism, controllability, and privacy-preserving data generation, current limitations include:

• Indirect enforcement of global constraints, such as consistent map–agent interactions in traffic scenarios, or strict semantic adherence in image generation
• Reliance on quality and coverage of template databases or detector capacities
• Extension to next-gen domains (e.g., video, spatio-temporal data, continuously annotated tabular records) is still exploratory

Future research may incorporate symbolic or rule-based constraints directly into embedding or decoder modules and further augment detector-side supervision for adversarial robustness (Ding et al., 2023, Ye et al., 29 Nov 2025).

7. Conclusion

RealGen defines a principled set of approaches for high-fidelity, controllable, and safe generative modeling across heterogeneous data domains. The paradigm marries advanced generative architectures with explicit feedback—retrieval, detector signals, masking, or statistical quantiles—to close the gap between synthetic and real data distributions, enable nuanced user-driven control, and address privacy and safety constraints fundamental to practical deployment (Ding et al., 2023, Solatorio et al., 2023, Ye et al., 29 Nov 2025).