Social Perception-Driven Data Generation

• SPDG is a data generation paradigm that aligns synthetic content with socially relevant objectives instead of focusing solely on visual realism.
• It enables targeted augmentation in fields like autonomous driving, social robotics, and personalized image generation by utilizing specific control interfaces.
• SPDG methodologies improve downstream perception by explicitly optimizing for social metrics, long-tail support coverage, and task-specific evaluation standards.

Social Perception-Driven Data Generation (SPDG) denotes a family of data-generation paradigms in which the target of generation, the control variables, and the validation protocol are chosen according to socially salient perception objectives rather than visual plausibility, density matching, or generic controllability alone. In the recent literature, the term is used for synthetic driving video augmentation, support-covering persona synthesis, benchmark construction from real user tasks, affect-conditioned motion generation for social robots, procedurally generated social-interaction simulation, controllable traffic scenario synthesis, and feedback-guided personalized image generation [2510.19195; 2602.03545; 2603.02586; 2205.00763; 2103.01933; 2512.01363; 2507.16095]. Across these settings, SPDG is less a single algorithm than a design principle: generate data so that the generated variation is aligned with the social or socially consequential phenomena that a downstream system must perceive, reason about, or withstand.

1. Conceptual scope and definitional variants

SPDG does not have a single canonical meaning across the cited papers. In autonomous driving, it refers to aligning synthetic generation with downstream perception needs such as detection and tracking under long-tail corner cases, and to judging synthetic data primarily by gains in metrics such as mAP, NDS, and AMOTA on real validation sets rather than by video quality alone [2510.19195]. In synthetic persona generation, it refers to producing populations that reflect the breadth of social perceptions, attitudes, and behaviors, with an explicit shift from density matching toward support coverage of what is possible, including rare combinations [2602.03545]. In benchmark construction for general agents, it denotes a human–LLM collaboration workflow that converts real user problems from public platforms into tasks that are grounded in authentic intent, intrinsically tool-dependent, and deterministically verifiable [2603.02586].

A different strand uses SPDG to mean generation conditioned on socially interpreted latent variables. In social robotics, emotional body language is generated from valence conditioning and latent-space arousal control, then evaluated with anthropomorphism, animacy, dominance, and attention judgments, forming an explicit loop from social perception targets to generation and back to human evaluation [2205.00763]. In PHASE, the objective is to generate data expressly to exercise machine social perception, namely inference of latent social goals, relations, and interactions in physically grounded environments under partial observability [2103.01933]. In SocialDriveGen, SPDG is instantiated through explicit modeling of egoism and altruism together with task-oriented extrinsic goals, so that driver interaction styles can be varied from cooperative to adversarial [2512.01363]. In personalized image generation, SPDG is operationalized through pose, human–object interaction, face identity, and gaze feedback, which are treated as structured social perception signals that refine a diffusion model during finetuning [2507.16095].

A common misconception is to treat SPDG as synonymous with “socially realistic generation.” The papers do not support such a narrow reading. Some works target human social attitudes, some target social interaction dynamics, and some target safety-critical perception systems whose failures have social consequences. A more accurate synthesis is that SPDG uses socially meaningful targets to decide what should be generated, what should be controlled, and what counts as success.

2. Generated artifacts and control interfaces

Setting	Generated artifact	Primary controls
Autonomous driving	Multi-view photorealistic videos with inserted 3D assets	$D,N,E,O,M$, calibrated $K_v$ and $E_v$, insertion category, position, distance
Synthetic personas	Populations $P={p_1,\dots,p_N}$ and embeddings $Z$	context $c$, axes $D$, items $I$, Stage 1 axis positions, Stage 2 expansion
Agent benchmark construction	Closed, verifiable tasks from real user cases	platform sourcing, non-retrievability, tool dependency, capability/environment preservation
Social robotics	Emotional body-language animations	valence conditioning, latent-space radius for arousal
Social interaction simulation	2D interaction videos with automatic labels	goals $g_i$, social coupling $\alpha_{ij}$, partial observability, environment variation
Traffic scenario generation	Joint multi-agent future trajectories	VLM proposals, extrinsic rewards, $\lambda$, $\phi$
Personalized image generation	Personalized images	pose, HOI, identity, gaze, timestep-gated feedback

The driving formulation in Dream4Drive is unusually explicit about geometric controls. An input RGB image $I \in \mathbb{R}^{H\times W\times 3}$ is decomposed into background guidance maps—depth $D$, surface normal $N$, and edge $E$—while foreground guidance comes from rendering a 3D asset as object image $O$ and object mask $M$ in a calibrated multi-camera rig. Multi-view consistency is enforced by standard pinhole projection, $\lambda \mathbf{x} = \mathbf{K}_v[\mathbf{R}_v \mid \mathbf{t}_v]\tilde{\mathbf{X}}$, so that the same 3D insertion yields view-consistent 2D projections across cameras. Dream4Drive also contributes DriveObj3D, whose pipeline uses Grounded-SAM for instance localization and segmentation, Qwen-Image for multi-view image synthesis, and Hunyuan3D for multi-view 3D reconstruction to high-fidelity meshes with PBR materials [2510.19195].

Persona Generators define the control interface through a questionnaire $q=(c,D,I)$, a target population size $N$, a fixed LLM $\theta$, and generator code $\phi$. The generator produces a population $P = G_{\phi,\theta}(c,D,N)$, then maps role-play responses into behavioral embeddings $Z=\Psi(P,I)$, where each persona is positioned along axes such as threat appraisal, opportunity appraisal, trust in institutions, or social preference dimensions. The architecture is two-stage: Stage 1 shapes positions in the social perception space and produces intermediate descriptors such as traits, stances, triggers, and rules; Stage 2 expands each descriptor into a richer persona with beliefs, values, heuristics, reactions, or memories [2602.03545].

LiveAgentBench uses SPDG not to synthesize personas or scenes, but to transform real user cases into benchmark tasks. Its control surface is procedural rather than latent: data collection from Zhihu, Quora, Baidu Knows, Xiaohongshu, BiliBili, Douyin, Stack Overflow, CSDN, TikTok, and Kuaishou; screening for non-retrievability and tool dependency; construction of closed questions that preserve capability and environment; and multilayer quality control. The result is a task package with closed-form instruction, labeled multi-step plan, single correct answer, and capability and environment metadata [2603.02586].

In PHASE, the generated artifact is a 2D animation of two embodied agents, movable objects, landmarks, and obstacles. The central controls are causal rather than stylistic: personal goals $g_i$, social relation coefficients $\alpha_{ij}$, agent embodiment parameters, wall layouts, object weights, and limited field of view with occlusion. In SocialDriveGen, the primary controls are the semantic proposals selected by a VLM, the LLM-instantiated task-oriented extrinsic rewards, and two high-level parameters: $\lambda$, which controls the relative importance of intrinsic social concerns versus extrinsic task goals, and $\phi$, a social orientation angle that trades off self versus others’ intrinsic values [2103.01933; 2512.01363].

The social-robotics and personalized-image formulations are more directly perception-conditioned. The robot-motion model conditions on valence labels in $[0,1]$ and modulates arousal by the latent-space sampling radius $r \in {3,4,5}$, with higher radius yielding higher motion amplitude and variance [2205.00763]. The personalized-image method keeps four pretrained detectors frozen—X-Pose, CMMP, ArcFace, and Sharingan—and uses their outputs on partially denoised images as control signals for pose, HOI, identity, and gaze, respectively [2507.16095].

3. Model classes and optimization strategies

Dream4Drive uses a Diffusion Transformer finetuned from MagicDriveDiT, extended with a multi-condition fusion adapter. The five dense guidance signals $C_k \in {D,N,E,O,M}$ are encoded by a VAE, patchified by 3D embedders, fused by a FusionNet, and injected into control blocks inside the DiT. Its latent diffusion objective is the usual MSE on noise prediction, augmented with a foreground mask loss in VAE-decoded space and an LPIPS perceptual loss; the reported weights are $\lambda_{\text{diffusion}}=1.0$, $\lambda_{\text{mask}}=0.1$, and $\lambda_{\text{lpips}}=0.1$. Finetuning is performed on nuScenes with six synchronized cameras per scene, at resolution $512\times 768$, 33 frames per clip, for 2000 iterations on $8\times$ NVIDIA H200 GPUs using AdamW with weight decay $0.01$, cosine learning rate with 10% warm-up, learning rate $2\times 10^{-4}$, and batch size per GPU $1$ [2510.19195].

Persona Generators cast SPDG as generator optimization. The formal objective is
$$
\phi^* = \arg\max_\phi \mathbb{E}{q\sim Q}\left[M\left(\Psi\left(G{\phi,\theta}(c,D,N), I\right)\right)\right],
$$
where $M$ is a portfolio of diversity metrics over the behavioral embeddings. Optimization uses an AlphaEvolve-style loop with 10 parallel evolutionary islands, 3 initial $\phi$ implementations, 500 iterations total, periodic extinction every (\sim 8) hours or (\sim 100) iterations, and Gemini 2.5 Pro as mutation operator. Each candidate is evaluated by generating $N=25$ personas across 40 questionnaires, simulating 10 items per person, for 10,000 responses per evaluation point. The tracked metrics are coverage, convex hull volume, mean pairwise distance, minimum pairwise distance, dispersion, and KL divergence to a quasi-random Sobol reference [2602.03545].

PHASE and SIMPLE use explicit causal models rather than diffusion. PHASE formulates generation as a decentralized partially observable Markov decision process with reward
$$
\mathcal{R}i(s,a) = R(s,g_i) + \alpha{ij} R(s,g_j) - \mathcal{C}_i(a),
$$
so helping, hindering, following, avoiding, stealing, and coordination are expressed through reward coupling and hierarchical planning. Agents have limited field of view and maintain beliefs through a particle filter with $N=50$ unweighted particles. SIMPLE then performs Bayesian inverse planning over latent goals and plans, combining theory-based simulation with local estimation, and is reported to outperform state-of-the-art feed-forward neural networks on the benchmark tasks [2103.01933].

The social-robotics formulation uses a Conditional Variational Autoencoder with MLP encoder and decoder, a 3-dimensional Gaussian latent space, and concatenative conditioning on valence. The training objective is a CVAE ELBO with MSE reconstruction and a $\beta$-weighted KL term, where $\beta=0.001$. Arousal is not labeled per animation; instead it is induced by geometry-aware horn-torus latent sampling at radii $r=3,4,5$. The model is trained for 250 epochs with Adam at learning rate (10^{-4}), Xavier uniform initialization, dropout (0.5), and an 80/20 train-validation split [2205.00763].

SocialDriveGen is hierarchical. A VLM first produces a scene description (D_{\text{scene}}) from the scenario (\mathcal{S}=(\cup_{i=1}^{N}\tau_i,\mathcal{M})), selects a focal vehicle pair, and generates adversarial proposals. An LLM then constructs extrinsic rewards from those proposals and combines them with intrinsic social rewards using the parameters (\lambda) and (\phi). Joint multi-agent trajectories are synthesized by a diffusion model guided during denoising by an evolutionary strategy, with reward-weighted elite selection
$$
q(\mathbf{x}) = \frac{\exp(\tau \cdot R(\mathbf{x}))}{\sum_{j=1}^{M}\exp(\tau \cdot R(\mathbf{x}_t^{(j)}))},
$$
and an annealed temperature (\tau) that shifts sampling from exploration to exploitation [2512.01363].

The personalized-image method keeps the base personalized diffusion architecture of SSR-Encoder on Stable Diffusion v1.5, but replaces purely denoising-based finetuning with detector-driven feedback. The predicted noise is converted into (\hat{z}0) and (\hat{x}_0), then losses are applied for identity, pose, HOI, gaze, boundary coherence, and the original regularizer. The total objective uses timestep-dependent gating,
$$
L{\text{total}}(t)=L_{\text{diffusion}}(t)+\lambda_{\text{reg}}L_{\text{reg}}+\lambda_{\text{boundary}}(t)L_{\text{boundary}}+\lambda_{\text{id}}(t)L_{\text{id}}+\lambda_{\text{gaze}}(t)L_{\text{gaze}}+\lambda_{\text{pose}}(t)L_{\text{pose}}+\lambda_{\text{HOI}}(t)L_{\text{HOI}},
$$
with (\lambda_k(t)=\alpha_k \mathbf{1}[t\le \tau_k]), (\alpha_k=0.01), and thresholds (\tau_{\text{gaze}}=200), (\tau_{\text{id}}=400), (\tau_{\text{HOI}}=500), (\tau_{\text{pose}}=700) [2507.16095].

4. Evaluation logics and evidence standards

A defining feature of SPDG is that evaluation is downstream and task-specific. Dream4Drive explicitly argues that prior synthetic-data evaluations are often unfair because they pretrain on synthetic and finetune on real, thereby doubling the epoch budget; under equal epochs, those methods often show negligible or negative gains over real-only training. Dream4Drive therefore evaluates real-only and real-plus-synthetic under matched 1×, 2×, and 3× epochs and reports detection on nuScenes with mAP, mean Average Velocity Error, mean Average Orientation Error, and NDS, and tracking with AMOTA, AMOTP, MOTA, and Recall [2510.19195].

Persona Generators make an equally explicit methodological choice, but in a different direction. Their claim is that density matching underexplores the long tail, so support coverage should be optimized directly. Coverage is approximated by Monte Carlo in continuous (D)-space; KL divergence is computed against a quasi-random Sobol reference; convex hull volume, mean pairwise distance, minimum pairwise distance, and dispersion penalize clustering and empty regions. The evaluation therefore asks whether a generator spans the feasible perception space, not whether it reproduces the modal persona [2602.03545].

LiveAgentBench emphasizes benchmark validity rather than generative fidelity. Its SPDG pipeline has four stages—data collection, data screening, task construction, and quality control—and every stage uses explicit acceptance criteria. Questions answerable via simple RAG or search are excluded; only cases that inherently require at least one tool are retained; tasks are discarded and reconstructed if more than a 50% mismatch exists between original and rewritten environment and examined capabilities; and any task with fewer than 2 planned steps or no tool requirement is removed. Final answers are double-blind with third-person tie-break, and evaluation uses deterministic string matching rather than judge models [2603.02586].

Human perceptual evaluation remains central in the social-robotics and PHASE traditions. The body-language study evaluates generated and hand-designed robot animations with Godspeed Anthropomorphism and Animacy scales, affective ratings of valence, arousal, and dominance, and Likert items for attention and emotionality [2205.00763]. PHASE validates its simulated events through human experiments showing that humans perceive rich interactions in the animations and that simulated agents behave similarly to humans, while benchmark tasks are split into social recognition and social prediction problems with multi-label accuracy, macro and micro (F_1), AUROC, Hamming loss, log-likelihood, and goal inference accuracy [2103.01933].

The traffic and image-generation formulations again use domain-specific outcome metrics. SocialDriveGen reports Engagement Ratio, maximum relative velocity, maximum acceleration, and realized Extrinsic Reward, emphasizing interaction intensity and task completion rather than ADE or FDE [2512.01363]. The personalized-image paper evaluates HOI mAP on HICO-DET, ArcFace cosine similarity for identity preservation, gaze accuracy on GazeFollow, and CLIP-T, CLIP-I, and DINO for image quality, thereby treating social-context fidelity as a measurable image-generation outcome rather than an informal qualitative property [2507.16095].

5. Empirical findings and comparative patterns

Dream4Drive provides the clearest empirical case that SPDG can improve downstream perception under a fair protocol. At (256\times 512), the real-only detector at 1× epochs reaches mAP 34.5 and NDS 46.9, whereas Dream4Drive reaches mAP 36.1 and NDS 47.8; at 2× epochs the real-only baseline reaches mAP 38.4 and NDS 50.4, and Dream4Drive reaches mAP 38.7 and NDS 50.6. For tracking at the same resolution, 1× AMOTA is 30.1 for real-only and 31.2 for Dream4Drive, while AMOTP improves from 137.9 to 135.4; at 2×, AMOTA is 34.1 versus 34.4 and AMOTP 134.1 versus 133.5. At (512\times 768), detection improves from 36.1/47.9 to 40.7/52.0 at 1×, from 42.2/53.2 to 43.6/54.3 at 2×, and from 43.1/53.6 to 44.5/55.0 at 3×; tracking improves from 32.8/134.0 to 37.9/128.0 at 1×, from 39.7/125.1 to 42.6/123.3 at 2×, and from 41.3/124.1 to 43.5/121.3 at 3×. These gains are achieved with only 420 synthetic insertions over 28,130 real samples, that is, fewer than 2% synthetic samples. The paper further reports FVD 31.84 and FID 5.80 at (512\times 768), and a measurable but incomplete domain gap: a pre-trained StreamPETR reaches 82% of real-data detection performance on generated validation videos, with Gen-nuScenes mAP 24.4 and NDS 39.1 versus real mAP 36.1 and NDS 47.9. The ablations show left-side insertions outperform right-side ones, far insertions outperform close ones, and gains concentrate in large vehicles such as bus, construction_vehicle, and truck [2510.19195].

Persona Generators report that evolved generators consistently outperform all baselines across all six diversity metrics on held-out contexts. Coverage exceeds 80% on held-out test questionnaires, convex hull volume and mean pairwise distances improve steadily, dispersion decreases, KL divergence drops toward the quasi-random reference, and minimum pairwise distance, though noisier because of single-pair sensitivity, also improves over baselines. The reported explanation is structural: Stage 1 quasi-random axis sampling plus explicit niche targeting, together with evolutionary pressure on multiple metrics and mutations that introduce strong diversity instructions, contradictions, and legible rules and values [2602.03545].

LiveAgentBench demonstrates a different empirical consequence of SPDG: stronger benchmark difficulty and clearer measurement. The released benchmark contains 374 tasks across 104 scenarios, with 125 validation tasks and 249 test tasks. Human overall success is 69.25%; agents outperform base LLMs by approximately 56.51% on average, but the best reported product, Manus, still reaches only 35.29%. The failure analysis attributes errors frequently to tool instability and lack of environment background knowledge, especially when navigating unfamiliar sites or processing audio and video, which is consistent with the benchmark’s insistence on tool dependency and non-retrievability [2603.02586].

The social-robotics study shows that SPDG-style conditioning can alter human judgments in controlled ways without sacrificing believability. Valence conditioning affects perceived valence with (F(2,38)=13.5), (p<0.001), (\eta_g^2=0.16), and post hoc differences for Negative versus Neutral and Negative versus Positive; arousal conditioning affects perceived arousal with (F(1.5,28.42)=10.19), (p=0.001), (\eta_g^2=0.15), with significant Low versus High and Medium versus High contrasts. More positive valence or higher arousal also increases perceived dominance. At the same time, generated expressions are not perceived differently from hand-designed ones on anthropomorphism or animacy, indicating that the generative method preserves lifelikeness while still yielding controlled affective variation [2205.00763].

SocialDriveGen shows that SPDG can be used not only to label or augment data, but also to span a controlled interaction spectrum. In the ablation, Random Proposal yields Engagement Ratio 34.55, relative velocity 3.01 m/s, and acceleration 6.50 m/s(^2); Single Stage VLM yields 50.91, 3.65, and 7.25; terminal-step-only Diffusion-ES yields 60.00, 4.10, and 7.85; and the full method yields 67.27, 4.32, and 8.11. The social-preference sweeps show that (\phi=\pi/4) reduces high-stakes interactions relative to (\phi=-\pi/4), while decreasing (\lambda) from 1.0 to 0.3 raises realized extrinsic reward from 0.37 to 0.69 while maintaining high engagement [2512.01363].

The personalized-image results indicate that detector-based social feedback changes different aspects of image generation in different ways. On HICO-DET, the SSR-Encoder baseline reports full mAP 15.87 and identity cosine 0.6531, whereas the all-feedback model reaches mAP 16.43 and identity 0.6398; the HOI-only ablation gives the highest reported HICO-DET mAP at 17.84, while the gaze-only ablation gives the highest reported identity cosine at 0.6986. On GazeFollow, the baseline gaze accuracy is 53.81%, the all-feedback model reaches 53.73%, and the gaze-only ablation reaches 56.39%. On Concept101, the all-feedback model reports CLIP-I 0.6738 and DINO 0.3704, slightly improving over the baseline. The paper also reports that finetuning without social feedback can degrade performance, and that inverse timestep weighting causes severe degradation on HICO-DET, with mAP 14.09 [2507.16095].

6. Limitations, controversies, and future directions

The SPDG literature is unified by purpose but heterogeneous in ontology. This suggests that future work will need clearer taxonomies separating at least three cases: generation for human social perception, generation for machine social perception, and generation for safety-critical downstream perception whose failures have social consequences. The present papers already expose the resulting tensions. Dream4Drive, for example, explicitly states that its current version focuses on geometric and visual realism rather than multi-agent social interactions such as pedestrian intent, vehicle courtesy, or gap acceptance; it also notes that automatically keeping trajectories inside drivable areas and avoiding collisions is not yet solved, and proposes behavior priors, constrained optimization over generation parameters, active learning, and multi-objective SPDG as natural extensions [2510.19195].

Several papers identify a trade-off between long-tail coverage and plausibility. Persona Generators emphasize support coverage and therefore risk drift in the long tail; the paper notes bias amplification, stereotype reinforcement, harmful or implausible rare combinations, and misuse for bot creation or disinformation, and proposes safety filters, plausibility checks, human-in-the-loop review, and post-generation audits. It also states that synthetic personas should complement—not replace—human research participants [2602.03545]. The personalized-image paper reports an analogous problem at the detector level: errors in HOI or gaze estimation can misguide training, schedules are manually chosen rather than adaptive, and domain shift can reduce detector reliability [2507.16095].

Benchmark-oriented SPDG raises a different set of concerns. LiveAgentBench acknowledges that the current release is predominantly Chinese-language and may contain some unnatural details due to disambiguation. The paper also does not describe consent procedures, licensing of user content, anonymization, or formal bias assessments and metrics. Its recommended safeguards are explicitly presented as practical guidance rather than claims of the paper: respect platform terms of service, strip PII and usernames, track demographic and domain coverage, and audit for harmful tasks [2603.02586].

The simulation-based works have more conventional modeling limits. The robot body-language generator is trained on only 36 hand-designed animations, uses no temporal model, conditions directly only on valence while treating arousal as a latent-sampling effect, and excludes audio [2205.00763]. PHASE remains a 2D abstraction with limited agent types, simplified observation noise, and a persistent domain gap to real-world video despite its strong causal structure [2103.01933]. SocialDriveGen does not specify training architecture or hyperparameters for its diffusion prior, leaves broader realism metrics unreported, depends on VLM and LLM correctness for scene understanding and reward construction, and is exposed to reward-hacking risk during guided sampling [2512.01363].

A plausible implication is that the next phase of SPDG will emphasize closed-loop adaptation between generation and evaluation. That direction is already explicit in several of the papers: Dream4Drive proposes active learning and constrained parameter selection; Persona Generators optimize the generator itself rather than a fixed population; LiveAgentBench is designed for continuous refresh from live user interactions; and SocialDriveGen uses reward-guided denoising to steer sampling at every step. Taken together, these works suggest that SPDG is evolving from a data-augmentation heuristic into a broader methodology for designing synthetic data around the exact perceptual failures, social edge cases, and evaluation bottlenecks that matter in deployment.