VirtualCrowd: Synthetic Crowd Research

• VirtualCrowd is a research family that uses procedural simulation and rendering to create fully labeled synthetic crowd data in controllable virtual environments.
• It enables systematic variation in scene factors such as lighting, density, viewpoint, and behavior to improve training and benchmark crowd analytics.
• The approach supports robust evaluation of crowd understanding systems by scaling data generation and offering reproducible, high-quality ground truth.

VirtualCrowd denotes a line of research in which crowds are constructed, observed, or analyzed in synthetic environments so that motion, behavior, sensing conditions, and scene geometry can be controlled while labels remain exact. In the literature, the term appears both as a general paradigm—explicitly exemplified by LCrowdV’s “procedural virtual crowd dataset generation framework” for labeled crowd videos (Cheung et al., 2016)—and as the name of the synthetic benchmark introduced by DyCrowd for dynamic crowd reconstruction from large-scene video (Wen et al., 18 Aug 2025). Across these uses, the common principle is that virtualized crowds make it possible to scale data generation, vary scene factors systematically, evaluate spatio-temporal consistency under occlusion, and support downstream crowd-understanding systems that are difficult to train or benchmark from real footage alone.

1. Conceptual scope and research setting

The published usages of VirtualCrowd span several adjacent but technically distinct objectives. One branch uses virtual crowds to generate supervised data for perception and recognition; LCrowdV synthesizes crowd movements and renders them into labeled videos or images under varied environment, density, behavior, flow, lighting, viewpoint, and noise conditions (Cheung et al., 2016). A second branch uses virtual crowds as application-specific simulators; UAV-CROWD generates violent and non-violent aerial crowd-activity sequences from a UAV viewpoint, with automatically computed bounding boxes derived from semantic segmentation (Rahmun et al., 2022). A third branch uses virtual crowds as evaluation benchmarks for 3D scene understanding; VirtualCrowd, introduced by DyCrowd, was created because there had been no well-annotated video dataset suitable for evaluating spatio-temporally consistent 3D reconstruction of many people in a unified global scene (Wen et al., 18 Aug 2025).

A broader reading of the literature shows that VirtualCrowd-style systems are not confined to low-level pedestrian steering. Gen-C treats a crowd scene as a time-expanded crowd scenario graph and learns to generate high-level interactions such as waiting, talking, queuing, entering, exiting, reading, or meeting from text-conditioned latent variables (Panayiotou et al., 2 Apr 2025). Other work targets structured audience behavior such as queues, moshpits, and circlepits by extending BioCrowds rather than redesigning the simulator from scratch (Mello et al., 2023). This suggests that VirtualCrowd is best understood as a research family organized around controllable synthetic populations rather than as a single standardized framework.

2. Procedural synthesis and automatic annotation

The canonical VirtualCrowd-style data-generation workflow is procedural and two-stage. LCrowdV consists of a procedural simulation framework for generating crowd movements and behaviors and a procedural rendering framework for generating videos or images (Cheung et al., 2016). On the simulation side, the framework defines a scene or environment, specifies the number of pedestrians, assigns behaviors and movement flows, simulates pedestrian motion over time, and generates crowd formations and trajectories. On the rendering side, pedestrians are rendered into a scene while viewpoint, lighting, resolution, noise, and background environment are varied. Output can be synthetic-only or a hybrid in which synthetically generated behaviors are combined with real-world background videos. Each video or image is automatically labeled by environment, number of pedestrians, density, behavior, flow, lighting conditions, viewpoint, noise, and related attributes. The practical significance is that every synthetic sample comes with accurate labels automatically, avoiding the cost and limited coverage of manual annotation.

LCrowdV was introduced for crowded-scene understanding tasks including pedestrian detection, crowd behavior classification, crowd counting, tracking, abnormal behavior detection, and crowd segmentation (Cheung et al., 2016). The reported benefits are improved pedestrian detection accuracy and considerable improvements in crowd behavior classification, with the conclusion emphasizing that a main benefit came from generating videos from a different camera angle. That result ties synthetic data quality not only to realism but also to coverage of underrepresented viewing conditions.

UAV-CROWD instantiates the same synthetic-data logic for aerial surveillance. It is built in Unreal Engine and uses the Airsim plugin for UAV flight and image capture, generating photo-realistic synthetic RGB videos together with segmentation and depth ground truth (Rahmun et al., 2022). Crowd NPCs are assigned a fixed segmentation color, RGB $(81, 13, 36)$, in segmentation images; those maps are then processed to obtain ground-truth bounding boxes for the corresponding RGB frames. The simulator supports 150 concurrent crowd agents interacting at once and maintains an average of 25 frames per second on a mid-range computer with an Intel Core i5-6500 @ 3.20 GHz, RX 470 with 4GB VRAM, and 16GB RAM. The synthetic dataset contains 240 video sequences split equally into violent and non-violent categories; all videos are normalized to $640 \times 480$, 30 FPS, and maximum length of 10 seconds. When augmented with real-world data, the synthetic sequences improve binary video classification accuracy by 5% on average across two different models, with gains ranging from 2.1% to 8.2% (Rahmun et al., 2022).

3. High-level behavior authoring and structured social patterns

A major development within VirtualCrowd research is the shift from low-level locomotion toward high-level scenario authoring. Gen-C addresses the limitation that most crowd simulation methods focus on low-level motion control such as steering, flocking, goal seeking, or collision avoidance, while richer crowd scenes require high-level interactions between agents, their environment, and each other over time (Panayiotou et al., 2 Apr 2025). Its core representation is a time-expanded crowd scenario graph. Starting from a single input sentence $S_{in}$, GPT-4o is queried four times in sequence: first to expand the crowd situation and social context, second to define an environment layout with locations and their approximate size, position, and orientation, third to assign plausible high-level actions from a predefined action vocabulary with probabilities, and fourth to provide movement probabilities between locations. Unity then instantiates the environment, spawns agents individually or in small groups of two or three, and records full action sequences. These structured records are converted into graphs whose nodes represent one agent at one timestep and whose edge types are sequence edges and share edges. Gen-C learns separate latent spaces for graph structure and node features using two conditional Variational Graph Auto-Encoders, VGAE-S and VGAE-F, with GINE-based encoders and decoders.

The modeling decision to separate graph structure from node features is central. VGAE-S learns how nodes are connected through temporal sequence and shared interaction relations, while VGAE-F learns the action and location labels attached to those nodes (Panayiotou et al., 2 Apr 2025). The condition prior network encodes the text description together with the number of agents, making generation context-aware at both inference and decoding time. The resulting system can condition on natural language and populate environments such as a University Campus or a Train Station with dynamic interactions that remain aligned with prompt semantics.

A different route to high-level structure is to extend an emergent locomotion model with explicit control rules. The BioCrowds extension for structured behaviors retains the original marker-based, collision-free movement model—markers are generated by the Dart Throwing algorithm, associated with the closest agent, and weighted according to goal direction—but introduces new parameters and a repulsion rule to reproduce a generic queue, a moshpit, and a circlepit (Mello et al., 2023). The key moshpit modification replaces the standard BioCrowds weight with “1 minus the original weight,” causing agents to move away from a central goal and opening a space in the crowd. Additional parameters include NumberAgentsPit, Area of Effect Radius, Moshpit Center Goal, Circlepit Goal List, Reflect Threshold (Min/Max), and Time To Start. The queue experiments show that simple goal-following does not produce a convincing queue, whereas spatial restriction, lower marker density, larger capture radius, and intermediate goals produce more organized behavior. The moshpit and circlepit simulations were judged primarily by visual plausibility and whether the expected structure emerged and later dissolved.

4. Continuous, probabilistic, and field-based motion generation

Another major strand of VirtualCrowd research models uncertainty directly rather than treating the crowd as a deterministic field plus local collision avoidance. The anisotropic-fields approach introduces a three-layer crowd model consisting of global path planning, an anisotropic layer, and local navigation (Li et al., 2024). Instead of assigning a single direction to each cell, an anisotropic field stores a probability distribution over directions $\theta \in [0^\circ, 360^\circ)$. These fields can be authored by sketching or extracted from real crowd videos via optical flow using the classical constraint

$I_xu + I_yv + I_t = 0.$

During simulation, agents sample a motion tendency from the local directional distribution, and behavior inertia modifies the sampling density according to

$$p'(\theta)=p(\theta)\,|\cos(\theta-\theta_v)|^\delta,$$

which suppresses abrupt frame-to-frame directional oscillation.

The practical consequence is increased behavioral diversity within the same scene. In the multi-intersection channel experiment with 400 agents over 2 minutes on a $10\text{ cm} \times 10\text{ cm}$ grid, the “Free move” criterion produced 218 collisions, velocity variance 0.0177, and 1212 intersection passes, whereas “Right-keeping move” produced 87 collisions, velocity variance 0.0080, and 1441 intersection passes (Li et al., 2024). In data-driven simulations derived from MOT20 videos, AF-based systems reduced the similarity score $D$ to below 0.2 after 180 iterations, more than 50% lower than the initial value, and scene construction required 20 seconds automatically rather than 1500–1800 seconds manually. The method remains macroscopic and approximate, but it provides a probabilistic middle layer between designer intent and local navigation.

Continuous crowd generation pushes the same idea into long-horizon population dynamics. CrowdES alternates a crowd emitter and a crowd simulator over time windows $T_w$, so that new agents are continually introduced and existing agents continue to move (Bae et al., 7 Apr 2025). From a single input image $\mathcal{I}$, the system derives a semantic segmentation map $640 \times 480$0, an appearance map $640 \times 480$1, a population density map $640 \times 480$2, and a population probability $640 \times 480$3. The diffusion-based emitter jointly generates each agent’s type $640 \times 480$4, walking pace $640 \times 480$5, start time $640 \times 480$6, start position $640 \times 480$7, and destination position $640 \times 480$8. The simulator then uses a navigation mesh and a Markov chain over $640 \times 480$9 behavior states to generate trajectory segments recurrently, allowing stopping, detouring, and other intermediate behavior changes. Default settings include $S_{in}$0 frames at 5 fps, diffusion steps $S_{in}$1, $S_{in}$2, and $S_{in}$3. The benchmark protocol evaluates scene-level density, frequency, coverage, and population similarity, together with kinematics, DTW, diversity, and collision rate. CrowdES generally performs best across almost all metrics and datasets, although it can have slightly higher collision rates than SE-ORCA because it prioritizes richer behavioral switching (Bae et al., 7 Apr 2025).

5. Forecasting, crowd-flow representation, and crowd-aware control

VirtualCrowd research also encompasses predictive models that operate on aggregate crowd states rather than explicit agent simulations. “Over-crowdedness Alert! Forecasting the Future Crowd Distribution” formulates the task of predicting a future crowd density map from sparsely sampled video frames without identity annotations (Niu et al., 2020). The model is a global-residual two-stream recurrent network composed of a frame-to-density stream, F2D-Net, and a density-to-density stream, D2D-Net, fused by channel-wise and spatial attention and refined by an optical-flow-based global residual branch. Scene-specific synthetic density maps, generated using social-force-type simulation in world space, projected by homography, and converted with a geometry-adaptive kernel, are used to pretrain D2D-Net. The framework was evaluated on UCSD, Mall, and Station; on Mall at $S_{in}$4, the reported improvements over the second-best method are 6.6% for $S_{in}$5 sec and 7.8% for $S_{in}$6 sec (Niu et al., 2020). The output is not a trajectory set but a density forecast that supports future crowd count estimation and high-density region alerting.

A related but distinct formulation predicts long-term crowd flow in built environments in one shot. “Deep Crowd-Flow Prediction in Built Environments” encodes a scenario with the CAGE representation,

$S_{in}$7

where the two capacity channels record compression along the two axes, $S_{in}$8 encodes agent density, $S_{in}$9 is the distance-transform goal map, and $\theta \in [0^\circ, 360^\circ)$0 stores the navigable environment (Sohn et al., 2019). This representation is designed to be environmentally lossless under compression, so arbitrarily large built environments can be mapped into a fixed-size tensor, passed through a modified SegNet, and decoded back to the original scale. The predicted output is the average crowd density over an entire simulation for each navigable cell. The approach is one-shot rather than frame-by-frame, and it is aimed at large realistic environments and never-before-seen crowd scenarios.

In robotics, the VirtualCrowd idea becomes a real-time macroscopic predictor embedded in planning. “A Lightweight Crowd Model for Robot Social Navigation” models the crowd as a $\theta \in [0^\circ, 360^\circ)$1 grid field

$\theta \in [0^\circ, 360^\circ)$2

with $\theta \in [0^\circ, 360^\circ)$3 cell features: density $\theta \in [0^\circ, 360^\circ)$4, $\theta \in [0^\circ, 360^\circ)$5-velocity $\theta \in [0^\circ, 360^\circ)$6, $\theta \in [0^\circ, 360^\circ)$7-velocity $\theta \in [0^\circ, 360^\circ)$8, and isotropic velocity variance $\theta \in [0^\circ, 360^\circ)$9 (Eskeri et al., 27 Aug 2025). The experiments use a $I_xu + I_yv + I_t = 0.$0 grid with $I_xu + I_yv + I_t = 0.$1 cells, input history 10 frames, and prediction horizon 10 frames. The model is a simplified encoder–forecaster ConvRNN with two strided convolutional layers, one ConvRNN layer, and transposed-convolution decoding, trained with density-weighted smooth $I_xu + I_yv + I_t = 0.$2. It is integrated into a socially aware planner through the invasiveness metric

$I_xu + I_yv + I_t = 0.$3

The reported results show a 3.6 times reduction in inference time, a 3.1% improvement in prediction accuracy, and the best non-omniscient planning cost for the “Proposed (online)” mode, with actual cost $I_xu + I_yv + I_t = 0.$4 (Eskeri et al., 27 Aug 2025).

6. Large-scene reconstruction and the VirtualCrowd benchmark

VirtualCrowd also names a benchmark for reconstructing dynamic crowds in 3D. DyCrowd formulates the task as recovering, for each person $I_xu + I_yv + I_t = 0.$5 over frames $I_xu + I_yv + I_t = 0.$6, a temporally coherent sequence of global SMPL parameters

$I_xu + I_yv + I_t = 0.$7

where $I_xu + I_yv + I_t = 0.$8 are pose parameters, $I_xu + I_yv + I_t = 0.$9 shape parameters, $$p'(\theta)=p(\theta)\,|\cos(\theta-\theta_v)|^\delta,$$0 global rotation, and $$p'(\theta)=p(\theta)\,|\cos(\theta-\theta_v)|^\delta,$$1 global translation (Wen et al., 18 Aug 2025). The SMPL model is used to map these parameters to mesh vertices and joints. VirtualCrowd was introduced because prior large-scene crowd methods such as Crowd3D and GroupRec handled static images and therefore could not evaluate temporal consistency or long-term occlusion recovery. The dataset was generated in Blender using Blender’s iCity plugin for scene construction, ground plane extraction to generate human motion trajectories, SynBody human models, DIMOS motion generation, and Blender rendering; it was extended with two additional videos of a sloped scene generated using simulation techniques from RESCUE.

The dataset contains 4 scenes, each covering more than 2500 square meters, with 2 camera configurations per scene—a high-angle view and a low-angle view—for a total of 8 validation videos (Wen et al., 18 Aug 2025). Each video is $$p'(\theta)=p(\theta)\,|\cos(\theta-\theta_v)|^\delta,$$2 at 30 fps, total length 1600 frames, and crowd density varies from 60 to 200 people per scene. The full benchmark contains 931 motion sequences and 186,200 poses. Annotation includes 2D joints, MOT annotations, 3D joints, 3D positions, and SMPL-X parameters. Evaluation uses PA-PPDS and PCOD for global arrangement and depth ordering, MPJPE and PA-MPJPE for pose accuracy, WA-MPJPE and W-MPJPE for sequence accuracy, and ACCEL for motion smoothness. DyCrowd achieves the best overall benchmark numbers among the reported methods, including PA-PPDS 89.10, PCOD 92.20, MPJPE 69.74, and PA-MPJPE 48.57, while the variant with ground-truth object tracking reaches PA-PPDS 91.23 and PCOD 95.38 (Wen et al., 18 Aug 2025). Ablations show that removing the coarse-to-fine group-guided motion optimization degrades performance substantially, especially under occlusion.

This dynamic benchmark builds directly on the static large-scene reconstruction problem introduced by Crowd3D. Crowd3D is the first framework to reconstruct the 3D poses, shapes, and locations of hundreds of people with global consistency from a single large-scene image (Wen et al., 2023). Its core geometric concept is the Human-scene Virtual Interaction Point (HVIP), defined as the projection point of a person’s 3D torso center onto the ground plane in the global camera space. Together with scene-level camera and ground-plane estimation and adaptive human-centric cropping, HVIP converts difficult absolute 3D localization into a more tractable pixel-localization problem. Crowd3D also contributes the LargeCrowd dataset: 733 gigapixel images at resolution $$p'(\theta)=p(\theta)\,|\cos(\theta-\theta_v)|^\delta,$$3, over 100K labeled humans, and 9 scenes total. On LargeCrowd, Crowd3D reaches PPDS 81.53, PA-PPDS 89.36, PCOD 92.63, and OKS 71.72, outperforming the modified baselines reported in the paper (Wen et al., 2023). In historical terms, Crowd3D establishes the static large-scene setting, while VirtualCrowd extends evaluation into the dynamic, temporally consistent regime.

7. Applications, limitations, and evolving directions

The application range of VirtualCrowd-style systems is broad. LCrowdV targets pedestrian detection, crowd behavior classification, crowd counting, tracking, abnormal behavior detection, and crowd segmentation (Cheung et al., 2016). UAV-CROWD specializes in violent and non-violent crowd surveillance from UAV viewpoints, where dangerous behaviors are difficult or unsafe to record directly (Rahmun et al., 2022). Gen-C and CrowdES are oriented toward populating virtual environments, games, simulations, VR, and other interactive spaces with heterogeneous, controllable agents (Panayiotou et al., 2 Apr 2025, Bae et al., 7 Apr 2025). The anisotropic-fields framework emphasizes games, film/VFX, urban planning, VR, and replication of real crowd behavior from video (Li et al., 2024). The BioCrowds extension is explicitly motivated by entertainment settings such as rock concerts and virtual audiences (Mello et al., 2023). DyCrowd and Crowd3D address city surveillance, crowd analysis, security alert, and scene-level spatial reasoning (Wen et al., 18 Aug 2025, Wen et al., 2023). A plausible implication is that VirtualCrowd has become an interface layer between simulation, perception, and decision-making rather than a narrowly defined dataset technology.

Several recurring limitations also appear across the literature. Synthetic data is consistently presented as complementary to real data rather than a replacement for it; both LCrowdV and UAV-CROWD report gains when synthetic data augments real-world training sets (Cheung et al., 2016, Rahmun et al., 2022). Some systems retain explicit realism constraints: UAV-CROWD states that the goal is not perfect realism in every detail but to simulate crowd behavior as closely as possible (Rahmun et al., 2022). Gen-C limits actions to a fixed predefined list, uses probabilistic action sequences rather than optimal or learned high-level planning, does not allow agents to switch actions mid-execution, and manually defines action durations (Panayiotou et al., 2 Apr 2025). The anisotropic-fields method states that AFs are approximations, macroscopic rather than individual-cognitive models, and not a full substitute for modeling personality or decision-making (Li et al., 2024). The BioCrowds moshpit is described as a positional approximation rather than a true physical interaction model, and its validation is qualitative rather than metric-driven (Mello et al., 2023). CrowdES reports a tradeoff in which richer behavioral switching can slightly increase collision rates relative to SE-ORCA (Bae et al., 7 Apr 2025).

An additional extension of the VirtualCrowd idea moves from geometric crowds to networked virtual populations. The framework “Crowd: A Social Network Simulation Framework” represents agents on graphs, supports YAML-based setup, no-code diffusion simulations, interactive visualization, batch runs, parameter sweeps, and generative agents integrated with Python ML tooling (Rende et al., 2024). Its case studies—generative agents in epidemics, influence maximization, and networked trust games—show that virtual crowd research is broadening from physical movement to synthetic communities whose local interaction rules produce macro-level diffusion and collective behavior. This suggests an increasingly hybrid future in which virtual crowds are not only rendered and reconstructed, but also authored through language, optimized for occlusion-robust 3D recovery, forecast as macroscopic fields, and embedded in socially aware autonomous systems.