FantasyHSI: Graph-based 4D Human Synthesis

Updated 4 July 2026

FantasyHSI is a video-generation-centric framework for 4D human synthesis that leverages symbolic graph planning and multi-agent control for long-horizon human–scene interactions.
It employs a dynamic directed graph where nodes represent human–scene states and edges denote atomic action units, enabling efficient planning and corrective feedback.
The system integrates an image-to-video diffusion model, 3D motion lifting via WHAC, and Direct Preference Optimization to improve physical realism and task adherence.

FantasyHSI denotes a video-generation-centric framework for 4D human synthesis in arbitrary 3D scenes from high-level natural language instructions, designed for human–scene interaction (HSI) without requiring paired human–scene motion capture data. It models long-horizon interaction as a dynamic directed graph whose nodes are human–scene states and whose edges are atomic action units, and it realizes that graph through a VLM-based multi-agent system, a video diffusion action generator, and a critic agent that corrects generative drift to preserve long-term logical consistency (Mu et al., 1 Sep 2025). In the current literature, the same label also appears in a separate fantasy-sports human–system-interface context; the named framework, however, refers specifically to graph-based 4D HSI generation (Martinez et al., 23 Aug 2025, Mu et al., 1 Sep 2025).

1. Scope and problem setting

FantasyHSI is situated within HSI and 4D human generation, with an explicit focus on two persistent difficulties: handling long-horizon, high-level tasks and generalizing to unseen scenes. The framework addresses these issues by combining symbolic planning over a graph with generative video synthesis, rather than learning directly from synchronized human–scene trajectory pairs (Mu et al., 1 Sep 2025).

The system is explicitly video-generation-centric. Instead of training a 3D motion model directly on paired scene-interaction data, it generates short action clips with an image-to-video diffusion model, lifts the generated video back into 3D through motion capture, and iteratively updates the scene state. This makes the video model the primary execution engine, while the graph and agents provide planning and control (Mu et al., 1 Sep 2025).

The formulation starts from a 3D human state

$\mathcal{H} = F(R, T, \theta, \beta),$

where $R$ is global rotation, $T$ is global translation, $\theta$ denotes pose parameters, $\beta$ denotes body shape parameters, and $F$ is the SMPL-X rendering function generating a 3D human mesh. The scene state $\mathcal{S}$ is a 3D mesh of the environment. A node is then defined as

$N_k = \{\mathcal{S}_k, \mathcal{H}_k\},$

representing the coupled human–scene state at the start or end of an action unit (Mu et al., 1 Sep 2025).

A central design choice is that FantasyHSI places graph nodes only at the start and end of action units, not at every frame. This avoids the combinatorial explosion that would result from frame-level graph construction, while retaining sufficient structure for long-horizon planning and correction (Mu et al., 1 Sep 2025).

2. Graph formulation of human–scene interaction

FantasyHSI represents interaction as a directed graph

$G = (\mathcal{N}, E),$

with node set $\mathcal{N}$ and edge set $R$ 0. Two node subsets are distinguished: key nodes

$R$ 1

which encode milestones or sub-goals, and non-key nodes

$R$ 2

which capture intermediate states introduced during action decomposition and execution (Mu et al., 1 Sep 2025).

A directed edge corresponds to an atomic action unit:

$R$ 3

with the associated state transition

$R$ 4

In this representation, an action unit has a complete semantic meaning, such as walking toward an object, climbing a portion of a ladder, or sitting on a windowsill, and each unit lasts up to approximately three seconds (Mu et al., 1 Sep 2025).

The graph is dynamic rather than static. Initially, the system knows only a sequence of planned key nodes. The planning agent expands the segments between adjacent key nodes into chains of intermediate nodes and edges. During execution, however, the stochastic video generator may introduce unplanned behaviors, such as moving farther than intended or producing additional motions. The critic agent then evaluates those deviations, prunes unsuitable portions of the path, backtracks to a more acceptable node, and updates the future plan. The graph therefore functions simultaneously as a high-level task scaffold and as an editable execution trace (Mu et al., 1 Sep 2025).

FantasyHSI does not use an explicit message-passing GNN. The graph is processed through explicit VLM-based reasoning over rendered scene views, planned sub-goals, and generated motion segments. This makes the graph a symbolic state abstraction rather than a learned graph-embedding structure (Mu et al., 1 Sep 2025).

3. Multi-agent planning, execution, and correction

FantasyHSI uses a VLM-based multi-agent system, implemented with Gemini-2.5-Pro, organized around four components: a Scene Navigation Agent, an Action-Chain Planner Agent, an Action Generator, and a Critic Agent (Mu et al., 1 Sep 2025).

The Scene Navigation Agent receives the high-level task description, the initial human state, and the initial scene state. It analyzes multiple rendered views of the 3D scene, identifies navigable and non-traversable regions, recognizes interactive objects, infers intent from the task description, and produces a sequence of sub-goals and a global path. Its output is the initial backbone of key nodes $R$ 5 for the interaction graph (Mu et al., 1 Sep 2025).

The Action-Chain Planner Agent refines each segment between adjacent key nodes into a sequence of atomic natural-language action descriptions. These action units may be simple or composite, but each is treated as a minimal semantic motion unit suitable for short-horizon execution. This decomposition bridges symbolic path planning and low-level motor realization by producing the edge descriptions $R$ 6 that will later condition the video generator (Mu et al., 1 Sep 2025).

The Action Generator instantiates those edges. Given a rendered first frame showing the current human pose in the current 3D scene and a text description of the next action unit, it synthesizes a short video clip and then lifts that clip into 3D motion. In the overall loop, it functions as the simulator that realizes graph edges and advances the system from node to node (Mu et al., 1 Sep 2025).

The Critic Agent provides the closed-loop correction mechanism that distinguishes FantasyHSI from a purely open-loop text-to-video pipeline. For each generated segment, it evaluates pose, distance to target, orientation, and trajectory adherence. It then backtracks temporally through the generated frames to find the latest acceptable frame, prunes the remainder, adjusts orientation when necessary, selects a new camera pose for the next conditioning frame, and modifies the upcoming action sequence so that execution returns to the intended route (Mu et al., 1 Sep 2025).

This architecture implements an iterative control loop:

Plan → Decompose → Generate action → Evaluate/Correct → Update plan → Next action

The paper attributes FantasyHSI’s long-horizon coherence substantially to this loop. In the ablation study, removing the critic agent lowers task alignment and obstacle-response metrics, while removing the multi-agent planning structure sharply reduces performance on complex multi-step tasks (Mu et al., 1 Sep 2025).

4. Video diffusion, motion lifting, and preference optimization

The action generator is built on Wan2.1-I2V-14B, an image-to-video diffusion model. Conditioning is bimodal: the current 3D scene and pose are supplied as a rendered initial frame, while the intended action is supplied as a text prompt. At inference time, FantasyHSI uses 30 diffusion inference steps and classifier-free guidance with guidance scale $R$ 7 (Mu et al., 1 Sep 2025).

Generated clips are converted back into 3D with WHAC motion capture. The recovered SMPL-X motion sequence is then applied in the original 3D scene, updating the human state and producing the next graph node. The full 4D interaction is thus constructed as a sequence of generated clips and lifted state transitions rather than as a monolithic trajectory synthesized in one pass (Mu et al., 1 Sep 2025).

To improve physical realism and instruction following, the authors fine-tune the video model using Direct Preference Optimization (DPO). The preference dataset is constructed from videos generated by four proprietary systems—VEO, HunYuan-Video, Runway, and Kling—which professional annotators label as positive or negative according to instruction-following quality, motion artifacts, limb deformation, and scene inconsistency. The resulting dataset contains 10,000 preference pairs $R$ 8 (Mu et al., 1 Sep 2025).

Training starts from the pre-trained Wan2.1-I2V-14B model and applies DiffusionDPO-style optimization with temperature coefficient $R$ 9, learning rate $T$ 0, and approximately 20 hours of training on 8× A100 GPUs. The paper also reports SFT as a comparison point, but the final FantasyHSI system uses DPO rather than positive-only supervised fine-tuning (Mu et al., 1 Sep 2025).

Empirically, DPO reduces limb distortions, foot sliding, and penetration into scene geometry, while improving CLIP-S and obstacle-response metrics relative to both the pre-trained baseline and the SFT variant. The paper treats this as evidence that preference alignment is effective not only for perceptual quality but also for physically grounded scene interaction (Mu et al., 1 Sep 2025).

5. SceneBench and empirical performance

FantasyHSI is evaluated on SceneBench, a custom benchmark comprising 20 distinct 3D scenes: 10 indoor scenes such as bedrooms, cowsheds, and gyms, and 10 outdoor scenes such as grasslands, riversides, urban streets, and rural farms. The benchmark includes 120 text–scene–position test instances for static-environment evaluation (Mu et al., 1 Sep 2025).

A second evaluation regime tests scene perception and response by inserting obstacles on pre-planned paths. These obstacles include 15 seen objects from TRUMANS, such as chairs, sofas, and vases, and 15 novel objects from the internet, such as pumpkins and rocks. This setting measures not only whether a model collides less, but whether it actually reacts to obstacles through path or motion changes (Mu et al., 1 Sep 2025).

The benchmark uses a set of geometric and semantic metrics. For scene interaction, the reported metrics are Penetration Score (P-Score), Foot Sliding Score (FS), CLIP-Score (CLIP-S), CLIP-Consistency (CLIP-C), and Diversity. For obstacle response, the reported metrics are Penetration Obstacle Score (POS) and Reaction Divergence Score (RDS) (Mu et al., 1 Sep 2025).

FantasyHSI is compared against TRUMANS, LINGO, and PedGen. The comparison is notable because those baselines receive more structured inputs: TRUMANS is provided with a full trajectory, while LINGO and PedGen are given start and goal positions. FantasyHSI, by contrast, plans from high-level text via its agents (Mu et al., 1 Sep 2025).

On the main benchmark table, FantasyHSI reports P-Score $T$ 1, FS $T$ 2, CLIP-S $T$ 3, CLIP-C $T$ 4, Diversity $T$ 5, POS $T$ 6, and RDS $T$ 7. The corresponding ranges for the three main baselines are P-Score $T$ 8– $T$ 9, FS $\theta$ 0– $\theta$ 1, CLIP-S $\theta$ 2– $\theta$ 3, Diversity $\theta$ 4– $\theta$ 5, POS $\theta$ 6– $\theta$ 7, and RDS $\theta$ 8– $\theta$ 9 (Mu et al., 1 Sep 2025).

The ablation results further isolate the contribution of the main components. The SFT variant yields P-Score $\beta$ 0, FS $\beta$ 1, and CLIP-S $\beta$ 2, while the Baseline pre-trained model yields P-Score $\beta$ 3, FS $\beta$ 4, and CLIP-S $\beta$ 5. Removing multi-agent planning lowers CLIP-S from 0.31 to 0.24, and removing the critic agent lowers it to 0.23 while worsening POS and RDS. The paper interprets these results as showing that DPO improves physical realism, multi-agent decomposition improves long-horizon task execution, and the critic preserves task consistency under generative stochasticity (Mu et al., 1 Sep 2025).

Qualitatively, FantasyHSI is reported to handle tasks such as climbing a roughly 20 m ladder to a rooftop, sitting on a windowsill, and reacting to inserted pumpkins by stepping over them rather than passing through them. The baselines are described as weaker at novel-object interaction, precise geometry handling, or rich non-locomotor behavior (Mu et al., 1 Sep 2025).

The term “FantasyHSI” is not terminologically uniform across the supplied literature. In SPORTSQL, it appears as a fantasy sports human–system interface perspective, where the system serves as a reference design for a conversational analysis layer, a real-time decision-support backend over live FPL data, and automatic visualization of sports statistics (Martinez et al., 23 Aug 2025). In daily fantasy sports research, related interface goals are coupled to lineup portfolio construction under top-heavy payoff structures through integer programming, with an emphasis on high expected score, high variance, and low cross-lineup correlation (Hunter et al., 2016). In fantasy basketball, dynamic valuation is formalized through H-scoring and the $\beta$ 6 implementation, which adapt category weights, positional assignments, and format-specific objectives during a head-to-head draft (Rosenof, 2024). In fantasy cricket, FanCric uses a LangGraph-based multi-agent framework with agents such as Researcher, Career Profiler, Form Assessor, Strategizer, Selector, and Evaluator to construct Dream11 teams from structured and unstructured data (Bhatnagar, 2024).

This suggests that the label has acquired two distinct meanings in current usage: a specific graph-based 4D HSI generator in computer vision and graphics, and a broader fantasy-sports human–system-interface perspective in sports analytics. The two usages are conceptually separate, even though both emphasize multi-agent reasoning, structured state representations, and decision support (Bhatnagar, 2024, Martinez et al., 23 Aug 2025, Mu et al., 1 Sep 2025).

Within the computer-vision sense of the term, the stated limitations are direct. FantasyHSI remains computationally heavy because diffusion-based video generation and VLM-based multi-agent reasoning are expensive at inference time, which hinders real-time or highly interactive deployment. It also focuses on static 3D scenes, because the reported benchmark and data regime do not cover environments with moving obstacles or changing layouts (Mu et al., 1 Sep 2025).

The paper therefore positions future work around two axes: improving efficiency for real-time interaction and extending the framework to dynamic environments. In its present form, FantasyHSI is best understood as an overview of symbolic graph planning, VLM-based multi-agent control, preference-aligned video diffusion, and video-to-3D motion lifting for long-horizon HSI in previously unseen scenes (Mu et al., 1 Sep 2025).