---
title: 'FantasyHSI: Graph-based 4D Human Synthesis'
url: https://www.emergentmind.com/topics/fantasyhsi
type: topic
---

# FantasyHSI: Graph-based 4D Human Synthesis

FantasyHSI denotes a **video-generation-centric framework for 4D human synthesis** in arbitrary 3D scenes from **high-level natural language instructions**, designed for **human–scene interaction (HSI)** without requiring paired human–scene motion capture data. It models long-horizon interaction as a **dynamic directed graph** whose nodes are human–scene states and whose edges are **atomic action units**, and it realizes that graph through a **VLM-based multi-agent system**, a **video diffusion action generator**, and a **critic agent** that corrects generative drift to preserve long-term logical consistency [2509.01232]. In the current literature, the same label also appears in a separate fantasy-sports human–system-interface context; the named framework, however, refers specifically to graph-based 4D HSI generation [2508.17157][2509.01232].

## 1. Scope and problem setting

FantasyHSI is situated within HSI and 4D human generation, with an explicit focus on two persistent difficulties: **handling long-horizon, high-level tasks** and **generalizing to unseen scenes**. The framework addresses these issues by combining symbolic planning over a graph with generative video synthesis, rather than learning directly from synchronized human–scene trajectory pairs [2509.01232].

The system is explicitly **video-generation-centric**. Instead of training a 3D motion model directly on paired scene-interaction data, it generates short action clips with an image-to-video diffusion model, lifts the generated video back into 3D through motion capture, and iteratively updates the scene state. This makes the video model the primary execution engine, while the graph and agents provide planning and control [2509.01232].

The formulation starts from a 3D human state
$$
\mathcal{H} = F(R, T, \theta, \beta),
$$
where \(R\) is global rotation, \(T\) is global translation, \(\theta\) denotes pose parameters, \(\beta\) denotes body shape parameters, and \(F\) is the SMPL-X rendering function generating a 3D human mesh. The scene state \(\mathcal{S}\) is a 3D mesh of the environment. A node is then defined as
$$
N_k = \{\mathcal{S}_k, \mathcal{H}_k\},
$$
representing the coupled human–scene state at the start or end of an action unit [2509.01232].

A central design choice is that FantasyHSI places graph nodes only at the **start and end of action units**, not at every frame. This avoids the combinatorial explosion that would result from frame-level graph construction, while retaining sufficient structure for long-horizon planning and correction [2509.01232].

## 2. Graph formulation of human–scene interaction

FantasyHSI represents interaction as a directed graph
$$
G = (\mathcal{N}, E),
$$
with node set \(\mathcal{N}\) and edge set \(E\). Two node subsets are distinguished: **key nodes**
$$
\mathcal{K} = \{K_1, K_2, ..., K_p\} \subset \mathcal{N},
$$
which encode milestones or sub-goals, and **non-key nodes**
$$
\mathcal{U} = \{U_1, U_2, ..., U_q\} = \mathcal{N} \setminus \mathcal{K},
$$
which capture intermediate states introduced during action decomposition and execution [2509.01232].

A directed edge corresponds to an atomic action unit:
$$
E(N_i, N_j) = A_{i,j},
$$
with the associated state transition
$$
\mathcal{H}_j = \mathcal{H}_i + A_{i,j}.
$$
In this representation, an action unit has a complete semantic meaning, such as walking toward an object, climbing a portion of a ladder, or sitting on a windowsill, and each unit lasts up to approximately three seconds [2509.01232].

The graph is **dynamic** rather than static. Initially, the system knows only a sequence of planned **key nodes**. The planning agent expands the segments between adjacent key nodes into chains of intermediate nodes and edges. During execution, however, the stochastic video generator may introduce unplanned behaviors, such as moving farther than intended or producing additional motions. The critic agent then evaluates those deviations, prunes unsuitable portions of the path, backtracks to a more acceptable node, and updates the future plan. The graph therefore functions simultaneously as a high-level task scaffold and as an editable execution trace [2509.01232].

FantasyHSI does not use an explicit message-passing GNN. The graph is processed through explicit VLM-based reasoning over rendered scene views, planned sub-goals, and generated motion segments. This makes the graph a symbolic state abstraction rather than a learned graph-embedding structure [2509.01232].

## 3. Multi-agent planning, execution, and correction

FantasyHSI uses a **VLM-based multi-agent system**, implemented with **Gemini-2.5-Pro**, organized around four components: a **Scene Navigation Agent**, an **Action-Chain Planner Agent**, an **Action Generator**, and a **Critic Agent** [2509.01232].

The **Scene Navigation Agent** receives the high-level task description, the initial human state, and the initial scene state. It analyzes multiple rendered views of the 3D scene, identifies navigable and non-traversable regions, recognizes interactive objects, infers intent from the task description, and produces a sequence of sub-goals and a global path. Its output is the initial backbone of key nodes \(\mathcal{K}\) for the interaction graph [2509.01232].

The **Action-Chain Planner Agent** refines each segment between adjacent key nodes into a sequence of atomic natural-language action descriptions. These action units may be simple or composite, but each is treated as a minimal semantic motion unit suitable for short-horizon execution. This decomposition bridges symbolic path planning and low-level motor realization by producing the edge descriptions \(A_{i,j}\) that will later condition the video generator [2509.01232].

The **Action Generator** instantiates those edges. Given a rendered first frame showing the current human pose in the current 3D scene and a text description of the next action unit, it synthesizes a short video clip and then lifts that clip into 3D motion. In the overall loop, it functions as the simulator that realizes graph edges and advances the system from node to node [2509.01232].

The **Critic Agent** provides the closed-loop correction mechanism that distinguishes FantasyHSI from a purely open-loop text-to-video pipeline. For each generated segment, it evaluates pose, distance to target, orientation, and trajectory adherence. It then backtracks temporally through the generated frames to find the latest acceptable frame, prunes the remainder, adjusts orientation when necessary, selects a new camera pose for the next conditioning frame, and modifies the upcoming action sequence so that execution returns to the intended route [2509.01232].

This architecture implements an iterative control loop:
> Plan → Decompose → Generate action → Evaluate/Correct → Update plan → Next action

The paper attributes FantasyHSI’s long-horizon coherence substantially to this loop. In the ablation study, removing the critic agent lowers task alignment and obstacle-response metrics, while removing the multi-agent planning structure sharply reduces performance on complex multi-step tasks [2509.01232].

## 4. Video diffusion, motion lifting, and preference optimization

The action generator is built on **Wan2.1-I2V-14B**, an image-to-video diffusion model. Conditioning is bimodal: the current 3D scene and pose are supplied as a rendered **initial frame**, while the intended action is supplied as a **text prompt**. At inference time, FantasyHSI uses **30 diffusion inference steps** and **classifier-free guidance** with **guidance scale \(4.5\)** [2509.01232].

Generated clips are converted back into 3D with **WHAC** motion capture. The recovered SMPL-X motion sequence is then applied in the original 3D scene, updating the human state and producing the next graph node. The full 4D interaction is thus constructed as a sequence of generated clips and lifted state transitions rather than as a monolithic trajectory synthesized in one pass [2509.01232].

To improve physical realism and instruction following, the authors fine-tune the video model using **Direct Preference Optimization (DPO)**. The preference dataset is constructed from videos generated by four proprietary systems—**VEO**, **HunYuan-Video**, **Runway**, and **Kling**—which professional annotators label as positive or negative according to instruction-following quality, motion artifacts, limb deformation, and scene inconsistency. The resulting dataset contains **10,000 preference pairs** \((x^w, x^l)\) [2509.01232].

Training starts from the pre-trained Wan2.1-I2V-14B model and applies DiffusionDPO-style optimization with temperature coefficient \(\beta = 5000\), learning rate \(1 \times 10^{-5}\), and approximately **20 hours** of training on **8× A100** GPUs. The paper also reports **SFT** as a comparison point, but the final FantasyHSI system uses DPO rather than positive-only supervised fine-tuning [2509.01232].

Empirically, DPO reduces **limb distortions**, **foot sliding**, and **penetration into scene geometry**, while improving **CLIP-S** and obstacle-response metrics relative to both the pre-trained baseline and the SFT variant. The paper treats this as evidence that preference alignment is effective not only for perceptual quality but also for physically grounded scene interaction [2509.01232].

## 5. SceneBench and empirical performance

FantasyHSI is evaluated on **SceneBench**, a custom benchmark comprising **20 distinct 3D scenes**: **10 indoor** scenes such as bedrooms, cowsheds, and gyms, and **10 outdoor** scenes such as grasslands, riversides, urban streets, and rural farms. The benchmark includes **120 text–scene–position test instances** for static-environment evaluation [2509.01232].

A second evaluation regime tests **scene perception and response** by inserting obstacles on pre-planned paths. These obstacles include **15 seen objects** from TRUMANS, such as chairs, sofas, and vases, and **15 novel objects** from the internet, such as pumpkins and rocks. This setting measures not only whether a model collides less, but whether it actually reacts to obstacles through path or motion changes [2509.01232].

The benchmark uses a set of geometric and semantic metrics. For scene interaction, the reported metrics are **Penetration Score (P-Score)**, **Foot Sliding Score (FS)**, **CLIP-Score (CLIP-S)**, **CLIP-Consistency (CLIP-C)**, and **Diversity**. For obstacle response, the reported metrics are **Penetration Obstacle Score (POS)** and **Reaction Divergence Score (RDS)** [2509.01232].

FantasyHSI is compared against **TRUMANS**, **LINGO**, and **PedGen**. The comparison is notable because those baselines receive more structured inputs: TRUMANS is provided with a full trajectory, while LINGO and PedGen are given start and goal positions. FantasyHSI, by contrast, plans from high-level text via its agents [2509.01232].

On the main benchmark table, FantasyHSI reports **P-Score \(0.020\)**, **FS \(0.102\)**, **CLIP-S \(0.31\)**, **CLIP-C \(0.9937\)**, **Diversity \(0.1911\)**, **POS \(0.063\)**, and **RDS \(0.3285\)**. The corresponding ranges for the three main baselines are **P-Score \(0.063\)–\(0.085\)**, **FS \(0.148\)–\(0.199\)**, **CLIP-S \(0.17\)–\(0.20\)**, **Diversity \(0.1102\)–\(0.1703\)**, **POS \(0.119\)–\(0.133\)**, and **RDS \(0.1290\)–\(0.1806\)** [2509.01232].

The ablation results further isolate the contribution of the main components. The **SFT** variant yields **P-Score \(0.034\)**, **FS \(0.124\)**, and **CLIP-S \(0.27\)**, while the **Baseline** pre-trained model yields **P-Score \(0.051\)**, **FS \(0.149\)**, and **CLIP-S \(0.23\)**. Removing multi-agent planning lowers **CLIP-S** from **0.31** to **0.24**, and removing the critic agent lowers it to **0.23** while worsening **POS** and **RDS**. The paper interprets these results as showing that DPO improves physical realism, multi-agent decomposition improves long-horizon task execution, and the critic preserves task consistency under generative stochasticity [2509.01232].

Qualitatively, FantasyHSI is reported to handle tasks such as climbing a roughly **20 m ladder** to a rooftop, sitting on a windowsill, and reacting to inserted pumpkins by stepping over them rather than passing through them. The baselines are described as weaker at novel-object interaction, precise geometry handling, or rich non-locomotor behavior [2509.01232].

## 6. Terminology, related usages, and open issues

The term **“FantasyHSI”** is not terminologically uniform across the supplied literature. In **SPORTSQL**, it appears as a **fantasy sports human–system interface** perspective, where the system serves as a reference design for a conversational analysis layer, a real-time decision-support backend over live FPL data, and automatic visualization of sports statistics [2508.17157]. In **daily fantasy sports** research, related interface goals are coupled to lineup portfolio construction under top-heavy payoff structures through integer programming, with an emphasis on high expected score, high variance, and low cross-lineup correlation [1604.01455]. In **fantasy basketball**, dynamic valuation is formalized through **H-scoring** and the **\(H_0\)** implementation, which adapt category weights, positional assignments, and format-specific objectives during a head-to-head draft [2409.09884]. In **fantasy cricket**, **FanCric** uses a **LangGraph**-based multi-agent framework with agents such as Researcher, Career Profiler, Form Assessor, Strategizer, Selector, and Evaluator to construct Dream11 teams from structured and unstructured data [2410.01307].

This suggests that the label has acquired two distinct meanings in current usage: a specific **graph-based 4D HSI generator** in computer vision and graphics, and a broader **fantasy-sports human–system-interface** perspective in sports analytics. The two usages are conceptually separate, even though both emphasize multi-agent reasoning, structured state representations, and decision support [2410.01307][2508.17157][2509.01232].

Within the computer-vision sense of the term, the stated limitations are direct. FantasyHSI remains computationally heavy because diffusion-based video generation and VLM-based multi-agent reasoning are expensive at inference time, which hinders real-time or highly interactive deployment. It also focuses on **static 3D scenes**, because the reported benchmark and data regime do not cover environments with moving obstacles or changing layouts [2509.01232].

The paper therefore positions future work around two axes: improving efficiency for **real-time interaction** and extending the framework to **dynamic environments**. In its present form, FantasyHSI is best understood as a synthesis of symbolic graph planning, VLM-based multi-agent control, preference-aligned video diffusion, and video-to-3D motion lifting for long-horizon HSI in previously unseen scenes [2509.01232].

Source: https://www.emergentmind.com/topics/fantasyhsi