MVGameHuman: Multimodal Human-Centered Game Agents

• MVGameHuman is a research agenda for multimodal, human-centered virtual humans and game agents that perceive game worlds, model people, render controllable avatars, and interact through speech, vision, gesture, and gameplay.
• Its applications span physically grounded animation, accessibility interfaces, adaptive health-oriented gaming, neural avatar generation, game commentary, human–LLM development, and closed-loop game-agent benchmarking.
• The agenda emphasizes measurable human benefit alongside task performance, combining metrics such as win rate, progress, latency, avatar fidelity, perceived agency, workload, and personalized improvement over baseline agents.

MVGameHuman is an Editor’s term for a research agenda concerning multimodal, human-centered virtual humans and game agents: systems that perceive game worlds and human behavior, represent and render controllable digital humans, support natural interaction, generate or adapt game content, and evaluate agent performance against human capabilities. The term is not defined by any single cited paper. Its constituent research problems are distributed across physically responsive virtual humans, vision-based and multimodal interaction, adaptive health-oriented game AI, human–LLM game development, neural avatar generation, game commentary, interactive MLLM evaluation, and standardized game-agent benchmarking.

1. Conceptual scope and system dimensions

MVGameHuman encompasses several technically distinct but interconnected layers.

At the avatar representation and animation layer, the objective is to construct digital humans that preserve appearance, articulation, contact, balance, facial expression, and hand motion. “Balanced Virtual Humans Interacting with their Environment” treats the avatar as an articulated physical system controlled through joint torques, motion-capture targets, task-space control, collision handling, and unilateral balance constraints (0707.3562). GoMAvatar instead combines an explicit deformable mesh with attached Gaussian primitives to produce compact, animatable avatars from monocular video (Wen et al., 2024). MVFGA focuses on upper-body avatars, separately modeling FLAME facial parameters and MANO hand parameters before attaching dynamic 3D Gaussians to an upper-body mesh (Javanmardi et al., 5 Aug 2026).

At the human–game interaction layer, MVGameHuman includes natural, multimodal, and accessibility-oriented input. Vision-based interaction can map nose-tip motion to pointer coordinates and unilateral eye blinks to mouse clicks using inexpensive USB cameras at approximately 30 frames per second (Sumathi et al., 2010). GEMINI maps Kinect body poses, speech, and Wii input to keyboard and mouse events without requiring modifications to the target game (Teófilo et al., 2013). Such systems distinguish sensing, recognition, input management, and native game-control execution.

At the control-abstraction layer, the central issue is how much of a virtual human’s movement the human operator specifies. High-level control maps commands to predefined actions; low-level control exposes joint variables or torques; intelligent middle-level control allows a human to specify goals or body-part relations while an optimizer generates physically valid motion (Babadi et al., 2018). This layer can mediate between speech, gesture, gaze, language-model commands, and joint-level actuation.

At the adaptive agent layer, game intelligence is evaluated not only by victory or task reward, but also by its effect on the human collaborator. RLHG defines positive human gain as improvement over a primitive human-performance baseline and combines this gain with the original task advantage (Gao et al., 2024). Adaptive Motion Gaming AI similarly selects fighting-game actions partly according to the expected effect on the player’s body-segment balance (Paliyawan et al., 2017).

At the perception, reasoning, and evaluation layers, the agenda includes visual game understanding, multimodal commentary, memory, long-horizon planning, action grounding, and state-verifiable assessment. VideoGameBunny provides a game-specific visual-LLM for screenshot interpretation (Taesiri et al., 2024). Game-MUG combines game events, caster transcripts, audio, and audience chat for situation understanding and commentary generation (Zhang et al., 2024). V-MAGE and GameWorld evaluate MLLMs in closed-loop game environments, exposing failures in dynamic perception, timing, planning, and control (Zheng et al., 8 Apr 2025, Ouyang et al., 8 Apr 2026). A modular harness further separates perception, memory, and reasoning to analyze their game-dependent contributions (Zhang et al., 15 Jul 2025).

2. Physically grounded virtual humans

The physically grounded virtual-human paradigm treats animation as a control problem rather than as a fixed sequence of joint poses. In “Balanced Virtual Humans Interacting with their Environment,” the system follows the architecture

$$\text{motion-capture targets} \rightarrow \text{controller} \rightarrow \text{joint torques} \rightarrow \text{physical simulation} \rightarrow \text{rigid-body states and contacts}.$$

Motion-capture input supplies target positions and velocities for selected body parts or task-space frames. Passive error projection generates virtual guides, task-space control converts task errors into compensating commands, internal control exploits redundancy, and balance control adds a unilateral constraint. Simulation handles articulated-body dynamics, collision detection, contact points, joint limits, nonpenetration, and environmental interaction (0707.3562).

The principal retargeting problem is morphological mismatch. Targets reachable by a tall actor may be mutually incompatible for a shorter avatar. A purely kinematic retargeting system can therefore produce infeasible configurations or place the avatar’s center of mass outside its support region. The proposed controller permits the avatar to deviate from the captured pose when required to preserve balance, avoid obstacles, or generate physically plausible environmental forces.

Static balance is represented by the projection of the center of mass into an elliptical approximation of the support polygon. The balance-distance variable is

$$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$

with the static-balance condition $\delta\geq 0$. The Jacobian

$J_\delta=\frac{\partial \delta}{\partial q}$

relates changes in generalized coordinates to changes in the balance margin. Balance is enforced as a unilateral constraint through a Linear Complementarity Problem:

$0\leq \alpha \perp z\geq 0,\qquad \alpha=Mz+q.$

When the distance-to-limit would become negative, an additional torque contribution, denoted $\tau_{\mathrm{LCP}}$, acts through the balance Jacobian. The exact LCP matrices, solver, friction model, and complete dynamics-to-LCP reduction are not specified in the supplied text.

Environmental contact is handled by the physical simulation rather than by purely visual inverse-kinematics corrections. Demonstrations include leaning on a table, pushing, pulling, hitting, and collision avoidance. Virtual guides impose task-specific restrictions without requiring haptic hardware; in the drilling experiment, they constrain drill-axis alignment while leaving the operator control over selected degrees of freedom.

The demonstrated balance model remains limited to static, foot-supported balance on a planar surface. It does not adequately model dynamic walking, running, stepping recovery, slipping, friction cones, hand-supported balance, arbitrary multicontact configurations, or moving support surfaces. The system is therefore a low-level adaptive control architecture rather than a complete autonomous humanoid animation or locomotion system.

3. Avatar reconstruction, appearance, and animation

MVGameHuman requires avatar representations that separate identity, appearance, pose, articulation, and rendering. The cited neural-avatar methods adopt different points in this design space.

GoMAvatar reconstructs a subject-specific animatable human from a single monocular RGB video. Its representation combines a canonical deformable mesh, linear-blend-skinning weights, per-face Gaussian primitives, pose-dependent non-rigid deformation, pose refinement, mesh normals, and a learned shading network (Wen et al., 2024). One Gaussian is associated with every mesh triangle. The mesh supplies explicit geometry and articulation, while local Gaussian rotation, scale, and color provide high-frequency appearance.

The canonical avatar is initialized from the SMPL T-pose. A pose-conditioned non-rigid deformer predicts vertex offsets, after which linear blend skinning produces posed vertices. Gaussian means and covariances are recomputed from the current triangle geometry. The final rendering combines Gaussian-splat pseudo-albedo with a learned shading map. The system is compatible with rasterization-oriented graphics pipelines such as OpenGL.

On ZJU-MoCap, GoMAvatar reports 23.2 milliseconds per frame, approximately 43 FPS, and 3.63 MB of inference parameters per subject on an NVIDIA A100. On PeopleSnapshot, it reports 25.82 milliseconds per frame. The method achieves strong rendering and normal-consistency results while remaining substantially faster and smaller than the compared monocular human-modeling systems. These figures concern inference after avatar construction; they do not imply real-time monocular reconstruction or game-engine deployment on consoles or mobile hardware.

MVFGA addresses upper-body modeling, emphasizing facial expressions and two-handed gestures. It uses calibrated multi-view RGB video, FLAME facial parameters, MANO hand parameters, an upper-body SMPL-X mesh, semantic masks, and surface-attached dynamic Gaussians (Javanmardi et al., 5 Aug 2026). The capture rig contains 17 synchronized cameras, with 15 front-facing cameras distributed over approximately $150^\circ$ and two rear-side cameras. The data are recorded at $1920\times1080$ resolution and 25 frames per second.

Face and hand estimates are selected using multi-view reprojection error and integrated into a unified upper-body fit. The model represents body, hand, face, global rotation, and global translation parameters separately from identity shape and Gaussian appearance. Each Gaussian is attached to a parent mesh triangle and transformed with that triangle, preventing unconstrained spatial drift.

MVFGA reports an upper-body rendering rate of approximately 49 FPS on an NVIDIA RTX 3090. It achieves an image-quality $L_1$ error of 4.06, SSIM of 0.938, PSNR of 25.93, and LPIPS of 0.053 in the reported comparison. It obtains the best reported hand AKD, torso AKD, and facial identity similarity among the compared methods. The associated MVFGA-MoCap dataset contains 15 participants, 8 male and 7 female, aged 24–32, performing controlled facial expressions, two-handed gestures, and free-form communication.

Both representations expose important requirements for MVGameHuman. Explicit skeletal structure supports retargeting, animation blending, inverse kinematics, procedural posing, and animation state machines. Gaussian appearance supports detailed rendering and identity preservation. However, neither system by itself solves arbitrary clothing topology, physically correct materials, relighting, collision handling, facial interaction, full-body locomotion, or large-scale multi-avatar deployment. MVFGA additionally remains dependent on calibrated multi-view capture, while GoMAvatar cannot reliably reconstruct completely unseen regions from monocular video.

4. Multimodal interaction and control abstraction

Natural interaction in MVGameHuman can involve body pose, facial movement, speech, gaze-like pointing, mouse and keyboard events, or semantic commands. These modalities differ in bandwidth, latency, precision, and accessibility.

The vision-based game interface in “Vision Based Game Development Using Human Computer Interaction” uses facial-feature detection, eye localization, nose-tip tracking, eyebrow detection, and frame differencing (Sumathi et al., 2010). Nose movement controls a two-dimensional pointer, while left- and right-eye blinks generate left- and right-click events. Automatic initialization creates an online open-eye template from involuntary blinking. Tracking recovery occurs when the eye-template correlation score falls below a threshold. The system operates at approximately 30 FPS using inexpensive USB webcams.

The interaction protocol assumes that pointer motion stops before a blink click. A user moves the nose, becomes stationary, and then performs a deliberate blink. A configurable blink duration suppresses duplicate detections during eyelid closure and reopening. The system supports first-person shooter, role-playing, action, and general mouse-driven applications, but the paper does not report latency, false-positive rates, calibration time, camera resolution, or detailed performance across facial diversity, glasses, occlusion, or head pose.

GEMINI provides a broader multimodal middleware architecture. Its input layer receives Kinect skeleton data, Kinect microphone-array audio, and Wii Nunchuk or WiiMote events. The logic layer contains a Pose Recognizer, Speech Recognizer, and Input Manager. The application layer contains a configuration GUI and an external game. Pose recognition uses spatial restrictions such as Distance, InFront, LeftTo, AboveOf, and Above Value. Speech uses Microsoft SAPI, while Wii events are received through WiimoteLib (Teófilo et al., 2013).

GEMINI maps recognized events into keyboard presses, holds, mouse movement, mouse-button presses, and mouse-button holds. In its evaluation, pose processing takes approximately 16–25 milliseconds, while speech recognition introduces a delay of approximately 1–2 seconds. The study involved 25 participants for pose tests and a 16-participant Skyrim pilot. Pose accuracy varied across actions, with crouch detection reported at 76%, while several arm and punch poses exceeded 95%. The authors also describe interaction schemes for Devil May Cry 4, Super Mario, and The Legend of Zelda.

Intelligent middle-level control addresses a different problem: how a human specifies movement goals. In the two-player martial-arts prototype, clicking and dragging body parts defines a movement objective, while a rolling-horizon CMA-ES controller produces detailed trajectories in a physically simulated upper-body humanoid (Babadi et al., 2018). The prototype uses a 9-bone model with 16 actuated degrees of freedom and Open Dynamics Engine. CMA-ES optimizes cubic-spline action parameters over a 0.6-second horizon, executing only the first action before replanning. The controller runs four CMA-ES updates per frame with a population size of 16.

The prototype demonstrates a continuum between action labels and joint-level manipulation. Its evaluation used 12 participants, three slow-motion settings, qualitative questionnaires, and measures of command precision and preferred tempo. The preferred mode was $0.20\times$ for 7 participants, compared with 4 for $$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$0 and 1 for $$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$1. Eight participants reported high command-execution precision, while four found slow-motion control difficult. The model is upper-body-only and does not establish robust full-body balance, locomotion, or general humanoid control.

5. Human-centered adaptive game agents

A central MVGameHuman principle is that an agent’s task competence and its contribution to human experience are not identical. RLHG formalizes this distinction in collaborative reinforcement learning (Gao et al., 2024).

The human-centered objective represents human experience through task-related goals. Let $$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$2 be a pretrained self-centered agent and $$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$3 the human’s expected goal achievement when collaborating with it. Human gain is defined as

$$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$4

Only positive gain is rewarded. The total advantage is

$$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$5

where $$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$6 preserves task competence and $$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$7 measures improvement over the primitive baseline. Training mixes human-agent and agent-only settings, and an adaptive task gate can activate enhancement only when the original task value exceeds a threshold $$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$8.

In Honor of Kings, RLHG improves human-goal measures while reducing win rate. For high-level human participants, win rate changes from 52.0% with Wukong to 46.7% with RLHG, while MVP score changes from $$\delta = d^2-\left\|P(x_{\mathrm{com}}-x_c)\right\|_Q^2,$$9 to $\delta\geq 0$0. For general-level participants, win rate changes from 34.0% to 30.0%, while MVP score changes from $\delta\geq 0$1 to $\delta\geq 0$2. Participants rated RLHG more favorably in behavioral rationality, enhancement degree, gaming experience, and overall preference.

The paper distinguishes targeted assistance from pathological over-assistance. Helpful behaviors include protecting the human, sharing resources, supporting a highlight, and relinquishing a resource so the human can obtain it. HRE, which directly optimizes human reward without primitive-baseline subtraction, may follow the human excessively and sacrifice victory performance. RLHG’s positive-gain formulation is intended to reward assistance attributable to the agent rather than human outcomes that would have occurred independently.

Adaptive Motion Gaming AI applies a related human-centered principle to full-body motion gaming (Paliyawan et al., 2017). The system uses FightingICE and UKI, with Kinect-derived movement accumulated over four segments: right arm, left arm, right leg, and left leg. The balancedness score is computed from pairwise movement deficits. HP-AI uses MCTS candidate actions and an action-to-counteraction probability model to predict the body segments a player will move. The selected action is intended to reduce the predicted movement gap.

Five subjects played against the conventional MctsAi and HP-AI. Balancedness improved for four of the five subjects. The reported final values include improvements of 6.23 points, 12.24 points, 6.73 points, and 2.98 points for four subjects, while one subject declined by 9.91 points. The study is preliminary: it contains a small sample, uses a narrow movement-balance proxy, and does not establish long-term health effects, injury reduction, or clinical benefit.

Together, these studies support a human-centered evaluation model with separate dimensions for task success, human-goal achievement, collaboration quality, and subjective experience. They also show that human-centered optimization introduces trade-offs rather than uniformly improving all objectives.

6. Perception, game understanding, and multimodal generation

MVGameHuman requires perception systems that connect visual observations and human-relevant semantics to game state, commentary, or action.

VideoGameBunny, also called dimgrey in the supplied material, is a LLaVA-style visual-LLM based on Bunny, with Llama 3 8B and a multi-scale SigLIP vision encoder (Taesiri et al., 2024). Its training resource contains 185,259 video-game images from 413 titles and instruction data involving captions, image-to-JSON descriptions, and question-answer pairs. The structured representation includes scene description, dialogue, subtitles, inventory, HUD, NPC status, characters, animation states, objects, textures, lighting, weather, environmental effects, and player status.

The model reaches 85.1% on the reported evaluation set, compared with 83.9% for LLaVA-1.6-34B. Its strongest relative improvements concern anomaly and glitch recognition and HUD/UI understanding. The model remains screenshot-based: it does not directly model temporal frame sequences, multiple viewpoints, player action histories, or closed-loop control. Documented failure modes include hallucinated objects and characters, incorrect OCR, misread spatial relations, and invented dialogue.

Game-MUG addresses a complementary problem: multimodal situation understanding and commentary generation for League of Legends esports broadcasts (Zhang et al., 2024). It contains approximately 70,000 clips, 70,711 caster-transcript sentences, 3,657,611 audience-chat instances, and 15,221 game events from 216 matches collected from Twitch and YouTube during 2020–2022. Modalities include structured events, Whisper transcripts, GeMAPS acoustic features, audience chat, emojis, emotes, and generated commentary.

The event-understanding model combines text, audio, and multimodal Transformers. With all modalities, DeBERTaV3 achieves 69.65% overall event accuracy, with 86.71% for Kill, 31.53% for Tower, and 59.02% for Dragon. Audio and previous event history generally improve performance, while chat alone contributes less. The commentary component conditions GPT-2 or Pythia on transcript, audience chat, and special event tokens. Game-MUG is relevant to socially enriched commentary and affective game understanding, but it does not provide embodied human behavior, player actions, interactive dialogue, or visual gameplay understanding.

AMG extends controlled human generation from static avatars to photorealistic video. It renders articulated SMPL or SMPL-X avatars and supplies the rendered sequence to a pretrained text-to-video diffusion model (Yang et al., 2024). The avatar condition carries motion, approximate geometry, identity proxies, depth ordering, and camera layout, while the diffusion model supplies photorealistic appearance and background harmonization. AMG demonstrates multi-person generation, camera changes, identity and clothing preservation, and motion-conditioned video generation. It reports a motion score of 69.88 and a CLIP score of 33.59 in the supplied comparison.

The main weaknesses are reconstruction errors from TRACE, limited fine-grained limb control, camera-rotation ambiguity, textual rather than photometric identity specification, a relatively small training corpus, and the lack of explicit multi-view, identity, pose, or temporal losses. For MVGameHuman, the most direct implication is a hybrid architecture in which a game engine supplies deterministic avatar motion, camera, depth, segmentation, and interaction layout, while diffusion supplies photorealistic rendering and background integration.

7. Evaluation, human comparison, and open problems

Interactive game-agent evaluation must distinguish static recognition from closed-loop perception, reasoning, timing, and action execution.

V-MAGE evaluates MLLMs in five modified games—RaceGame, FlappyBird, PongGame, SuperMario, and Tempest Run—with more than 30 handcrafted levels (Zheng et al., 8 Apr 2025). Models receive screenshots, recent history, game rules, and output constraints, then emit one action per step. Five human participants provide an average human baseline. The results show that models approach human performance on simple positioning tasks but fail on dynamic control. In RaceGame Level 4, the best model scores 23.47 against 100 for humans. In Pong Level 2, the best model scores 0.35 against 10. In SuperMario Level 10, the best model scores 108.15 against a human average of 800.

The benchmark’s “no history” conditions show that GPT-4o benefits significantly from multi-frame input, while other models do not consistently exploit additional frames. The results expose failures in velocity estimation, temporal prediction, action timing, and long-horizon reasoning.

A modular harness for multi-turn gaming environments separates perception, memory, and reasoning (Zhang et al., 15 Jul 2025). It evaluates Sokoban, Candy Crush, 2048, and Tetris through Gymnasium and Stable Retro interfaces. Backend symbolic state, vision, or combined perception can be supplied. Memory stores recent trajectories and generates reflections about state changes, action effectiveness, recurring problems, and future strategy. The harness improves performance over un-harnessed baselines, with perception especially important for visually complex games and memory especially valuable for long-horizon tasks.

GameWorld provides a more standardized and verifiable benchmark (Ouyang et al., 8 Apr 2026). It contains 34 browser games and 170 tasks, supports both Computer-Use Agents and semantic Generalist agents, and evaluates 18 model–interface pairs. Its default budget is 100 model actions per task. State-verifiable metrics are read from serialized game state rather than inferred by OCR or a vision-language judge.

GameWorld separates success rate and progress:

$\delta\geq 0$3

and

$\delta\geq 0$4

The best reported agent, Gemini-3-Flash-Preview, reaches 21.2% success rate and 41.9% progress. A novice human reaches 55.3% success rate and 64.1% progress, while an expert reaches 77.1% and 82.6%. GameWorld therefore reports a large agent–human gap even when agents generate mostly valid actions. It also distinguishes paused evaluation, which isolates decision quality, from real-time evaluation, which includes inference latency.

A comprehensive MVGameHuman evaluation would combine these benchmark principles with human-centered measures. It should report:

• Task competence: win rate, completion, progress, damage, survival, and objective contribution.
• Human benefit: personalized human-goal achievement relative to a baseline agent.
• Control quality: action validity, latency, timing precision, recovery, and motor burden.
• Avatar quality: pose fidelity, facial and hand accuracy, identity similarity, temporal stability, collision violations, and novel-view consistency.
• Interaction quality: usability, learnability, perceived agency, trust, workload, frustration, accessibility, and preference.
• Generalization: unseen games, unseen users, unseen poses, new cameras, new identities, new clothing, and multi-person interaction.
• Verification: deterministic state evaluators, repeated runs, participant-level statistics, confidence intervals, and clear separation of paused and real-time conditions.

The principal unresolved problems are full-body multimodal control, dynamic balance and multicontact interaction, robust facial and hand tracking, animation-ready neural avatars, temporally and multi-view-consistent rendering, human-adaptive planning, semantic-to-physical action grounding, and evaluation with sufficiently broad human baselines. The cited work collectively supports a modular MVGameHuman architecture in which perception, memory, human modeling, semantic planning, physical control, avatar rendering, and state-verifiable evaluation are explicit components rather than an undifferentiated end-to-end system.