Infinite Worlds with Versatile Interactions

• Infinite Worlds with Versatile Interactions are generative-simulation systems that enable unbounded interaction horizons by dynamically generating narratives, visuals, and mechanics.
• The systems integrate deterministic code updates with stochastic model-driven enrichment to preserve coherent state and persistent memory across extended interactions.
• They blend web-native frameworks with advanced generative techniques to achieve scalable, responsive, and multimodal interactive environments.

Infinite worlds with versatile interactions denotes a class of world-model and generative-simulation systems designed to sustain unbounded or effectively unbounded interaction horizons while preserving coherence, responsiveness, and a broad action vocabulary. In recent arXiv literature, the term spans web-native persistent worlds, chunk-autoregressive video simulators, progressive 2.5D-to-3D environments, apartment-scale interactive scene generators, and embodied agents operating in open-ended settings. Despite large architectural differences, these systems repeatedly converge on a shared problem: how to extend a world beyond fixed clips, finite maps, or static assets without surrendering control over state, memory, and interaction semantics (Gao et al., 8 Jul 2026, Feng et al., 29 Dec 2025, Xiong et al., 16 Jun 2026, Wu et al., 2 Feb 2026).

1. Conceptual scope

A central formulation comes from the distinction between finite and infinite games. In "Unbounded," finite games have fixed boundaries, known players, stable rules, a clear beginning and end, and are “played to be won,” whereas infinite games are “played for the purpose of continuing the play,” with evolving rules, no definite endpoint, and an open set of participants. That work operationalizes an infinite game by placing both game logic and graphics inside generative models: a distilled Gemma-2B model dynamically generates mechanics, narratives, and character interactions in real time, while a text-to-image backbone with a dynamic regional IP-Adapter produces fresh visuals every second (Li et al., 2024).

Other systems instantiate “infinite” in more engineering-oriented terms. Web World Models describe “deterministic generation to achieve unlimited but structured exploration” and emphasize “zero persistent storage beyond standard code deployments” (Feng et al., 29 Dec 2025). ActWorld defines arbitrarily long, chunk-by-chunk rollouts with mid-rollout object interaction (Xiong et al., 16 Jun 2026). Infinite-World targets coherent visual memory over 1000+ frames in complex real-world environments (Wu et al., 2 Feb 2026). The Matrix reports continuous, uncut hour-long sequences at 16 FPS (Feng et al., 2024). LingBot-World 2.0 frames the goal as an “unbounded interaction horizon” together with 720p video streams at 60 fps (Gao et al., 8 Jul 2026). This suggests that, in current usage, “infinite” usually means horizon scalability, on-demand extension, or persistent replayable generation rather than literal precomputation of a complete world.

A common misconception is to equate this literature with the older “many interacting worlds” program in quantum foundations. Hall, Deckert, and Wiseman use “world” to mean an entire universe with well-defined properties, and propose that quantum effects arise from a universal inter-world interaction, with the wave function recovered only in the limit of infinitely many worlds (Hall et al., 2014). In contemporary world-model research, by contrast, the phrase refers to interactive simulators and generative environments rather than a quantum ontology.

2. State, dynamics, and conditioning structures

One influential decomposition is the Web World Model formalism. At time step $t$, state is written as $S_t=(S_t^\phi,S_t^\psi)$, where $S_t^\phi\in\Im$ is the Physics Layer and $S_t^\psi\in\mathcal O$ is the Imagination Layer. Dynamics proceed in two sequential steps: a deterministic code-only update, $S_{t+1}^\phi=F_{\text{code}}(S_t^\phi,a_t)$, followed by stochastic model-driven enrichment, $$S_{t+1}^\psi\sim\pi_\theta(S^\psi\mid S_{t+1}^\phi)$$. The same framework also specifies just-in-time deterministic generation by seeding each generation with a pure hash function, $\text{seed}=h(\text{state\_id},\text{action\_id})$, so repeated state-action pairs yield identical outputs under temperature $0$. Typed JSON or TypeScript interfaces, together with schema validation, enforce that model outputs cannot violate physics invariants (Feng et al., 29 Dec 2025).

Video-based systems usually formulate the problem autoregressively in latent space. The Matrix writes

$$p(x_{1:T}\mid a_{1:T})=\prod_{t=1}^T p_\theta(x_t\mid x_{<t},a_{\le t}),$$

compresses 1280×720 RGB frames into 3D-VAE latents, translates keyboard and mouse actions into short English prompts, and injects the resulting action embeddings into a Diffusion Transformer through causal cross-attention (Feng et al., 2024). LingBot-World 2.0 adopts a closely related causal factorization over frames and actions, but couples it to Mixture of Bidirectional and Autoregressive attention masks and a flow-matching objective

$$\mathcal L_{fm}=E_{x,i,t,\epsilon}\Bigl\|v_\theta(x_i^t,t\mid x_{<i},p_{\le i},a_{\le i})-(\epsilon-x_i)\Bigr\|^2$$

to improve long-horizon fidelity (Gao et al., 8 Jul 2026). ActWorld extends the formulation further by jointly modeling video chunks and user actions,

$S_t=(S_t^\phi,S_t^\psi)$0

with $S_t=(S_t^\phi,S_t^\psi)$1 a VAE-latent chunk of approximately 33 frames and real-time inference distilled to $S_t=(S_t^\phi,S_t^\psi)$2 denoising steps (Xiong et al., 16 Jun 2026).

Three-dimensional world generators tend to separate semantic planning, structured latent generation, and reconstruction. SceneFoundry maps a natural-language prompt through an LLM module into floor-plan parameters for Infinigen, then uses a diffusion-based scene populator with differentiable guidance functions for object quantity, articulation clearance, and walkable space, before retrieving articulated CAD models from 3D-FRONT and GAPartNet (Chen et al., 9 Jan 2026). HunyuanWorld 1.0 stages panorama generation, agentic world layering, layer-wise 3D reconstruction, and long-range extension through Voyager (Team et al., 29 Jul 2025). NeoWorld organizes the scene as a hybrid union of object-centric 2.5D Gaussians and fully reconstructed 3D objects, with progressive unfolding triggered by proximity or explicit user request (Zhao et al., 29 Sep 2025). A plausible implication is that the field increasingly treats “world generation” not as a monolithic model, but as a pipeline in which semantic control, persistent state, and rendering are partially disentangled.

3. Interaction vocabularies and control channels

A recurrent theme is dissatisfaction with navigation-only control. ActWorld explicitly argues that in prior interactive world models the action vocabulary is “largely confined to navigation,” and that visually explorable worlds are “not truly actionable.” It addresses this gap by combining symbolic text commands for object interaction with low-level WASD+mouse control, chain-of-thought per-chunk captioning, and action-aware memory. On I-Bench, it reports a VLM-Action-Judge success rate of 57.8% at level 3 versus 20.1% for Yume-1.5, while retaining Key-Mouse-Following joint accuracy of 20.62% versus 20.00% for Matrix-Game 3.0 (Xiong et al., 16 Jun 2026).

LingBot-World 2.0 broadens the action taxonomy in explicitly game-like terms. Character-centered actions include W/A/S/D for walking, I/J/K/L for camera control, “U” for melee combat, “O” for archery or shooting, numeric slots for spell-casting, “Space” for jump, and “P” for glide. Text-driven events include day-night shifts, weather changes such as summoned rain or snowstorm, and local entity injection such as birds, cloud-whales, or arbitrary creatures registered on keys 1–9. The same framework introduces an agentic harness in which a Director agent proposes event candidates and a Pilot agent grounds them into video transitions, and it sketches a multi-player interface with a shared frame index $S_t=(S_t^\phi,S_t^\psi)$3 over low-latency WebSocket or gRPC channels (Gao et al., 8 Jul 2026).

AlayaWorld offers a related but modular design. Camera control is encoded both geometrically, through a 3D cache render, and parametrically, through AdaLN camera modulation. Non-navigational interactions such as “cast a fireball” and “summon a monster” are realized through chunk-level prompt switching: the active prompt tokens are replaced only at chunk boundaries, and the cross-attention key-value cache is flushed so new actions take effect in the next chunk without regenerating earlier chunks. After DMD-style distillation, the system pushes per-chunk synthesis time below 200 ms on one A100 GPU, with semantic latency bounded by one chunk of approximately one second (Team et al., 7 Jul 2026).

Embodied and object-centric interaction extends beyond synthetic game worlds. QuadrupedGPT uses GPT-4o as a large multimodal model over RGB and depth frames, proprioception, and language; a global planner operates on a semantic cost map, while a local policy translates waypoints into commanded velocities and behavior parameters for adaptive locomotion. In real-world path planning, the success rate improves by approximately 30 percentage points when using the semantic cost map rather than a naïve shortest path (Mei et al., 2024). NeoWorld grounds natural-language instructions into JSON actions specifying instance identity and geometric or physical edits, including translation, scale, rotation, Young’s modulus, friction, and force vectors; depending on the returned action, it executes either direct keyframe animation or Material Point Method simulation (Zhao et al., 29 Sep 2025). HunyuanWorld exposes explicit runtime control signals such as move_object, rotate_object, and rescale_layer, reflecting a more graphics-oriented notion of interactivity (Team et al., 29 Jul 2025).

4. Memory, persistence, and long-horizon stability

The principal technical obstacle in infinite-world systems is not initial generation but persistence under continued interaction. Web World Models solve a narrow version of this problem through deterministic seeding. Because each just-in-time generation is seeded by $S_t=(S_t^\phi,S_t^\psi)$4, repeated visits to the same location or repeated action patterns reproduce the same LLM output without any backing store. The same framework also formalizes “graceful degradation”: high fidelity with live LLM generation, medium fidelity with cached generations, and base fidelity with code-only templates, while keeping world logic intact even if the model is offline (Feng et al., 29 Dec 2025).

ActWorld diagnoses a more specific failure mode, the “action-forgetting pathology,” in which recency-biased compression discards event-transition frames that causally determine later object states. Its response is hierarchical action-aware memory: a routed sliding-window history with an importance score weighted by interaction and phase, an action-conditioned attention amplification mechanism, and a persistent memory bank that stores event-update and object-identity tokens. The training corpus comprises a 100K interaction video dataset with 55 K synthetic clips and 45 K real-world clips covering 40 action categories. Inference is chunk-streamed at approximately 0.1 s on a single A100 GPU, and on I-Bench-mini the addition of PersistentMem raises IF from 2.413 to 2.557 and SC from 0.844 to 0.871 relative to the preceding ablation state (Xiong et al., 16 Jun 2026).

Infinite-World addresses long-range consistency in real-world videos where pose estimates are noisy and viewpoint revisits are sparse. Its Hierarchical Pose-free Memory Compressor recursively distills historical latents to a fixed budget $S_t=(S_t^\phi,S_t^\psi)$5, guaranteeing $S_t=(S_t^\phi,S_t^\psi)$6 memory growth as horizon length increases, while Uncertainty-aware Action Labeling preserves a tri-state logic of No-op, Discrete Action, and Uncertain rather than collapsing noisy motion into a deterministic label. Revisit-Dense Finetuning then activates loop-closure behavior using a compact 30-minute dataset with frequent revisits. In reported results, Memory Consistency in a large-scale user study is 1.92, Visual Fidelity is 1.67, and Overall ELO reaches 1719; qualitatively, global landmarks remain stable after 1000+ frames (Wu et al., 2 Feb 2026).

Streaming video simulators make similar interventions at different abstraction levels. The Matrix uses a Shift-Window Denoising Process Model, a FIFO cache for just-decoded tokens, and a 4-step Stream Consistency Model to reach approximately 8–16 FPS on a single cloud GPU while avoiding quadratic growth in attention cost (Feng et al., 2024). AlayaWorld combines explicit 3D memory, compressed temporal history, loop-consistency loss, and error-bank drift correction, improving Loop SSIM from 0.55 ± 0.07 to 0.72 ± 0.05 against its baseline over revisit frames (Team et al., 7 Jul 2026). LingBot-World 2.0 attacks drift through causal pretraining, MoBA masking, consistency distillation, distribution matching distillation, dynamic KV-cache scheduling, and asynchronous pipelining, and reports an uninterrupted 60 minute session over 20 distinct scenarios with no visible quality decay (Gao et al., 8 Jul 2026).

5. Modalities and representative systems

The breadth of the field is visible in the range of substrate choices.

Domain	Representative system	Distinctive mechanism
Web-native persistent worlds	Web World Models	typed interfaces, deterministic seeding, code-defined physics
Generative life simulation	Unbounded	distilled Gemma-2B plus dynamic regional IP-Adapter
Real-scene interactive video	The Matrix / Infinite-World	streaming DiT control / pose-free hierarchical memory
Object-interactive video	ActWorld	action-aware memory and persistent event-object tokens
Apartment-scale interactive 3D	SceneFoundry	LLM-guided floor plans plus diffusion posterior sampling
Hybrid 2.5D-to-3D exploration	NeoWorld	progressive 3D unfolding from a single image
Panorama-to-mesh worlds	HunyuanWorld 1.0	semantically layered 3D mesh representation
Unbounded scene synthesis	WorldGrow	3D block inpainting with coarse-to-fine extension

Web World Models are notable because the “world” is implemented on a realistic web stack rather than as a fully latent simulator. The paper’s case studies include an Infinite Travel Atlas grounded in real geography, a Galaxy Travel Atlas, WWMPedia as an encyclopedic world, and Bookshelf as an infinite narrative reader. Across them, logical consistency is delegated to ordinary web code and typed interfaces, while context, narrative, and observation are delegated to the model (Feng et al., 29 Dec 2025).

SceneFoundry and HunyuanWorld represent a more graphics- and robotics-oriented branch. SceneFoundry generates apartment-scale 3D worlds with articulated furniture, differentiable guidance for object quantity and articulation collisions, and walkable-area optimization; it demonstrates 5×5 block apartment compounds with more than 25 rooms in supplemental material and reports 15–20% higher success rates than baseline procedurals on pick-and-place and door-opening tasks (Chen et al., 9 Jan 2026). HunyuanWorld instead starts from a 360° panoramic proxy, decomposes the scene into semantic layers, reconstructs them into meshes, and extends exploration through a point-cloud cache, while preserving mesh export compatibility for VR, physics simulation, and game engines (Team et al., 29 Jul 2025).

WorldGrow and NeoWorld illustrate two distinct strategies for scalable 3D extension. WorldGrow uses curated scene blocks, 3D block inpainting, and a coarse-to-fine generation hierarchy to support infinite scene generation with structurally consistent outputs (Li et al., 24 Oct 2025). NeoWorld adopts on-demand “worldbuilding”: key foreground objects are reconstructed in full 3D, backgrounds and non-interacted regions remain 2D, and only regions actively explored by the user are unfolded into higher-fidelity structure. On the WorldScore benchmark, it substantially outperforms prior 2D and 2.5D baselines, including an object-centric segmentation IoU of 70.53% and PromptAlign of 61.95% (Zhao et al., 29 Sep 2025).

6. Evaluation regimes, limitations, and research tensions

Evaluation remains heterogeneous because the literature spans multiple substrates. Video-centric systems report FVD, FID, CLIP-Score, Move-LPIPS, PSNR, VBench, VLM-Action-Judge, Key-Mouse-Following, loop SSIM, and user-study ELO (Feng et al., 2024, Xiong et al., 16 Jun 2026, Wu et al., 2 Feb 2026, Team et al., 7 Jul 2026). Three-dimensional systems report BRISQUE, NIQE, Q-Align, CLIP-I, CLIP-T, Scene Classification Accuracy, Category-KL, object quantity success rate, articulation collision ratio, reachability, and walkable-area success rate (Team et al., 29 Jul 2025, Chen et al., 9 Jan 2026). NeoWorld adds WorldScore-specific metrics such as CIQA+, 3D-Const, SceneQuality, and VideoCLIP (Zhao et al., 29 Sep 2025). The diversity of metrics reflects a deeper fact: “world quality” simultaneously involves visual fidelity, spatial consistency, action controllability, semantic alignment, and persistence under revisitation.

The limitations are correspondingly varied. QuadrupedGPT identifies a perception bottleneck from depth-camera noise and segmentation errors, persistent weakness in precise numerical grounding, and hallucination risk in high-level decomposition under highly novel conditions (Mei et al., 2024). ActWorld frames its central challenge as a combination of data bottleneck and memory bottleneck (Xiong et al., 16 Jun 2026). Infinite-World is motivated by noisy pose estimation and the scarcity of viewpoint revisits in real-world training videos (Wu et al., 2 Feb 2026). SceneFoundry reports a compute cost of approximately 300 s per 3-room apartment on an NVIDIA 3090, acknowledges that its articulation model is a heuristic box-expansion, and notes biases toward contemporary Western interior styles (Chen et al., 9 Jan 2026). WorldGrow is motivated by the limits of 2D lifting, the scalability problems of 3D implicit representations, and the object-centric bias of current 3D foundation models (Li et al., 24 Oct 2025).

A persistent research tension concerns how much of a world should be hard-coded, explicitly represented, or freely generated. Web World Models state this tension directly by positioning themselves between conventional web frameworks with reliable but fixed contexts and fully generative world models with unlimited environments but weaker controllability (Feng et al., 29 Dec 2025). This suggests that the most durable systems may continue to hybridize deterministic transition code, typed or geometric state, explicit memory structures, and model-driven semantic enrichment rather than treating unbounded generation as a purely end-to-end problem.