Code-as-World Paradigm

Updated 31 August 2026

Code-as-World is a paradigm where executable code functions as a generative, stateful, and inspectable representation of a world, specifying entities, geometry, rules, and observations.
The paradigm allows for the creation of various worlds including virtual, physical, graphical, or artistic, facilitating the representation of complex systems in a controllable and verifiable manner.
It uses various layers such as entities, state, rules, transitions, and verification that enable simulation, testing, and control and can serve as a backbone for applications such as embodied AI, robotics, and creative coding.

Code-as-World is a paradigm in which executable code functions as a generative, stateful, and inspectable representation of a world rather than merely as an implementation hidden behind a finished artifact. In this paradigm, programs may specify entities, geometry, spatial relations, physical parameters, affordances, transition rules, observations, persistence, and verification procedures. Natural-language intentions, perceptual observations, or agent actions are compiled into executable world representations that can be rendered, simulated, edited, tested, and revised. The resulting world may be virtual, physical, graphical, computational, industrial, or artistic. Its defining property is not that everything is represented in code, but that selected aspects of reality become executable, stateful, controllable, and subject to machine-mediated verification.

1. Concept and scope

Code-as-World distinguishes several functions of code that are often conflated. Code as world representation specifies what exists, how entities are related, and how state evolves. Code as action constitutes an intervention on an environment, such as a file edit, shell command, GUI event, robot skill, or simulator control. Code as observation interface exposes the consequences of actions through traces, diagnostics, rendered frames, test results, or sensor outputs. Code as harness provides the surrounding runtime, tools, memory, permissions, validators, sandboxes, and feedback loops that make an agent’s interaction persistent and governable (Ning et al., 18 May 2026).

A code-defined world can contain several layers:

Entities: objects, variables, files, pointers, agents, NPCs, physical bodies, GUI elements, or memory locations.
State: positions, attributes, inventories, variables, object identity, event history, geometry, or repository contents.
Rules: assignments, API effects, collision laws, navigation constraints, reward functions, workflow policies, or physical equations implemented by an engine.
Transitions: changes induced by actions, events, interpreter steps, tool calls, or numerical integration.
Observations: screenshots, videos, execution traces, compiler diagnostics, test outputs, rendered scenes, or sensor data.
Persistence: databases, caches, files, event logs, deterministic seeds, versioned programs, or retained simulator state.
Verification: tests, static analysis, runtime diagnostics, collision checks, visual comparison, physics analysis, or human review.

The world need not be fully autonomous or physically realistic. “Steps towards prompt-based creation of virtual worlds” demonstrates a three-party co-creation loop in which a human supplies intent, a LLM generates code or semantic consequences, and a Unity runtime materializes the result (Roberts et al., 2022). WorldCoder instead uses executable Python programs as learned transition and reward models for planning in symbolic environments (Tang et al., 2024). Other systems use web code, renderable HTML, declarative scene specifications, Blender programs, physics simulators, or creative-coding repositories as world substrates (Feng et al., 29 Dec 2025, Zheng et al., 10 Feb 2026, Sun et al., 14 Jan 2026, Wang et al., 19 May 2026, Baudry et al., 26 Jun 2026).

The paradigm therefore includes both explicit simulators and narrower executable representations. A program becomes a world model when it represents relevant state and dynamics, can be executed to produce consequences, persists across interactions, and exposes observations suitable for comparison with expectations. A program that merely issues an action without modeling environmental state is better characterized as an interface or policy language.

2. Historical and conceptual development

Early prompt-based virtual-world systems treated LLMs as generators of scene-editing code and semantic gameplay rules. In Codex VR Pong, players transform balls and paddles into arbitrary 3D objects, while GPT-3 generates open-ended semantic collision outcomes such as salmon plus knife producing sushi or water plus fire producing steam. Codex generates Unity C# code for scene changes; GPT-3 generates textual consequences; Sketchfab supplies meshes; and Unity performs instantiation, physics, animation, and networking (Roberts et al., 2022).

A distinct line of work represents the environment itself as executable code. WorldCoder asks an LLM to synthesize and revise Python programs implementing transition and reward functions. A conventional planner then executes those programs to imagine futures. The model is updated through counterexamples when its predicted transition or reward disagrees with the environment. Its optimism constraint requires the synthesized model both to fit observed data and to expose a modeled path to positive reward (Tang et al., 2024).

Subsequent systems broadened the substrate:

Web World Models use ordinary web code, typed interfaces, deterministic procedural generation, caches, and databases to define persistent but open-ended worlds. LLMs generate narrative and semantic content above a code-controlled physics layer (Feng et al., 29 Dec 2025).
Code2World represents GUI transitions as generated HTML whose deterministic browser rendering produces predicted screenshots. This makes DOM structure, layout, controls, and overlays explicit and potentially editable (Zheng et al., 10 Feb 2026).
World Craft compiles natural-language intent into semantic topology, structured scene data, and executable 2D or tile-based environments through a World Scaffold and a multi-agent World Guild (Sun et al., 14 Jan 2026).
SceneCode represents indoor objects as executable Blender Python programs that generate part-wise geometry, materials, collision proxies, and articulated simulation assets (Wang et al., 19 May 2026).
VisualPatchWorld induces compact transition programs from state-action traces and uses them inside model-predictive control (Bai et al., 28 Jul 2026).
Code World Model separates persistent world evolution implemented by a coding agent from visual realization produced by a video model (Chen et al., 26 Aug 2026).
Coding Agent Is Good As World Simulator uses executable Project Chrono programs as physics-based world representations and combines code generation, visual review, physics analysis, and iterative repair (Wang et al., 14 May 2026).
Code-as-World: Executable World Representations for Physical Reasoning explicitly formulates an executable world representation containing physical composition, dynamic evolution, and visual appearance (Wang et al., 27 Aug 2026).

Creative coding supplies a broader cultural interpretation. Processing, p5.js, openFrameworks, SuperCollider, TouchDesigner, and related systems use code as a medium for generative images, sound, interaction, performance, education, and installation. A large-scale study of 1,613,571 repositories shows that code constructs not only aesthetic outputs but also communities, curricula, creative industries, and preservation infrastructures (Baudry et al., 26 Jun 2026).

3. Representational architectures

Executable transition programs

An executable transition program represents the world through a function that maps current state and action to a next state. In VisualPatchWorld, an object-centric scene graph $g_t$ is transformed according to:

$g_{t+1}=f_\theta(g_t,a_t).$

The program may update coordinates, apply proportional-derivative control, gate object motion on grasp state, reconstruct articulated motion through forward kinematics, or implement contact-mediated dynamics. VPW selects a qualitative dynamical form through active probes and then fits its numerical parameters using multi-step rollout error. The resulting program is executed inside model-predictive control (Bai et al., 28 Jul 2026).

WorldCoder uses a related but more symbolic representation. Its transition model $\hat T$ and context-conditioned reward model $\hat R$ are separate Python subroutines. The transition program represents dynamics such as turning, movement, pushing, pickup, door opening, and collision constraints. The reward program represents task-specific success and termination. This separation permits reuse of dynamics across natural-language goals (Tang et al., 2024).

Declarative scene specifications

World Craft represents a scene as:

$\mathcal{G}=(M,A,L,P),$

where $M$ is metadata, $A$ is the asset set, $L$ is the layered layout, and $P$ contains physical, navigational, semantic, and interaction properties. The representation is primarily declarative rather than a general-purpose programming language. Scaffold interprets the specification to construct a playable scene with navigation meshes and interaction logic (Sun et al., 14 Jan 2026).

Typed properties encode rudimentary affordances. A wall may be solid and a navigation obstacle; a door may be passable and a walkable_door; an object may be an agent, NPC, tile, or ordinary object. The schema distinguishes visual size from floor footprint, allowing appearance and spatial occupancy to be represented separately.

Part-wise object programs

SceneCode extends executable representation to object structure. An object is produced by a Blender Python program composed of named semantic parts. Each part may have a primitive type, local transform, material, symmetry tag, and movable flag. Articulated components are preserved as separate links and compiled into revolute or prismatic joints, URDF/SDF-compatible structures, and simulator-loadable assets (Wang et al., 19 May 2026).

This representation differs from an opaque mesh in three respects. It preserves semantic decomposition, allows local regeneration, and retains the source program linking the request, geometry, rendering, and simulation artifacts. A drawer can remain an independent link; a cabinet can retain shelves and interior structure; a plant can expose parameters controlling repeated leaves or subdivisions.

Code-defined computational environments

In industrial software, the world may consist of repositories, compilers, simulators, profilers, testbenches, memory layouts, linker scripts, and hardware-specific constraints. InCoder-32B-Thinking’s Industrial Code World Model predicts backend observations from an environment bundle and candidate code. Observations include compilation errors, simulator logs, numerical differences, geometry reports, memory faults, and performance-related results (Yang et al., 3 Apr 2026).

CWM uses a neural rather than symbolic representation. It is trained on serialized Python interpreter traces and agentic Docker trajectories. A source line or shell action serves as an intervention; local-variable states, tool outputs, file contents, test results, and errors serve as observations. Its world model is implicit in a Transformer rather than exposed as a separate symbolic simulator (team et al., 30 Sep 2025).

Renderable code

Code2World predicts HTML from a current screenshot, action, and goal:

$C_{t+1}=M_\theta(I_t,a_t,G),\qquad \hat I_{t+1}=R(C_{t+1}),$

where $g_{t+1}=f_\theta(g_t,a_t).$ 0 is browser rendering. The representation simultaneously preserves visual appearance and structural organization. HTML elements, CSS layout, containment, buttons, lists, input fields, and overlays remain separately identifiable, while the rendered image provides visual grounding (Zheng et al., 10 Feb 2026).

The system’s Render-Aware Reinforcement Learning uses both visual-semantic fidelity and action consistency. A rendered screen that resembles the target but is not a plausible consequence of the specified action is penalized. This makes code a prospective GUI state representation rather than merely a format for reproducing screenshots.

Proxy-mediated visual worlds

Code World Model uses executable state for persistent evolution and a proxy representation for visual generation. The coding agent maintains entities, relations, event history, rules, schedules, resources, and consequences. A deterministic proxy compiler converts selected executable state into frame-wise spatial and temporal constraints. A video model then renders high-fidelity observations conditioned on the proxy and structured text (Chen et al., 26 Aug 2026).

The proxy deliberately omits textures, materials, detailed lighting, production-quality geometry, and fine skeletal animation. It encodes what must remain spatially and temporally controlled, such as entity trajectories, camera motion, approximate scale, occlusion-relevant geometry, and interaction-critical state.

4. Construction, execution, and verification

Code-as-World systems commonly use a closed loop:

$g_{t+1}=f_\theta(g_t,a_t).$ 1

The representation may be produced by an LLM, a multi-agent system, active probing, parameter fitting, reverse engineering, or direct human authoring.

Planning and semantic grounding

World Craft uses an Enricher to transform natural-language descriptions into coordinate-free semantic topology, a Manager to ground topology into a structured layout, a Critic to identify collisions and connectivity failures, and an Artist to synthesize or retrieve visual assets (Sun et al., 14 Jan 2026). SceneCode uses a room-level planner–designer–critic loop to generate AssetRequests before routing objects to construction strategies such as StaticFurn, StructManip, SimpleManip, WallArt, or Artic (Wang et al., 19 May 2026).

The intermediate representation is important because natural language typically specifies relations rather than coordinates. “A laptop on a table facing the chair” must be converted into support, orientation, scale, and placement constraints before it can become executable geometry.

Runtime execution

Execution may occur in Unity with Roslyn-compiled C#, Python interpreters, Docker environments, browsers, Blender, MuJoCo, Project Chrono, or web runtimes. The runtime is authoritative in many architectures: it performs collision detection, numerical integration, rendering, asset instantiation, networking, test execution, or simulator stepping.

In the physics-based framework of Coding Agent Is Good As World Simulator, generated programs define bodies, joints, contacts, terrain, sensors, cameras, materials, and numerical settings. Project Chrono produces physical trajectories, rendered frames, contacts, logs, and solver diagnostics. The code is an executable causal hypothesis, but physical correctness remains dependent on generated parameters, assets, collision approximations, and the simulator itself (Wang et al., 14 May 2026).

Feedback and repair

Verification can be static, dynamic, visual, physical, or semantic:

Static validation checks imports, method calls, schemas, signatures, or unsupported APIs.
Execution validation checks runtime errors, compiler failures, tests, and simulator logs.
Geometric validation checks collisions, support, dimensions, connectivity, and clearance.
Visual validation compares rendered images with reference images or intended layouts.
Physical validation checks trajectories, contacts, stability, settlement, and task-specific measurements.
Semantic validation checks whether generated content satisfies the prompt or intended interaction.
Human validation evaluates prompt faithfulness, realism, usability, or safety.

Code2World constructs AndroidCode by rendering synthesized HTML, comparing it with screenshots using SigLIP, and revising low-scoring examples (Zheng et al., 10 Feb 2026). SceneCode executes Blender programs, returns tracebacks for runtime repair, renders objects, and performs critic-driven refinement (Wang et al., 19 May 2026). WorldCoder sends counterexample transitions and failed plans to the LLM for program repair (Tang et al., 2024).

Planning with executable models

Once a code-defined model exists, it can be queried repeatedly without invoking a LLM for every step. WorldCoder uses depth-limited value iteration over synthesized Python dynamics. VisualPatchWorld rolls executable programs forward inside CEM-MPC. Code2World uses a Propose–Simulate–Select procedure in which candidate GUI actions are rendered prospectively before one is executed (Tang et al., 2024, Bai et al., 28 Jul 2026, Zheng et al., 10 Feb 2026).

A hybrid strategy may use code for broad search and a ground-truth engine for selective verification. VPW reports that verifying only the top fraction of candidate plans with MuJoCo can close much of the gap between an induced program and the full engine, particularly for contact-rich pushing (Bai et al., 28 Jul 2026).

5. Applications

Virtual worlds and games

Prompt-based Unity systems demonstrate scene editing, object replacement, voice interaction, multiplayer synchronization, physics, and stochastic gameplay. Codex VR Pong separates controlled player transformations from model-generated semantic consequences, producing interactions that were not explicitly enumerated by the developer (Roberts et al., 2022).

The limitation is that the resulting world depends on external assets, runtime compilation, asset metadata, network services, and prompt quality. Static meshes are easier to generate than articulated or semantically graspable objects, and stochastic outcomes complicate testing, reproducibility, and multiplayer synchronization.

Web and GUI environments

Web World Models use TypeScript, HTTP handlers, schemas, procedural generators, databases, caches, and renderers to define persistent travel atlases, fictional galaxies, games, encyclopedias, simulations, and narrative worlds (Feng et al., 29 Dec 2025). Their design separates deterministic state transitions from LLM-generated descriptions, dialogue, lore, missions, and explanations.

Code2World applies the same principle to GUI simulation. HTML provides an editable structured state, while browser rendering produces visual observations. The model improves downstream GUI navigation when used as a look-ahead simulator, but incorrect predictions can create false safety cues for irreversible actions such as payments or deletion (Zheng et al., 10 Feb 2026).

Embodied AI and robotics

SceneCode generates simulation-ready indoor scenes with articulated objects. Its programs preserve part hierarchy, movable components, joint metadata, collision geometry, and inertial estimates, enabling assets such as cabinets and drawers to be loaded into MuJoCo (Wang et al., 19 May 2026).

VisualPatchWorld learns executable dynamics for navigation, reaching, grasping, and contact pushing. Its results indicate that the qualitative form of a dynamical law can be more important for planning than low one-step prediction error. A model that represents joint-space dynamics, grasp gating, or contact-mediated motion can outperform a more superficially accurate but structurally incorrect model (Bai et al., 28 Jul 2026).

Industrial software and hardware

Industrial code world models treat compilers, simulators, profilers, testbenches, CAD systems, embedded environments, and hardware limits as executable environments. InCoder-32B-Thinking uses real execution traces and error-driven correction trajectories across chip design, GPU optimization, embedded systems, compiler optimization, and 3D modeling (Yang et al., 3 Apr 2026).

CWM extends this approach to Python interpreter traces and containerized repositories. Its reported trace evaluations show high state and action exact-match performance, while its agentic data model repository edits, shell commands, test results, and environment feedback (team et al., 30 Sep 2025).

CodeSense provides a benchmark for whether models can reconstruct fine-grained semantics from real Python, C, and Java projects. It evaluates statement, block, function, loop, branch, pointer-aliasing, and abstract-value reasoning using execution traces (Roy et al., 31 May 2025). The benchmark indicates that current models remain unreliable at reconstructing operational semantics even when they can recognize familiar code patterns.

Physical reasoning

The executable-world representation in Code-as-World: Executable World Representations for Physical Reasoning separates physical composition, dynamic evolution, and visual appearance. Text or video observations are converted into hypotheses, which are compiled, simulated, rendered, compared against evidence, and iteratively refined (Wang et al., 27 Aug 2026).

The resulting executable worlds supply exact geometry, trajectories, camera parameters, and physical labels for training quantitative physical reasoning. Code is useful because it makes counterfactual interventions possible: changing mass, initial velocity, camera configuration, or physical parameters allows the same world to be resimulated under altered conditions.

Art and creative coding

Creative coding treats algorithms as aesthetic systems rather than merely as production tools. Processing and p5.js programs can define images, sound, games, interactive installations, data visualizations, sensor-driven experiences, and autonomous processes. A p5.js sketch may produce a single static composition, a stochastic family of images, a webcam-mediated experience, or an interactive game (Baudry et al., 26 Jun 2026).

The study of 1,613,571 repositories shows that code-based creative practices are globally distributed and embedded in education, galleries, studios, performance, research, and open-source communities. In this context, Code-as-World includes social and institutional worlds: repositories preserve artworks, organize curricula, connect collaborators, and encode the infrastructures through which digital culture is produced.

6. Evaluation, limitations, and open problems

Evaluation dimensions

A Code-as-World system cannot be evaluated solely by whether a final image looks plausible or a final task succeeds. Relevant dimensions include:

State fidelity: whether the represented state corresponds to the actual environment.
Transition fidelity: whether actions produce correct state changes.
Long-horizon stability: whether errors compound over extended rollouts.
Visual fidelity: whether rendered observations match intended or observed appearances.
Physical consistency: whether contacts, dynamics, geometry, and constraints are coherent.
Semantic coherence: whether entities and consequences preserve identity and meaning.
Controllability: whether users or agents can reliably obtain intended changes.
Editability: whether local changes can be made without reconstructing the entire world.
Reproducibility: whether identical code, state, and seeds yield comparable outcomes.
Safety: whether generated actions, assets, rules, and outputs remain within acceptable boundaries.
Maintainability: whether programs remain compatible with APIs, dependencies, and engine versions.
Cost and latency: whether generation, execution, rendering, and verification are practical.

CodeSense demonstrates the importance of fine-grained semantic evaluation for code-defined worlds. Its results show that exact final outputs can conceal failures in intermediate state, control flow, loops, pointer relations, and API behavior (Achour et al., 11 Jun 2025). CWM similarly distinguishes local trace accuracy from long-horizon trajectory agreement, since one incorrect observation can redirect an entire agent trajectory (team et al., 30 Sep 2025).

Representation limitations

Code-as-World systems are constrained by the scope of the code-defined substrate. A symbolic grid-world program does not directly solve visual perception or continuous control. A web application does not automatically model real-world physics. A Blender program may preserve articulation while producing approximate mass, inertia, or collision geometry. A video proxy may control trajectories while leaving contact mechanics and identity appearance to a learned generator.

Many systems also depend on external services and libraries. Unity, Sketchfab, speech recognition, networking interfaces, Blender, PyChrono, MuJoCo, browser APIs, Docker images, compiler versions, and specialized toolchains can all introduce failure modes or prevent reproducibility.

Underdetermination and causal validity

Observed behavior rarely determines a unique world model. A short video may be consistent with multiple masses, friction coefficients, forces, camera configurations, or transition mechanisms. A code model may fit observed traces while failing on counterfactual inputs. An LLM may predict compiler feedback from correlations without representing the underlying hardware mechanism.

VPW addresses structural underdetermination through active probes and separates qualitative model selection from parameter fitting (Bai et al., 28 Jul 2026). WorldCoder uses optimism over programs consistent with replay data (Tang et al., 2024). These mechanisms improve exploration or planning, but neither establishes a complete posterior over possible worlds.

Persistence and shared state

Persistent worlds require stable identity, versioning, event histories, caches, and synchronization. Deterministic procedural generation can provide stable locations and object identities without storing every object, as in Web World Models (Feng et al., 29 Dec 2025). Multi-agent harnesses require stronger mechanisms: read and write sets, permissions, semantic merges, rollback, re-verification, and conflict explanations (Ning et al., 18 May 2026).

State synchronization becomes especially difficult when multiple agents edit shared repositories, when summaries become stale, or when one agent’s belief state diverges from authoritative runtime state. Code provides a shared substrate but does not automatically solve distributed-systems problems.

Safety and governance

Executable representations can make actions more controllable, but they also make generated behavior operational. The relevant safeguards include sandboxing, least privilege, schema validation, static analysis, test gates, human approval, audit logs, rollback, and explicit uncertainty handling.

The distinction between a valid program and a valid world model is essential. Executable code may be syntactically correct while physically unstable, semantically inappropriate, insecure, or harmful. Verification itself may be incomplete, and a model-generated prediction can produce false confidence when the action is irreversible.

Open research directions

Major open problems include:

General-purpose world discovery: learning executable representations without manually specified hypothesis families or domain-specific templates.
Multimodal state estimation: integrating images, video, depth, tactile signals, GUI structure, traces, and physical measurements.
Probabilistic executable worlds: representing stochastic dynamics, uncertainty, partial observability, and multiple consistent programs.
Long-horizon fidelity: preventing compounding errors in neural simulators, code rollouts, and agentic trajectories.
Rich physical dynamics: supporting fluids, cloth, fracture, combustion, elasticity, deformation, and articulated contact.
Repository-scale semantics: modeling concurrency, external services, operating-system state, memory behavior, and distributed execution.
Automatic library learning: extracting reusable classes, skills, mechanisms, abstractions, and subroutines from prior worlds.
World-model planning: integrating executable models with search, model-predictive control, reinforcement learning, and uncertainty-aware action selection.
Verification beyond final success: evaluating intermediate safety, causal validity, maintainability, and behavior under counterfactual interventions.
Human and multi-agent governance: preserving shared state consistency, provenance, permissions, oversight, and regression-free evolution.

Code-as-World is therefore best understood as an architectural and representational thesis: worlds become more controllable when their relevant entities, state, dynamics, observations, and constraints are expressed through executable artifacts. The paradigm does not imply that code replaces perception, learned visual models, physical reality, or human interpretation. Rather, it establishes an intermediate layer in which those components can be connected through programs that are persistent, inspectable, editable, simulatable, and verifiable.