AndroidCode: Renderable GUI World Modeling

Updated 4 July 2026

AndroidCode is a corpus for training GUI world models that pairs Android screenshots with renderable HTML, offering a structured alternative to text or pixel-only approaches.
Its construction pipeline leverages GPT-5 for constrained HTML synthesis and a visual-feedback revision loop with a SigLIP threshold of 0.9 to ensure high fidelity.
AndroidCode underpins a two-stage Code2World training process, significantly enhancing GUI state prediction accuracy and downstream planning performance.

AndroidCode is a large-scale, high-fidelity corpus built from Android GUI trajectories to support code-native GUI world modeling. In "Code2World: A GUI World Model via Renderable Code Generation," it is defined as a synthetic training resource derived from Android Control that provides over 80K high-quality screen-action pairs, where the future UI state is represented as renderable HTML rather than only text or pixels. Within that framework, AndroidCode is the data layer that makes it possible to train a model to predict next-state code $\hat{C}_{t+1}$ from the current screen, action, and goal, and then obtain the predicted next screenshot by rendering that code, $\hat{I}_{t+1} = R(\hat{C}_{t+1})$ (Zheng et al., 10 Feb 2026).

1. Definition and functional role

AndroidCode is not presented as an Android source-code repository dataset, an APK benchmark, or an intermediate representation in its own right. It is presented as a corpus for training Code2World, a GUI world model that predicts the next visual state by generating renderable code. The motivating problem is explicit: existing GUI datasets provide screenshots, actions, and trajectories, but not clean HTML representations aligned with those screenshots. AndroidCode fills that gap by supplying paired supervision between Android GUI states and dependency-free HTML approximations of those states (Zheng et al., 10 Feb 2026).

The representation choice is central. The paper contrasts text-only and pixel-only GUI world modeling with a code-native alternative. Text-based approaches are said to capture semantic intent but to lack visual information and discard spatial-visual detail; pixel-based approaches are said to struggle with precise, discrete GUI transitions and with fine-grained structural controllability. AndroidCode is therefore designed around renderable code, with the claim that code preserves explicit structure while still permitting deterministic rendering. This suggests that AndroidCode is best understood as a corpus for learning structured visual state transitions rather than as a corpus of native Android implementation artifacts (Zheng et al., 10 Feb 2026).

Within the formal setup of Code2World, each training example supplies the supervision required for the mapping

$\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$

where $M_\theta$ is the multimodal generator and $R$ is a browser renderer. AndroidCode provides the aligned pairs needed to learn this code-generation objective (Zheng et al., 10 Feb 2026).

2. Construction pipeline

The construction pipeline begins from image-based GUI trajectories in Android Control and proceeds in two stages: constrained initial synthesis and revision with visual feedback. For each GUI screenshot, the authors use GPT-5 as a multimodal coder to synthesize corresponding HTML. That synthesis is tightly constrained: the output must be a complete HTML document from <!DOCTYPE html> to </html>, all visible content must be wrapped in a single root container <div id="render-target"> ... </div>, the viewport dimensions must match the screenshot exactly, and the result must be self-contained and independent of external assets (Zheng et al., 10 Feb 2026).

The synthesized code is then rendered back into an image and compared against the original screenshot using SigLIP cosine similarity. The acceptance threshold is $T = 0.9$ . Samples with SigLIP score greater than $0.9$ are retained directly; samples below that threshold enter a revision loop. The revision prompt takes as input the original screenshot $I$ , the rendered screenshot $\hat{I}$ , and the current HTML code $C$ , and instructs GPT-5 to compare the target and rendered images, identify mismatches, and fix them in priority order: layout or geometry, component sizing or spacing, typography, colors, shadows or borders, and fine details. The revision loop is selective and shallow: the maximum number of revision iterations is $\hat{I}_{t+1} = R(\hat{C}_{t+1})$ 0, and samples still below the threshold after that pass are discarded (Zheng et al., 10 Feb 2026).

The paper formalizes this procedure as "Automated Data Synthesis with Visual-Feedback Revision." In its notation, initial synthesis is

$\hat{I}_{t+1} = R(\hat{C}_{t+1})$ 1

followed by rendering and scoring,

$\hat{I}_{t+1} = R(\hat{C}_{t+1})$ 2

If $\hat{I}_{t+1} = R(\hat{C}_{t+1})$ 3, the code is revised using the tuple $\hat{I}_{t+1} = R(\hat{C}_{t+1})$ 4, rendered again, and rescored. Only pairs $\hat{I}_{t+1} = R(\hat{C}_{t+1})$ 5 satisfying $\hat{I}_{t+1} = R(\hat{C}_{t+1})$ 6 are retained. The result is described repeatedly as a corpus of over 80K high-quality screen-action pairs (Zheng et al., 10 Feb 2026).

3. Representation, modalities, and example structure

AndroidCode uses HTML as its renderable code representation. The paper consistently describes the representation as structured HTML or high-fidelity HTML, with no separate dependency on external CSS or JavaScript assets. The document-level format is fixed: a complete HTML file, a standardized root container #render-target, exact width and height matching the screenshot, position: relative, overflow: hidden, and a body with margin: 0, padding: 0, and background: transparent (Zheng et al., 10 Feb 2026).

The representation is intentionally approximate rather than native. Images are not reconstructed through external URLs; instead, uncertain images are replaced with semantic text placeholders such as [IMG: Red Shoe], typically rendered as gray boxes with thin borders and short labels. Icons are rendered as inline SVGs or simple Unicode when needed. The prompt further emphasizes exact font sizes, margins, padding, alignment, typography hierarchy, colors, shadows, border radii, and z-order. This makes AndroidCode a corpus of renderable structural surrogates for GUI states, not a corpus of Android XML layouts or application source files (Zheng et al., 10 Feb 2026).

At the example level, the paper describes AndroidCode as organized into multimodal triplets

$\hat{I}_{t+1} = R(\hat{C}_{t+1})$ 7

where $\hat{I}_{t+1} = R(\hat{C}_{t+1})$ 8 is the current screenshot, $\hat{I}_{t+1} = R(\hat{C}_{t+1})$ 9 is the user action, $\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$ 0 is the task goal, $\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$ 1 is the ground-truth next-state HTML, and $\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$ 2 is the ground-truth next screenshot. The visual input is augmented with action cues: a red circle for point interactions such as click or long press, and a red arrow for motion interactions such as scroll or swipe. The textual side includes instruction expansion, action description, user intent or goal, and action metadata. The paper references action categories including click, long_press, scroll, input_text, open_app, navigate_home, navigate_back, wait, and none, while noting that these categories appear across prompts and evaluation definitions rather than as an explicitly quantified class distribution for the dataset itself (Zheng et al., 10 Feb 2026).

The sequence budgets used for training reflect the corpus’ long-form multimodal nature. Appendix A.2 gives a cutoff length of 24,576 tokens for supervised fine-tuning, a maximum prompt length of 24,576 tokens for reinforcement learning, and a maximum response length of 8,192 tokens. This indicates that AndroidCode examples are meant to support substantial code-generation outputs rather than short labels or compact symbolic traces (Zheng et al., 10 Feb 2026).

4. Role in Code2World training

AndroidCode provides the supervision for the two-stage optimization of Code2World: supervised fine-tuning as a cold start, followed by Render-Aware Reinforcement Learning. The paper allocates 70% of AndroidCode to Stage 1 and 30% to Stage 2. In the first stage, the goal is to teach the model the syntax of HTML and the logic of UI layout by learning the symbolic mapping from $\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$ 3 to $\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$ 4. The backbone is Qwen3-VL-8B-Instruct, with full parameter fine-tuning on the LLM, frozen vision encoder and multimodal projector, 2 epochs, global batch size 64, learning rate $\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$ 5, cosine schedule, warmup ratio 0.1, and cutoff length 24,576 (Zheng et al., 10 Feb 2026).

The second stage uses the rendered consequence of generated code as the reward signal. If the model outputs code $\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$ 6, it is rendered as

$\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$ 7

The reward combines a visual semantic term and an action consistency term: $\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$ 8

$\hat{C}_{t+1} = M_\theta(I_t, a_t, g), \quad \hat{I}_{t+1} = R(\hat{C}_{t+1}),$ 9

$M_\theta$ 0

The implementation uses equal weighting for the two reward components. Optimization uses GRPO with group size $M_\theta$ 1, rollout batch size 16, sampling temperature 1.0, learning rate $M_\theta$ 2, and KL penalty coefficient $M_\theta$ 3 (Zheng et al., 10 Feb 2026).

The paper gives the normalized advantage for group outputs $M_\theta$ 4 as

$M_\theta$ 5

and the GRPO objective as

$M_\theta$ 6

where

$M_\theta$ 7

AndroidCode is therefore not merely a supervised corpus; it is the basis for both symbolic imitation and rendered-outcome reinforcement (Zheng et al., 10 Feb 2026).

5. Empirical significance

The value of AndroidCode is evidenced indirectly through the ablations of Code2World. On Android Control, the base Qwen3-VL-8B achieved $M_\theta$ 8, $M_\theta$ 9, $R$ 0, $R$ 1, SigLIP $R$ 2, and DINO $R$ 3. After supervised fine-tuning on AndroidCode, the same backbone improved to $R$ 4, $R$ 5, $R$ 6, $R$ 7, SigLIP $R$ 8, and DINO $R$ 9. The full Code2World system reached $T = 0.9$ 0, $T = 0.9$ 1, $T = 0.9$ 2, $T = 0.9$ 3, SigLIP $T = 0.9$ 4, and DINO $T = 0.9$ 5 on Android Control, and $T = 0.9$ 6, $T = 0.9$ 7, $T = 0.9$ 8, $T = 0.9$ 9, SigLIP $0.9$0, and DINO $0.9$1 on GUI Odyssey (Zheng et al., 10 Feb 2026).

The downstream agent results reinforce that AndroidCode is not merely a corpus for next-screen prediction in isolation. On AndroidControl-High, augmenting Mobile-Agent-v3-7B with Code2World increased success rate from 67.20 to 68.41 and grounding accuracy by +3.62; augmenting Qwen2.5-VL-7B increased success rate from 65.16 to 66.47 and grounding accuracy by +5.23. On AndroidWorld, the paper highlights that Gemini-2.5-Flash is boosted by +9.5%. An ablation with Gemini-2.5-Flash as the downstream agent reports 41.4% for the baseline, 42.6% with the base Qwen3-VL-8B simulator, 47.5% with SFT only, 49.2% with $0.9$2, 50.1% with $0.9$3, and 50.9% with full Code2World. This suggests that AndroidCode-derived training materially improves look-ahead simulation and planning, even before the full reinforcement stage is added (Zheng et al., 10 Feb 2026).

These results do not isolate AndroidCode as an independent benchmark variable, because the paper’s methodology treats the corpus as a prerequisite for the renderable-code paradigm. Still, the SFT jump from the base model to the AndroidCode-trained model is a direct empirical indication that the corpus encodes usable regularities of Android GUI state transitions (Zheng et al., 10 Feb 2026).

6. Scope, limitations, and relation to other Android code resources

AndroidCode is deliberately narrow in what it represents. It models Android GUI trajectories as synthesized HTML, not native Android code, XML layouts, APK bytecode, or repository histories. The paper states this implicitly through its construction and explicitly through its placeholder strategy: images and some icons are rendered as abstractions rather than preserved as original assets. A plausible implication is that some residual annotation noise may persist, because the corpus is generated automatically by GPT-5 and then filtered and lightly revised; the paper does not quantify such noise, but it does cap revision at $0.9$4, so low-scoring samples are discarded rather than repeatedly repaired (Zheng et al., 10 Feb 2026).

The representation also differs sharply from other Android-code artifacts studied in the literature. "AndroidCompass" is a historical repository dataset of 80,324 changed Build.VERSION.SDK_INT compatibility-check lines mined from 1,394 open-source projects, with one row per changed source line and explicit commit metadata (Nielebock et al., 2021). "ACVTool" instruments closed-source APKs at class, method, and instruction granularity by rewriting smali and collecting runtime coverage, while "AndroLog" instruments APKs through Soot/Jimple to obtain class-, method-, statement-, and Android component-level coverage without modifying the manifest (Pilgun et al., 2018, Samhi et al., 2024). "JuCify" unifies Dex and native call graphs into a Jimple-level model for cross-language static analysis, and "XAIDroid" represents APKs as API call graphs for malware detection and malicious code localization (Samhi et al., 2021, Ipek et al., 10 Mar 2025). In contrast, AndroidCode is neither a source-history corpus nor a bytecode-analysis substrate; it is a screen-action-to-renderable-code corpus for GUI world modeling.

This difference in scope is consequential. AndroidCode is useful for next-screen prediction, code-native GUI simulators, and simulator-assisted planning, but it does not provide the guarded method calls, repository metadata, control-flow graphs, smali traces, or native binaries that other Android analysis tasks require. Conversely, those other datasets and tools do not provide clean HTML reconstructions of GUI states aligned with actions and goals. The corpus therefore occupies a distinct position in the Android research landscape: it treats GUI state prediction as a code-generation problem, using HTML as the mediating representation between structure and visual fidelity (Zheng et al., 10 Feb 2026).