ReDemon UI: Navigation & Web Synthesis

• ReDemon UI is a demonstration-driven framework that encompasses both learning UI navigation agents via macro actions and synthesizing React web components from sketches and demos.
• The navigation variant transforms raw UI inputs into structured elements, using macro actions and demonstration augmentation to efficiently train policies with DQfD or behavior cloning.
• The web synthesis variant leverages reactive data identification and an enumerative synthesis backbone to automatically generate complete, stateful React components from user-provided sketches and demos.

Searching arXiv for the specified ReDemon UI papers to ground the article in the cited literature. ReDemon UI is the name used in the arXiv literature for two distinct demonstration-centered systems. In one usage, it denotes an end-to-end framework for learning UI navigation agents from human demonstrations by simplifying perception to UI elements, constraining control to macro actions, and training policies with DQfD or behavior cloning (Li, 2021). In a later usage, it denotes a system for synthesizing React applications from a static JSX sketch with event-handler holes plus demonstrations of desired runtime behavior, using enumerative synthesis for simple UIs and LLMs for more complex UIs (Lee et al., 14 Jul 2025). The shared label can obscure the fact that the two systems target different problem classes: autonomous navigation across existing interfaces in the former case, and program synthesis of interactive web interfaces in the latter.

1. Dual usage of the term

The term ReDemon UI refers to two separate technical artifacts with different objectives, inputs, and outputs. The earlier system is framed around learning user-interface navigation agents from human demonstrations. Its output is a policy that acts in an environment of apps and websites. The later system is framed around reactive synthesis by demonstration for web UI. Its output is a stateful React component synthesized from a sketch and demonstrations (Li, 2021).

Variant	Primary goal	Output
ReDemon UI for UI navigation	Learn UI navigation agents from demonstrations	DQfD or BC policy over UI elements and macro actions
ReDemon UI for web UI	Synthesize React applications from demonstrations	Full, stateful React component

This dual usage is important because the methodological overlap is limited to the central role of demonstrations. In the navigation setting, demonstrations supervise decision-making in a pre-existing UI. In the web-synthesis setting, demonstrations serve as partial specifications for event-handler code and state update logic. A plausible implication is that the name identifies a research style—demonstration-driven automation—rather than a single unified codebase or benchmark lineage.

2. ReDemon UI as a framework for learning UI navigation agents

The UI-navigation formulation begins by replacing raw-pixel input with a structured state space. Each state $s \in S$ is a finite set of UI elements $E = \{ e_1, \ldots, e_n \}$, and each element is encoded as an embedding $$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$. The screen understanding pipeline converts raw pixels into these UI elements using OCR, icon detection, and accessibility tree or DOM extraction. The neural policy network is a transformer encoder over the element embeddings plus attention and multi-head output predicting $$(\text{element\_index}, \text{action\_type}, \text{action\_args})$$ (Li, 2021).

The action space is similarly structured. The agent selects a tuple $(i, a, \theta)$, where $i \in \{1 \ldots n\}$ is an element index, or $i=0$ for a global action; $a \in A_{\text{elem}} \cup A_{\text{glob}}$; and $\theta$ contains optional arguments. The element-level action set includes operations such as click, focus_and_type, and scroll, while the global set includes back, wait, and enter_key. This design sharply narrows the control problem relative to pixel-level RL: the policy does not discover low-level cursor or touch primitives directly, but selects semantically typed actions grounded in detected interface objects.

The framework is explicitly iterative. Its agent-building loop consists of: evaluating the current agent in the environment, recording failed cases, collecting corrective demonstrations, augmenting demonstrations, and retraining the policy by DQfD or behavior cloning. The demonstrations are not merely initial supervision; they are integrated into an error-driven development cycle in which “failed episodes” from the current policy become the source of new human data.

3. Macro actions, demonstration augmentation, and customized DQfD

A defining feature of the navigation system is the use of macro actions. A macro action $M$ is implemented as a small script of micro-actions with status predicates; from the agent’s viewpoint $E = \{ e_1, \ldots, e_n \}$0 is atomic and returns one of {SUCCESS, FAILURE, CANCELLED}. The example focus_and_type(text) macro executes click, waits until cursor_blink_detected(e_idx) or timeout, aborts with FAILURE if focus is not acquired, then sends key events, sends ENTER, waits for screen_changes() or timeout, and returns SUCCESS (Li, 2021).

Macro composition is also supported: macros can call other macros, and the paper gives the example that a “search” macro might call focus_and_type and then click on ResultItem if visible. This makes the action abstraction hierarchical even though the policy itself operates over atomic macro choices. The discussion section states that macro actions collapse multi-step interactions such as input focus, typing, and scroll-to-end into single decisions, drastically reducing episode length and state complexity; it further states that this yields orders-of-magnitude fewer training samples than pixel-level RL baselines.

The same paper introduces demonstration augmentation. For each demonstration state $E = \{ e_1, \ldots, e_n \}$1, irrelevant UI elements may have their text_embedding replaced with a random vector with probability $E = \{ e_1, \ldots, e_n \}$2, and their bounding-box coordinates perturbed by small random offsets with probability $E = \{ e_1, \ldots, e_n \}$3. In pseudocode, this is applied only to elements not in CriticalSet. The stated goal is to amplify $E = \{ e_1, \ldots, e_n \}$4 human demonstrations to $E = \{ e_1, \ldots, e_n \}$5 synthetic ones without extra human effort, and the paper attributes the effect to invariance to distractors and reuse of the same demonstration across $E = \{ e_1, \ldots, e_n \}$6 variations.

The DQfD objective is customized as

$E = \{ e_1, \ldots, e_n \}$7

where $E = \{ e_1, \ldots, e_n \}$8 is 1-step TD loss for Q-learning, $E = \{ e_1, \ldots, e_n \}$9 is n-step return loss, $$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$0 is supervised BC loss on demos, and

$$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$1

A further customization handles screenshot demos: for rare failure screenshots with no emulator state, the collected $$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$2 tuples contribute only supervised and margin loss, not TD loss, because no next-state transitions exist.

4. Training loop, benchmarks, and limitations of the navigation system

The training and evaluation protocol is organized as an approximately 10-iteration loop. In each iteration, the current agent is run on tasks in 80+ real apps/sites with random initial UI states and random view parameters including pixel density, font scale, orientation, and random clicks. All failed episodes are logged. A human demonstrator then replays failed cases using the macro-action GUI, collecting full trajectories plus screenshot corrections. Demonstrations are augmented by a factor of $$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$3, and the DQfD or BC policy is retrained from scratch or fine-tuned (Li, 2021).

The reported results are task-specific and concrete. On AndroidSearch, “≈425 human demos + 261 screenshot demos + augmentation + action-masking → 98.7% success.” The same summary states that without augmentation or screenshot demos, success drops to \<95%. On AndroidInstall, “16 demos + augmentation (×100) → 99% success vs. 95.6% without augmentation using 512 demos.” These figures are presented as evidence that the macro-action representation and augmentation strategy substantially reduce the required number of human demonstrations.

The paper also states several limitations and extensions. The agent is memory-less (no LSTM) and may repeat failing actions on intermittent UI unresponsiveness; a lightweight retry counter feature could be added. Scaling to multi-task settings with different utterances may require a hierarchical policy or natural-language grounding to sub-task macros. The current macro library is handcrafted, and future work could learn or refine macros end-to-end. These limitations indicate that the abstraction layer solving sample efficiency also fixes part of the system design space in advance.

5. ReDemon UI as reactive synthesis by demonstration for web UI

In the 2025 usage, ReDemon UI addresses a different problem: synthesizing React applications from user demonstrations. The motivating gap is drawn among traditional visual prototyping tools, hand-coding React applications, and prompt-based LLM UI generators. ReDemon UI addresses these limitations through a sketch–plus–demonstration workflow in which a designer provides a static JSX sketch with event-handler holes, demonstrates desired dynamic behaviors directly on the rendered mockup and by editing the sketch, and the system automatically infers reactive data and synthesizes correct React state-update logic (Lee et al., 14 Jul 2025).

The input representation is formalized around a static sketch with holes. The sketch is a JSX fragment, internally represented as SketchAST with a finite set of holes

$$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$4

The paper presents the syntax SketchAST ∈ Node ::= Element(tag, attrs, children) | Hole(id) | Text(str). Each hole $$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$51 is bound to a button’s onClick.

Demonstrations are recorded in demo mode, after the user “locks” the sketch. The system renders the JSX sketch into a live DOM and records both user actions and sketch edits in a timeline data structure

$$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$6

Each step consists of a user action on the rendered DOM and zero or more structural edits to the SketchAST. Internally, ReDemon UI keeps both the pre-action and post-edit states of the sketch and diffs them to identify exactly which parts of the AST or rendered tree changed.

The key inference mechanism is reactive data identification. For each timeline and each step, the system records S_before, applies the user action, records S_after after the structural edits, and computes $$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$7\Delta$$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$8v_i`. Formally, if $$x_i = \operatorname{concat}(\text{type\_embed}, \text{text\_OCR\_embed}, \text{icon\_embed}, \text{position\_embed}, \text{optional\_state})$$9 covers nodes $$(\text{element\_index}, \text{action\_type}, \text{action\_args})$$0, the system introduces

$$(\text{element\_index}, \text{action\_type}, \text{action\_args})$$1

and rewrites the AST so that each occurrence of $$(\text{element\_index}, \text{action\_type}, \text{action\_args})$$2 is bound to the expression {v}. This makes structural edits serve as evidence of latent state that must become explicit in React.

6. Synthesis backends, generated code, workflow integration, and evaluation

The synthesis backend proceeds in two stages. For simple UIs, ReDemon UI uses enumerative synthesis. For each hole $$(\text{element\_index}, \text{action\_type}, \text{action\_args})$$3i$$(\text{element\_index}, \text{action\_type}, \text{action\_args})$$4f_i$$(\text{element\_index}, \text{action\_type}, \text{action\_args})$$5G$$of React/JavaScript operators over the identified reactive variables plus the event payload. The summary gives examples such as <code>Expr ::= v | v + number | v.length | items.push(v) | …</code>. The enumeration strategy is breadth-first up to a small size bound such as depth$$(\text{element\_index}, \text{action\_type}, \text{action\_args})$6, with pruning by scope constraints and example-guided filtration: candidate expressions are simulated on every (action, Δ) pair in the demo timelines, and candidates that fail to reproduce the observed $(\text{element\_index}, \text{action\_type}, \text{action\_args})$7 are discarded. A simple ranking heuristic is

$(\text{element\_index}, \text{action\_type}, \text{action\_args})$8

with the smallest satisfying expression selected (Lee et al., 14 Jul 2025).

When enumeration times out, the system falls back to LLM-assisted synthesis using Gemini 2.0 Flash. The prompt includes a one-shot “counter” example and the user’s elaborated sketch with reactive-variable annotations and full demo timelines. The explicit instruction is to fill the event handlers so that replaying them on the sketch reproduces the demo behavior. The returned code is type-checked and may be replayed against the recorded demo timelines; if replay matches the recorded $(\text{element\_index}, \text{action\_type}, \text{action\_args})$$9 values at each step, the result is accepted, otherwise the user refines the demos or retries synthesis.</p> <p>Once handler functions are obtained, ReDemon UI assembles the final React component in either a class-based or hooks-based pattern. The class-based form initializes <code>this.state = { v₁: init₁, v₂: init₂, … }</code> and updates it through <code>this.setState(s ⇒ ({ v₁: f₁(s.v₁, e), … }))</code>. The hooks-based form introduces <code>const [v₁, setV₁] = useState(init₁);</code> and analogous handlers. In both cases, the code for each$$(i, a, \theta)$0 is inserted verbatim into the handler bodies, and the result is formatted and pretty-printed.

The workflow is integrated into prototyping practice through a Sketch pane, a Timelines pane, and a Synthesized pane. The sketch can be exported from Figma or hand-written; the timelines record demonstrations in the browser; the synthesized pane shows the complete React component for copy-paste into a codebase or handoff to front-end engineers. The loop is characterized as “lock → demo → synthesize → refine.” Evaluation results are explicitly preliminary: simple benchmarks such as increment/decrement counter and text-mirroring input were successfully synthesized in under 1 s via enumeration; medium benchmarks such as a to-do list with add/remove/complete operations required LLM fallback; the quantitative metrics reported are “Enumerative success rate on 5 counter-style examples: 100% within 2 s each” and “LLM success rate on 3 to-do list variants: 3/3 with one retry in 80% of cases.” The paper also records qualitative feedback that designers appreciated the “no-prompt” feel and developers found the generated code idiomatic and easy to integrate.

A recurring source of confusion is whether this web-synthesis system is a direct continuation of the 2021 navigation framework. The supplied literature does not present such a continuity claim. What it does show is a common reliance on demonstration as the supervisory medium, but applied to different substrates: policy learning over UI elements and macro actions in one case, and event-handler and state-update synthesis over SketchAST and demo timelines in the other.