FronTalk: Multi-modal Front-end Code Generation

• FronTalk is a benchmark and research framework advancing front-end code generation by modeling iterative, multi-modal dialogues that blend text and visual cues.
• It curates a dataset of 100 websites with 10-turn dialogues per site, enabling rigorous testing of feature retention and dynamic interaction in UI coding.
• The framework introduces AceCoder to mitigate feature forgetting, enhancing both functional correctness and usability through agent-based evaluation protocols.

FronTalk refers to a benchmark and research framework designed to advance the study of front-end code generation in conversational contexts, with a distinctive emphasis on multi-turn, multi-modal feedback. Unlike traditional code-generation benchmarks centered around text-only instructions and static outputs, FronTalk systematically incorporates visual artifacts (e.g., sketches, mockups, annotated screenshots) alongside textual instructions, thereby capturing the richer dynamics and iterative nature of real-world front-end development processes (Wu et al., 5 Dec 2025).

1. Motivation and Scope

Front-end development uniquely relies on incremental user-driven refinements and densely interleaved textual and visual feedback. "Vibe coding" and AI-powered assistant workflows often manifest as iterative dialogue in which developers specify UI changes via both natural language and gestural or annotated visual cues. Prior code-generation benchmarks, typically single-turn and exclusively text-based, lack support for:

• Multi-turn integration of incremental design modifications
• Multi-modal instruction semantics, i.e., instructions simultaneously conveyed as language and visual intent

FronTalk addresses this gap by curating a dataset and evaluation suite capturing real-world conversational coding phenomena in diverse application domains (news, finance, art, etc.) (Wu et al., 5 Dec 2025).

2. Dataset Construction and Dialogue Curation

FronTalk consists of 100 seed websites, sampled from the C4 corpus and spanning content-rich domains such as news, e-commerce, financial dashboards, and digital art portfolios. For each site, a 10-turn dialogue is constructed, generating a total of 1,000 conversational turns.

Key dataset specifications include:

• Dialogue count: 100
• Turns per dialogue: $T=10$
• Total test cases: 3,676 ($\approx 3.7$ per turn)
• Average user intent length: 93.7 words/turn, 936.8 words/dialogue
• Validity: 1.7% of hallucinated cases removed, 73.4% edited for a final 94.7% validity rate

Each turn contains:

• Textual instruction: Simulated by an LLM (adapted to the current UI/code state)
• Visual instruction: Generated by a vision-LLM (VLM) with matplotlib-based sketching and annotation, matching the intent of the textual instruction

Both modalities express the same user intent, enabling systematic evaluation of cross-modal understanding and generation.

3. Multi-Turn, Multi-Modal Interaction Design

Each FronTalk dialogue is formalized as a sequence of user intents $\{\mathbf{i}_t\}_{t=1}^T$, mapped to associated test-case sets $\mathcal{C}(\mathbf{i}_t)$. At dialog turn $t$:

1. The user simulator adapts the static intent to the current UI/code context, forming $$\tilde{\mathbf{i}}_t = \mathrm{SimUser}(\mathbf{i}_t | \mathbf{H}_t)$$, with $\mathbf{H}_t$ representing the full conversational and output history.
2. The code-generation agent $\mathcal{M}$ produces code $$o_t = \mathcal{M}(\tilde{\mathbf{i}}_t | \mathbf{H}_t)$$.
3. After $T$ turns, the output $\approx 3.7$0 is rendered to produce the final website.

This design enables the systematic study of feature retention, history-dependent reasoning, and sensitivity to both text and visual UI cues across extended, realistic interaction chains.

4. Agent-Based Evaluation Framework

FronTalk introduces an agent-based protocol leveraging a web exploration agent, based on the WebVoyager environment, to quantitatively and qualitatively assess code-generation outputs:

Metrics

• Functional Correctness: For each turn, pass count $\approx 3.7$1 is computed as the number of test-cases passed by rendering $\approx 3.7$2 on $\approx 3.7$3. Pass rate $\approx 3.7$4 aggregates this across all test-cases.
• Forgetting Rate: Measures overwriting of prior features:

$\approx 3.7$5

• Usability Evaluation: First-time user simulation explores the final site, proposing real-world tasks and completing flows. For each run, LLM judges perform pairwise preference comparisons against reference implementations to compute win rates reflecting user experience.

Actions Available to the Agent

• Click, Scroll, Hover, Type, Compare, ViewAnimation, inspect screenshots, and parse textual DOM trees, supporting automated, robust, and reproducible interaction with generated sites.

5. Experimental Results and Insights

An extensive empirical study across 20 models (7 text-only LLMs, 6 proprietary VLMs, 7 open-source VLMs) yields the following key findings:

• Closed-source VLMs outperform open-source: On text-only instructions, proprietary models (e.g., GPT-4o, Gemini-2.5-Pro) achieve up to 75% pass rate, while the best open-source models reach only 62.5%.
• Higher visual feedback gap: Pass rate differential between proprietary and open-source models is 24.1% for visual feedback (vs. 12.5% for textual).
• Forgetting is pervasive: $\approx 3.7$6 varies between 4.3% and 46.3%, leading to up to 46% feature loss after ten conversational turns due to overwritten or ignored prior requirements.
• Visual instruction challenges: Open-source VLMs frequently replicate visual arrangements without implementing associated logic, neglect text annotations, and misinterpret subtle cues.
• Usability: Lower-capacity models struggle with navigation and completion of common user flows; high-capacity models omit UX best practices (e.g., accessible, responsive navigation).

6. AceCoder: Mitigating Forgetting via Critique-and-Revise

To counteract the model forgetting phenomenon, FronTalk introduces AceCoder—a baseline leveraging web agent feedback at every turn, structured as follows:

1. Generate tentative code $\approx 3.7$7.
2. For each previous intent $\approx 3.7$8 ($\approx 3.7$9), the agent checks if $\{\mathbf{i}_t\}_{t=1}^T$0 still passes all associated test-cases, generating explanations for failures.
3. If any failures are detected, the model is re-prompted with the set of failure explanations $\{\mathbf{i}_t\}_{t=1}^T$1 appended.

AceCoder reduces the forgetting rate to nearly zero and increases textual instruction pass rate by up to 9.3% (from 56.0% to 65.3%). For visual feedback, improvements are +5.2%. This protocol generalizes across different LLMs and VLMs (Wu et al., 5 Dec 2025).

7. Broader Implications and Future Research Directions

FronTalk defines the methodological standard for studying multi-turn, multi-modal conversational code generation by establishing:

• The necessity of robust long-context modeling: preserving and integrating all prior features and requirements over extended conversational interactions.
• The challenge of visual instruction interpretation: closing the gap between superficial layout mimicry and semantically faithful, functional UI implementation.
• The limitation of current open-source VLMs in handling design cues, textual annotations, and implicit design heuristics relative to proprietary alternatives.

Key future directions include advancing VLM generalization to multi-modal dialogue, enhancing long-context memory architectures for code generation, and integrating human-centered design priors into conversational agent frameworks. All code, data, and evaluation materials are publicly available at https://github.com/shirley-wu/frontalk, facilitating reproducibility and further research into the frontiers of conversational UI coding (Wu et al., 5 Dec 2025).