---
title: 'FronTalk: Multi-modal Front-end Code Generation'
url: https://www.emergentmind.com/topics/frontalk
type: topic
---

# FronTalk: Multi-modal Front-end Code Generation

FronTalk refers to a benchmark and research framework designed to advance the study of front-end code generation in conversational contexts, with a distinctive emphasis on multi-turn, multi-modal feedback. Unlike traditional code-generation benchmarks centered around text-only instructions and static outputs, FronTalk systematically incorporates visual artifacts (e.g., sketches, mockups, annotated screenshots) alongside textual instructions, thereby capturing the richer dynamics and iterative nature of real-world front-end development processes [2601.04203].

## 1. Motivation and Scope

Front-end development uniquely relies on incremental user-driven refinements and densely interleaved textual and visual feedback. "Vibe coding" and AI-powered assistant workflows often manifest as iterative dialogue in which developers specify UI changes via both natural language and gestural or annotated visual cues. Prior code-generation benchmarks, typically single-turn and exclusively text-based, lack support for:

- Multi-turn integration of incremental design modifications
- Multi-modal instruction semantics, i.e., instructions simultaneously conveyed as language and visual intent

FronTalk addresses this gap by curating a dataset and evaluation suite capturing real-world conversational coding phenomena in diverse application domains (news, finance, art, etc.) [2601.04203].

## 2. Dataset Construction and Dialogue Curation

FronTalk consists of 100 seed websites, sampled from the C4 corpus and spanning content-rich domains such as news, e-commerce, financial dashboards, and digital art portfolios. For each site, a 10-turn dialogue is constructed, generating a total of 1,000 conversational turns.

Key dataset specifications include:

- **Dialogue count**: 100
- **Turns per dialogue**: $T=10$
- **Total test cases**: 3,676 ($\approx 3.7$ per turn)
- **Average user intent length**: 93.7 words/turn, 936.8 words/dialogue
- **Validity**: 1.7% of hallucinated cases removed, 73.4% edited for a final 94.7% validity rate

Each turn contains:

- **Textual instruction**: Simulated by an LLM (adapted to the current UI/code state)
- **Visual instruction**: Generated by a vision-language model (VLM) with matplotlib-based sketching and annotation, matching the intent of the textual instruction

Both modalities express the same user intent, enabling systematic evaluation of cross-modal understanding and generation.

## 3. Multi-Turn, Multi-Modal Interaction Design

Each FronTalk dialogue is formalized as a sequence of user intents $\{\mathbf{i}_t\}_{t=1}^T$, mapped to associated test-case sets $\mathcal{C}(\mathbf{i}_t)$. At dialog turn $t$:

1. The user simulator adapts the static intent to the current UI/code context, forming $\tilde{\mathbf{i}}_t = \mathrm{SimUser}(\mathbf{i}_t | \mathbf{H}_t)$, with $\mathbf{H}_t$ representing the full conversational and output history.
2. The code-generation agent $\mathcal{M}$ produces code $o_t = \mathcal{M}(\tilde{\mathbf{i}}_t | \mathbf{H}_t)$.
3. After $T$ turns, the output $o_T$ is rendered to produce the final website.

This design enables the systematic study of feature retention, history-dependent reasoning, and sensitivity to both text and visual UI cues across extended, realistic interaction chains.

## 4. Agent-Based Evaluation Framework

FronTalk introduces an agent-based protocol leveraging a web exploration agent, based on the WebVoyager environment, to quantitatively and qualitatively assess code-generation outputs:

### Metrics

- **Functional Correctness**: For each turn, pass count $PC(o | \mathbf{i})$ is computed as the number of test-cases passed by rendering $o$ on $\mathcal{C}(\mathbf{i})$. Pass rate $PR(o)$ aggregates this across all test-cases.
- **Forgetting Rate**: Measures overwriting of prior features:
  $$
  FR = 1 - \frac{\sum_{t=1}^{T-1} PC(o_T | \mathbf{i}_t)}{\sum_{t=1}^{T-1} PC(o_t | \mathbf{i}_t)} \times 100\%
  $$
- **Usability Evaluation**: First-time user simulation explores the final site, proposing real-world tasks and completing flows. For each run, LLM judges perform pairwise preference comparisons against reference implementations to compute win rates reflecting user experience.

### Actions Available to the Agent

- Click, Scroll, Hover, Type, Compare, ViewAnimation, inspect screenshots, and parse textual DOM trees, supporting automated, robust, and reproducible interaction with generated sites.

## 5. Experimental Results and Insights

An extensive empirical study across 20 models (7 text-only LLMs, 6 proprietary VLMs, 7 open-source VLMs) yields the following key findings:

- **Closed-source VLMs outperform open-source:** On text-only instructions, proprietary models (e.g., GPT-4o, Gemini-2.5-Pro) achieve up to 75% pass rate, while the best open-source models reach only 62.5%.
- **Higher visual feedback gap:** Pass rate differential between proprietary and open-source models is 24.1% for visual feedback (vs. 12.5% for textual).
- **Forgetting is pervasive:** $FR$ varies between 4.3% and 46.3%, leading to up to 46% feature loss after ten conversational turns due to overwritten or ignored prior requirements.
- **Visual instruction challenges:** Open-source VLMs frequently replicate visual arrangements without implementing associated logic, neglect text annotations, and misinterpret subtle cues.
- **Usability:** Lower-capacity models struggle with navigation and completion of common user flows; high-capacity models omit UX best practices (e.g., accessible, responsive navigation).

## 6. AceCoder: Mitigating Forgetting via Critique-and-Revise

To counteract the model forgetting phenomenon, FronTalk introduces AceCoder—a baseline leveraging web agent feedback at every turn, structured as follows:

1. Generate tentative code $o_t = \mathcal{M}(\mathbf{H}_t, \tilde{\mathbf{i}}_t)$.
2. For each previous intent $\mathbf{i}_j$ ($j\le t$), the agent checks if $o_t$ still passes all associated test-cases, generating explanations for failures.
3. If any failures are detected, the model is re-prompted with the set of failure explanations $\mathcal{F}$ appended.

AceCoder reduces the forgetting rate to nearly zero and increases textual instruction pass rate by up to 9.3% (from 56.0% to 65.3%). For visual feedback, improvements are +5.2%. This protocol generalizes across different LLMs and VLMs [2601.04203].

## 7. Broader Implications and Future Research Directions

FronTalk defines the methodological standard for studying multi-turn, multi-modal conversational code generation by establishing:

- The necessity of robust long-context modeling: preserving and integrating all prior features and requirements over extended conversational interactions.
- The challenge of visual instruction interpretation: closing the gap between superficial layout mimicry and semantically faithful, functional UI implementation.
- The limitation of current open-source VLMs in handling design cues, textual annotations, and implicit design heuristics relative to proprietary alternatives.

Key future directions include advancing VLM generalization to multi-modal dialogue, enhancing long-context memory architectures for code generation, and integrating human-centered design priors into conversational agent frameworks. All code, data, and evaluation materials are publicly available at https://github.com/shirley-wu/frontalk, facilitating reproducibility and further research into the frontiers of conversational UI coding [2601.04203].

Source: https://www.emergentmind.com/topics/frontalk