---
title: 'DuetUI: Human-Agent Co-Creation Framework'
url: https://www.emergentmind.com/topics/duetui
type: topic
---

# DuetUI: Human-Agent Co-Creation Framework

Searching arXiv for DuetUI and closely related interface co-generation work.
arxiv_search.query({"search_query":"ti:\"DuetUI\" OR abs:\"DuetUI\"","start":0,"max_results":5})
DuetUI is a prototype and design framework for **human-agent co-generation of task-oriented interfaces**. It is proposed as an alternative to both full automation and static generated UIs for complex, multi-step real-world tasks in which user intent is vague, evolving, or only becomes clear during interaction. Its central mechanism is a **bidirectional context loop**: the agent scaffolds the interface by decomposing the task, while the user’s direct manipulations implicitly steer the agent’s next generation step. In this formulation, the interface is not merely an output artifact but the shared medium through which human intent and agent reasoning continuously shape each other [2509.13444].

## 1. Problem setting and conceptual orientation

DuetUI begins from three limitations in existing systems. First, **fully autonomous GUI agents** struggle with complex, multi-step, cross-app tasks because user intent is often vague and evolving. Second, **human-in-the-loop systems** improve control but remain largely **agent-centric**, with the agent proposing a plan and the human mainly supervising or correcting it. Third, **UI generation systems** frequently produce one-shot outputs or isolated mockups that are poorly matched to long task-oriented workflows spanning multiple services and requiring ongoing user input [2509.13444].

The paper frames these limitations in terms of two persistent gaps. The **gulf of execution** arises when users cannot easily translate their intent into the actions the AI system expects. The **gulf of evaluation** arises when users cannot easily judge what the system is doing or how to correct it. DuetUI addresses these problems by treating the interface itself as the shared communicative substrate between user and agent rather than as a terminal presentation layer.

A common misunderstanding is to read DuetUI as a conventional LLM UI generator. The paper argues for a different objective. The target is not a finished interface produced in one step, nor a rigid supervisory workflow in which the human only approves or rejects agent proposals. Instead, the intended mode is a **symbiotic process** in which an LLM-based agent and a user **co-create** an interface that evolves with the task. This suggests that DuetUI should be understood less as a static UI synthesis system and more as a dynamic mixed-initiative interaction framework.

## 2. Empirical basis: formative study and derived design goals

The conceptual design of DuetUI is grounded in a formative study with **12 participants** who used a design probe system to imagine and refine interfaces for **travel planning** and **relocating to a new city**. The probe system included a chatbot (**Gemini**) for Q&A and search, a UI prototyper (**dev0 / v0**), a notepad, and a drawing canvas. Participants designed ideal interfaces using think-aloud methods [2509.13444].

The study surfaced four findings. **F1** stated that intent is emergent and incrementally refined through interaction: participants did not begin with fully specified requirements, but instead started with a “basic motivation” and refined intent through intermediate outputs. **F2** found that opaque AI created a gulf of execution: participants described AI as a “black box” and were unsure what the agent could understand or how to control it. **F3** showed that user expressions entangle task and interface needs: utterances mixed goals such as search, bookmarking, summarization, and deletion, indicating that task structure and interface structure must be inferred jointly. **F4** found that users desired dynamic, mixed-initiative collaboration: they wanted AI assistance early under uncertainty but preferred taking over once sufficient context had been established.

From these findings, the paper derives four design goals: **DG1: Support a continuous and collaborative co-creation process**; **DG2: Achieve effortless instrumentality and bridge interaction gulfs**; **DG3: Enable task-oriented interface unfolding**; and **DG4: Foster mutual awareness for shared autonomy**. These design goals provide the rationale for the system’s staged interaction model, visible controls, structural task-to-interface mapping, and shared history mechanism. A plausible implication is that DuetUI treats ambiguity not as noise to be eliminated up front but as a central property of task-oriented interaction that must be handled through iterative interface evolution.

## 3. Bidirectional context loop and staged co-generation

The paper’s main conceptual contribution is the **bidirectional context loop**. Instead of asking the agent to generate a complete interface at once, DuetUI lets the **agent scaffold the task by decomposing it**, renders an interface from that decomposition, allows the **user to manipulate the interface directly**, and then interprets those manipulations as **implicit context** for the next generation step [2509.13444].

Operationally, the loop proceeds as follows: agent generation or update of task structure, rendering of the interface from that structure, user interaction with the interface, logging and interpretation of those actions, and subsequent revision of task and interface. The “bidirectional” designation refers to context flowing in both directions: from agent to interface and from user action back to agent. This is the mechanism by which direct manipulation becomes semantically meaningful for subsequent system behavior.

The staged model of co-generation consists of six stages: **Define**, **Empathize**, **Plan**, **Explore**, **Refine**, and **Duet**. The paper states that this is not a rigid sequence in practice, but a conceptual workflow. The stages move from high-level goal capture to final execution through increasingly interactive refinement.

The paper’s Barcelona travel example illustrates the model. In **Define**, the user says “I want to go to Barcelona.” The agent proposes clarifying options such as trip type. In **Empathize**, the system asks about travel mode and budget, and the user edits the structured profile directly. In **Plan**, the system generates subtasks such as flights, itinerary, and accommodation, after which the user reorders subtasks by drag-and-drop. In **Explore**, the system searches travel options based on preferences, while the user sets budget and lodging preferences. In **Refine**, the system updates results with filters and sorting, and the user selects a booking. In **Duet**, the system infers preferences from user behavior, recommends nearby attractions, and integrates booked items into a final itinerary.

This interaction pattern clarifies a second possible misconception: the user is not merely correcting an autonomous agent after the fact. Rather, ordinary interface actions are treated as a first-class input channel for steering future generations. The system therefore couples direct manipulation with mixed-initiative planning.

## 4. Design features, architecture, and formal structures

DuetUI operationalizes its paradigm through four design features. **Staged Co-Generation** replaces one-shot generation with an iterative workflow aligned with the six-stage model. **Tangible Agency** makes the agent’s capabilities concrete and manipulable through visible controls such as input fields, selections, action buttons, sliders, date pickers, dashboards, and navigation cards. **Task-Interface Duality** enforces a semantic mapping between task structure and interface structure. **Bidirectional Action History** maintains a shared log of actions from both agent and user, using visible interface changes for agent actions and explicit recording for user actions [2509.13444].

The architecture is organized as three layers with two operational loops. The **Core Layer** provides stateless infrastructure through **MemoryManager** for session persistence, **LLMManager** for LLM calls, prompts, retries, and JSON parsing, and **SchemaManager** for shared data structures using **Pydantic**. The **Context Layer** tracks the current **TaskStage**, maintains **TaskDecomposition** and **InterfaceDescription**, and stores **Bidirectional Action History**. The **Agent Layer** contains four specialized agents: **TaskAgent**, **InterfaceAgent**, **ServiceAgent**, and **RenderingAgent**, with the last rendering the interface using **Vue.js and Element Plus**.

The two operational loops separate task evolution from interface rendering. In the **Task Loop**, the **TaskAgent** reads the current task decomposition, action history, and current interface description, infers implicit user intent, updates the decomposition, and may invoke the **ServiceAgent**. In the **Interface Loop**, the **InterfaceAgent** reads the updated task decomposition and action history, generates a new interface description, and the **RenderingAgent** updates the UI accordingly. This separation is intended to keep the task plan and visual interface synchronized while preserving the ability of user actions to steer the process.

The paper also provides explicit schema-like definitions:

```latex
\begin{aligned}
\text{TaskDecomposition} &= (\text{goal}, \text{subtasks}) \\
\text{Subtask} &= (\text{subtask\_name}, \text{subtask\_id}, \text{step\_id}, \text{description}, \text{matched\_apis}, \text{dependent\_subtasks}, \text{page\_type}, \text{page\_state\_id})
\end{aligned}
```

```latex
\begin{aligned}
\text{Navigation} &\rightarrow \{\text{pageGroups}\} \\
\text{PageState} &\rightarrow (\text{sessionId}, \text{pageStateId}, \text{pageType}, \text{stateDetail}, \text{lastUpdated}) \\
\text{ComponentConfig} &\rightarrow \text{UI element configuration}
\end{aligned}
```

```latex
\text{Task} \leftrightarrow \text{Navigation}, \qquad
\text{Subtask} \leftrightarrow \text{Page}, \qquad
\text{Data} \leftrightarrow \text{Component}
```

These definitions show that DuetUI does not merely decorate a plan with widgets; it explicitly binds task decomposition to interface composition. The stated duality—**Task** ↔ **Navigation**, **Subtask** ↔ **Page**, **Data** ↔ **Component**—is the system’s central structural principle.

## 5. Evaluation and empirical findings

The evaluation used a **within-subjects study** with **24 participants** comparing **DuetUI** and **Stitch**. Each participant completed **3 tasks on DuetUI** and **3 tasks on Stitch**, yielding **6 sessions per participant** and **144 valid questionnaire sets**. Participants rated **SUS**, **NASA-TLX**, **Task Satisfaction**, **Interface Satisfaction**, and **AI Satisfaction** [2509.13444].

The principal quantitative results are summarized below.

| Measure | DuetUI | Stitch |
|---|---:|---:|
| SUS | 73.65 | 63.5 |
| NASA-TLX | 2.22 | 2.49 |
| Task Satisfaction | 3.7 | 3.25 |
| Interface Satisfaction | 3.96 | 3.54 |
| AI Satisfaction | 3.84 | 3.44 |

DuetUI significantly outperformed Stitch on **Usability**, **Task satisfaction**, **Interface satisfaction**, and **AI satisfaction**, while workload was **not significantly different overall**. The reported significance values were \( p = .0019 \) for usability, \( p = .0049 \) for task satisfaction, \( p = .0057 \) for interface satisfaction, and \( p = .0012 \) for AI satisfaction. Although overall workload differences were not statistically significant, DuetUI reduced **performance demand** within workload.

Qualitative feedback complements these measurements. Participants reported a **lower barrier to entry**, better **task efficiency**, better **alignment with their preferences**, clearer **AI interaction**, and more **effortless collaboration**. Interviews further indicated that users preferred **iterative dialogue over one-shot generation**, valued **predictability**, prioritized **logical structure** over visual polish, wanted **adaptive control**, and treated **trust** as foundational, particularly in the presence of hallucinations or incorrect information.

These findings support the paper’s claim that task-oriented co-generation can improve usability and perceived collaboration quality. At the same time, the results do not imply that workload is uniformly reduced; the paper states only that workload was not significantly different overall, with a reduction in performance demand.

## 6. Position within adjacent research, limitations, and future directions

DuetUI belongs to a broader design space of systems in which interface creation is shared between human intent and machine synthesis. In relation to demonstration-driven systems, a close comparison is **"ReDemon UI: Reactive Synthesis by Demonstration for Web UI"** [2507.10099]. ReDemon UI synthesizes React applications from a static mockup sketch with event-handler holes plus structured demonstrations of runtime behavior. Its workflow centers on demonstration timelines, diff-based identification of reactive parameters, and a synthesis backend that first uses enumerative synthesis and then falls back to Gemini 2.0 Flash for more complex React components. The comparison is informative because both systems reject prompt-only UI generation and instead rely on richer human input. However, the supplied description indicates an important distinction: ReDemon UI is a demonstration-driven partial-program completion system for reactive web UIs, whereas DuetUI is framed as a co-generative task-oriented interface system in which the interface unfolds alongside task progress.

The design rationale for DuetUI is explicitly tied to classic HCI concepts, including Norman’s **gulfs of execution/evaluation**, **direct manipulation**, **mixed-initiative interaction**, **shared autonomy**, and **man-computer symbiosis** [2509.13444]. This suggests that the paper is not only proposing a prototype but also positioning a conceptual model for human-agent collaboration in interface generation.

The paper is explicit about several limitations. The backend for service and data generation was **simulated rather than live**, which limits realism. The evaluation was a **short-term lab study**, so there is no longitudinal evidence about trust, learning, or sustained use. The tasks were **relatively well-defined**, meaning ambiguous or creative tasks were not deeply tested. Performance remains constrained by **LLM dependence** and hallucination risk. The participant sample was also **not fully representative of the general population**.

The future directions proposed in the paper include **more adaptive control**, with real-time inference of user expertise and desired autonomy level; **richer context**, incorporating sensor data, persistent memory, long-term user profiles, and broader situational awareness; extension **beyond GUI** to voice interfaces, AR, human-robot interaction, and accessibility-focused interfaces; **real-world tool integration** through live APIs and standardized tool-use protocols like **MCP**; and **longitudinal and diverse studies** with more varied users and more open-ended tasks. A plausible implication is that the long-term research agenda is not confined to interface rendering, but concerns the design of shared workspaces through which agents and users coordinate evolving intent, action, and control.

Source: https://www.emergentmind.com/topics/duetui