DuetUI: Human-Agent Co-Creation Framework

• DuetUI is a human-agent co-generation framework that uses a bidirectional context loop to iteratively decompose tasks into interface components.
• The design replaces one-shot UI generation with staged, mixed-initiative interactions that adapt to evolving and often vague user intent.
• Empirical studies show DuetUI outperforms alternatives in usability and satisfaction, highlighting its effectiveness for complex, multi-step tasks.

Searching arXiv for DuetUI and closely related interface co-generation work. arxiv_search.query({"^^^^^^^^3search_query3 OR abs:\3^^^^^^^^"DuetUI\"","start":^^^^3search_query3 DuetUI is a prototype and design framework for human-agent co-generation of task-oriented interfaces. It is proposed as an alternative to both full automation and static generated UIs for complex, multi-step real-world tasks in which user intent is vague, evolving, or only becomes clear during interaction. Its central mechanism is a bidirectional context loop: the agent scaffolds the interface by decomposing the task, while the user’s direct manipulations implicitly steer the agent’s next generation step. In this formulation, the interface is not merely an output artifact but the shared medium through which human intent and agent reasoning continuously shape each other (&&&^^^^3search_query3^^^^&&&).

^^^^3ti:\3^^^^. Problem setting and conceptual orientation

DuetUI begins from three limitations in existing systems. First, fully autonomous GUI agents struggle with complex, multi-step, cross-app tasks because user intent is often vague and evolving. Second, human-in-the-loop systems improve control but remain largely agent-centric, with the agent proposing a plan and the human mainly supervising or correcting it. Third, UI generation systems frequently produce one-shot outputs or isolated mockups that are poorly matched to long task-oriented workflows spanning multiple services and requiring ongoing user input (&&&^^^^3search_query3^^^^&&&).

The paper frames these limitations in terms of two persistent gaps. The gulf of execution arises when users cannot easily translate their intent into the actions the AI system expects. The gulf of evaluation arises when users cannot easily judge what the system is doing or how to correct it. DuetUI addresses these problems by treating the interface itself as the shared communicative substrate between user and agent rather than as a terminal presentation layer.

A common misunderstanding is to read DuetUI as a conventional LLM UI generator. The paper argues for a different objective. The target is not a finished interface produced in one step, nor a rigid supervisory workflow in which the human only approves or rejects agent proposals. Instead, the intended mode is a symbiotic process in which an LLM-based agent and a user co-create an interface that evolves with the task. This suggests that DuetUI should be understood less as a static UI synthesis system and more as a dynamic mixed-initiative interaction framework.

^^^^3 OR abs:\3^^^^. Empirical basis: formative study and derived design goals

The conceptual design of DuetUI is grounded in a formative study with ^^^^3ti:\3 OR abs:\3^^^^ participants who used a design probe system to imagine and refine interfaces for travel planning and relocating to a new city. The probe system included a chatbot (Gemini) for Q&A and search, a UI prototyper (dev^^^^3search_query3^^^^ / v^^^^3search_query3^^^^), a notepad, and a drawing canvas. Participants designed ideal interfaces using think-aloud methods (&&&^^^^3search_query3^^^^&&&).

The study surfaced four findings. F^^^^3ti:\3^^^^ stated that intent is emergent and incrementally refined through interaction: participants did not begin with fully specified requirements, but instead started with a “basic motivation” and refined intent through intermediate outputs. F^^^^3 OR abs:\3^^^^ found that opaque AI created a gulf of execution: participants described AI as a “black box” and were unsure what the agent could understand or how to control it. F3 showed that user expressions entangle task and interface needs: utterances mixed goals such as search, bookmarking, summarization, and deletion, indicating that task structure and interface structure must be inferred jointly. F4 found that users desired dynamic, mixed-initiative collaboration: they wanted AI assistance early under uncertainty but preferred taking over once sufficient context had been established.

From these findings, the paper derives four design goals: DG^^^^3ti:\3^^^^: Support a continuous and collaborative co-creation process; DG^^^^3 OR abs:\3^^^^: Achieve effortless instrumentality and bridge interaction gulfs; DG3: Enable task-oriented interface unfolding; and DG4: Foster mutual awareness for shared autonomy. These design goals provide the rationale for the system’s staged interaction model, visible controls, structural task-to-interface mapping, and shared history mechanism. A plausible implication is that DuetUI treats ambiguity not as noise to be eliminated up front but as a central property of task-oriented interaction that must be handled through iterative interface evolution.

3. Bidirectional context loop and staged co-generation

The paper’s main conceptual contribution is the bidirectional context loop. Instead of asking the agent to generate a complete interface at once, DuetUI lets the agent scaffold the task by decomposing it, renders an interface from that decomposition, allows the user to manipulate the interface directly, and then interprets those manipulations as implicit context for the next generation step (&&&^^^^3search_query3^^^^&&&).

Operationally, the loop proceeds as follows: agent generation or update of task structure, rendering of the interface from that structure, user interaction with the interface, logging and interpretation of those actions, and subsequent revision of task and interface. The “bidirectional” designation refers to context flowing in both directions: from agent to interface and from user action back to agent. This is the mechanism by which direct manipulation becomes semantically meaningful for subsequent system behavior.

The staged model of co-generation consists of six stages: Define, Empathize, Plan, Explore, Refine, and Duet. The paper states that this is not a rigid sequence in practice, but a conceptual workflow. The stages move from high-level goal capture to final execution through increasingly interactive refinement.

The paper’s Barcelona travel example illustrates the model. In Define, the user says “I want to go to Barcelona.” The agent proposes clarifying options such as trip type. In Empathize, the system asks about travel mode and budget, and the user edits the structured profile directly. In Plan, the system generates subtasks such as flights, itinerary, and accommodation, after which the user reorders subtasks by drag-and-drop. In Explore, the system searches travel options based on preferences, while the user sets budget and lodging preferences. In Refine, the system updates results with filters and sorting, and the user selects a booking. In Duet, the system infers preferences from user behavior, recommends nearby attractions, and integrates booked items into a final itinerary.

This interaction pattern clarifies a second possible misconception: the user is not merely correcting an autonomous agent after the fact. Rather, ordinary interface actions are treated as a first-class input channel for steering future generations. The system therefore couples direct manipulation with mixed-initiative planning.

4. Design features, architecture, and formal structures

DuetUI operationalizes its paradigm through four design features. Staged Co-Generation replaces one-shot generation with an iterative workflow aligned with the six-stage model. Tangible Agency makes the agent’s capabilities concrete and manipulable through visible controls such as input fields, selections, action buttons, sliders, date pickers, dashboards, and navigation cards. Task-Interface Duality enforces a semantic mapping between task structure and interface structure. Bidirectional Action History maintains a shared log of actions from both agent and user, using visible interface changes for agent actions and explicit recording for user actions (&&&^^^^3search_query3^^^^&&&).

The architecture is organized as three layers with two operational loops. The Core Layer provides stateless infrastructure through MemoryManager for session persistence, LLMManager for LLM calls, prompts, retries, and JSON parsing, and SchemaManager for shared data structures using Pydantic. The Context Layer tracks the current TaskStage, maintains TaskDecomposition and InterfaceDescription, and stores Bidirectional Action History. The Agent Layer contains four specialized agents: TaskAgent, InterfaceAgent, ServiceAgent, and RenderingAgent, with the last rendering the interface using Vue.js and Element Plus.

The two operational loops separate task evolution from interface rendering. In the Task Loop, the TaskAgent reads the current task decomposition, action history, and current interface description, infers implicit user intent, updates the decomposition, and may invoke the ServiceAgent. In the Interface Loop, the InterfaceAgent reads the updated task decomposition and action history, generates a new interface description, and the RenderingAgent updates the UI accordingly. This separation is intended to keep the task plan and visual interface synchronized while preserving the ability of user actions to steer the process.

The paper also provides explicit schema-like definitions:

\begin{aligned} \text{TaskDecomposition} &= (\text{goal}, \text{subtasks}) \ \text{Subtask} &= (\text{subtask\_name}, \text{subtask\_id}, \text{step\_id}, \text{description}, \text{matched\_apis}, \text{dependent\_subtasks}, \text{page\_type}, \text{page\_state\_id}) \end{aligned}

\begin{aligned} \text{Navigation} &\rightarrow \{\text{pageGroups}\} \ \text{PageState} &\rightarrow (\text{sessionId}, \text{pageStateId}, \text{pageType}, \text{stateDetail}, \text{lastUpdated}) \ \text{ComponentConfig} &\rightarrow \text{UI element configuration} \end{aligned}

\text{Task} \leftrightarrow \text{Navigation}, \qquad \text{Subtask} \leftrightarrow \text{Page}, \qquad \text{Data} \leftrightarrow \text{Component}

These definitions show that DuetUI does not merely decorate a plan with widgets; it explicitly binds task decomposition to interface composition. The stated duality—Task ↔ Navigation, Subtask ↔ Page, Data ↔ Component—is the system’s central structural principle.

5. Evaluation and empirical findings

The evaluation used a within-subjects study with ^^^^3 OR abs:\3^^^^4 participants comparing DuetUI and Stitch. Each participant completed 3 tasks on DuetUI and 3 tasks on Stitch, yielding 6 sessions per participant and ^^^^3ti:\3^^^^44 valid questionnaire sets. Participants rated SUS, NASA-TLX, Task Satisfaction, Interface Satisfaction, and AI Satisfaction (&&&^^^^3search_query3^^^^&&&).

The principal quantitative results are summarized below.

Measure	DuetUI	Stitch
SUS	73.65	63.5
NASA-TLX	^^^^3 OR abs:\3^^^^.^^^^3 OR abs:\3 OR abs:\3^^^^	^^^^3 OR abs:\3^^^^.49
Task Satisfaction	3.7	3.^^^^3 OR abs:\3^^^^5
Interface Satisfaction	3.96	3.54
AI Satisfaction	3.84	3.44

DuetUI significantly outperformed Stitch on Usability, Task satisfaction, Interface satisfaction, and AI satisfaction, while workload was not significantly different overall. The reported significance values were PRESERVED_PLACEHOLDER_^^^^3search_query3^^^^ for usability, PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^ for task satisfaction, PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^ for interface satisfaction, and $p = .0012$ for AI satisfaction. Although overall workload differences were not statistically significant, DuetUI reduced performance demand within workload.

Qualitative feedback complements these measurements. Participants reported a lower barrier to entry, better task efficiency, better alignment with their preferences, clearer AI interaction, and more effortless collaboration. Interviews further indicated that users preferred iterative dialogue over one-shot generation, valued predictability, prioritized logical structure over visual polish, wanted adaptive control, and treated trust as foundational, particularly in the presence of hallucinations or incorrect information.

These findings support the paper’s claim that task-oriented co-generation can improve usability and perceived collaboration quality. At the same time, the results do not imply that workload is uniformly reduced; the paper states only that workload was not significantly different overall, with a reduction in performance demand.

6. Position within adjacent research, limitations, and future directions

DuetUI belongs to a broader design space of systems in which interface creation is shared between human intent and machine synthesis. In relation to demonstration-driven systems, a close comparison is "ReDemon UI: Reactive Synthesis by Demonstration for Web UI" (Lee et al., 14 Jul 2025). ReDemon UI synthesizes React applications from a static mockup sketch with event-handler holes plus structured demonstrations of runtime behavior. Its workflow centers on demonstration timelines, diff-based identification of reactive parameters, and an overview backend that first uses enumerative synthesis and then falls back to Gemini ^^^^3 OR abs:\3^^^^.^^^^3search_query3^^^^ Flash for more complex React components. The comparison is informative because both systems reject prompt-only UI generation and instead rely on richer human input. However, the supplied description indicates an important distinction: ReDemon UI is a demonstration-driven partial-program completion system for reactive web UIs, whereas DuetUI is framed as a co-generative task-oriented interface system in which the interface unfolds alongside task progress.

The design rationale for DuetUI is explicitly tied to classic HCI concepts, including Norman’s gulfs of execution/evaluation, direct manipulation, mixed-initiative interaction, shared autonomy, and man-computer symbiosis (&&&^^^^3search_query3^^^^&&&). This suggests that the paper is not only proposing a prototype but also positioning a conceptual model for human-agent collaboration in interface generation.

The paper is explicit about several limitations. The backend for service and data generation was simulated rather than live, which limits realism. The evaluation was a short-term lab study, so there is no longitudinal evidence about trust, learning, or sustained use. The tasks were relatively well-defined, meaning ambiguous or creative tasks were not deeply tested. Performance remains constrained by LLM dependence and hallucination risk. The participant sample was also not fully representative of the general population.

The future directions proposed in the paper include more adaptive control, with real-time inference of user expertise and desired autonomy level; richer context, incorporating sensor data, persistent memory, long-term user profiles, and broader situational awareness; extension beyond GUI to voice interfaces, AR, human-robot interaction, and accessibility-focused interfaces; real-world tool integration through live APIs and standardized tool-use protocols like MCP; and longitudinal and diverse studies with more varied users and more open-ended tasks. A plausible implication is that the long-term research agenda is not confined to interface rendering, but concerns the design of shared workspaces through which agents and users coordinate evolving intent, action, and control.