---
title: 'ClearFairy: AI Assistant for UI Design'
url: https://www.emergentmind.com/topics/clearfairy
type: topic
---

# ClearFairy: AI Assistant for UI Design

ClearFairy is a think-aloud AI assistant for UI design that operationalizes the CLEAR framework for capturing professionals’ decision-making during creative work by structuring reasoning into cognitive decision steps and augmenting those steps with weak explanation detection, in-situ questioning, and rationale inference. It is designed to make otherwise implicit decisions traceable by linking self-explanations, interaction events, and artifact snapshots, with the stated aim of supporting reflection, collaboration, knowledge sharing, and downstream generative AI systems [2509.14537].

## 1. Conceptual basis: the CLEAR framework

The underlying conceptual framework is CLEAR, which structures reasoning into **Cognitive Decision Steps (CDSes)**. A CDS is defined as a minimal unit of reasoning in a creative workflow and grounds three data streams: self-explanations, actions, and artifacts. Formally, at time \(t\),

$$
\mathrm{CDS}_{t} \;=\;\langle E_{t},\,A_{t},\,\mathrm{Art}_{t}\rangle,
$$

where \(E_t=\{e_{t,1},\dots,e_{t,n}\}\) is a sequence of utterances, \(A_t=\{a_{t,1},\dots,a_{t,m}\}\) is the set of the designer’s interaction events, and \(\mathrm{Art}_{t}=\{s_{t,1},\dots,s_{t,p}\}\) is the set of visual snapshots taken immediately after each action [2509.14537].

CDSes are formed by chunking linked explanations and actions according to three cognitively motivated linking rules: semantically coherent utterances associated with the same goal, successive identical or object-focused actions, and explicit cross-references between speech and action or artifact. This formulation makes the unit of capture neither a raw event stream nor a purely verbal protocol; instead, it is an aligned representation of reasoning, manipulation, and resulting state. A plausible implication is that CLEAR is intended to function as a representational intermediate between human design practice and machine-processable workflow records.

## 2. System architecture and processing pipeline

ClearFairy is implemented as a Figma plugin front end with a Python/Flask back end, and all LLM/VLM calls use GPT-4o with temperature \(=0.3\). Its real-time operation is organized into five stages: segmentation, rationale evaluation, question generation, inference, and documentation [2509.14537].

The first substantive stage is **Weak Explanation Detection**. For each \(\mathrm{CDS}_t\), explanation strength is classified as strong (\(\mathrm{S}\)), weak (\(\mathrm{W}\)), or empty (\(\mathrm{E}\)) using a few-shot LLM prompt:

$$
f_{\mathrm{eval}:\mathrm{CDS}_{t}\;\longrightarrow\;\{\mathrm{S},\,\mathrm{W},\,\mathrm{E}\}.
$$

The prompt injects definitions of six sub-categories—S-SR, S-PK, S-CA, W-SR, W-PK, W-CA, and E—together with examples. No additional fine-tuning is used.

The second stage is **In-Situ Questioning**. Whenever \(f_{\mathrm{eval}}(\mathrm{CDS}_t)\in\{\mathrm{W},\mathrm{E}\}\), a zero-shot question-generation prompt produces a clarifying question \(Q_t\) targeted at the missing rationale:

$$
Q_{t} \;=\; \mathrm{LLM}\bigl(\text{“Given explanation type = W/E, decision ‘why?’”}\bigr).
$$

The question is rendered as an overlay on the relevant canvas region using the action timestamp to locate it, and users may answer by voice when convenient.

The third stage is **Rationale Inference**. To reduce burden, ClearFairy mines prior strong rationales \(\{R_1,\dots,R_{t-1}\}\) and infers a candidate rationale \(\hat R_t\) for a new weak or empty step:

$$
\hat R_{t} = \arg\max_{r}\;P\bigl(R_{t}=r \,\big|\, \mathrm{Transcript},\,\{R_{i}\}_{i<t}\bigr).
$$

In implementation, this is approximated by a single GPT-4o call that attends to “similar past decisions” and “contextual cues,” after which users confirm or refine the inferred rationale.

The fourth stage is **Decision Step Documentation**. Once a strong rationale is available, either originally or after question answering and inference, an LLM/VLM summary module generates a human-readable summary for a web-based documentation view. The summaries include the key decision, actions, rationale, progression, screenshots, and any Q&A record. This documentation layer is significant because it converts transient design activity into a persistent record organized at the CDS level rather than at the level of isolated edits.

## 3. Explanation quality and rationale capture

A central technical problem addressed by ClearFairy is that creative professionals’ rationales are often incomplete and many decisions remain implicit. The system therefore focuses on detecting weak explanations and recovering missing rationale with minimal disruption to the creative flow [2509.14537].

For weak explanation detection, the reported performance is approximately \(80.2\%\) accuracy with \(\kappa=0.71\) against human judgment. This result is specifically attributed to zero/few-shot reasoning rather than fine-tuning. The evaluation target is not design quality per se, but explanation strength at the CDS level.

The qualitative logic of the pipeline is tightly coupled. Detection identifies where a CDS lacks a sufficiently articulated rationale; in-situ questioning elicits clarification at the moment of action; rationale inference attempts to reduce user burden by proposing a candidate explanation based on prior strong rationales and transcript context; and documentation records the resulting decision unit in a structured form. This suggests that the system treats explanation recovery as a progressive process rather than a single classification task.

One potential misconception is that a think-aloud assistant of this kind must necessarily externalize all reasoning through manual annotation. The reported design of ClearFairy is more selective: it intervenes when explanations are weak or empty and uses inferred rationales that users can confirm or refine. In that sense, the assistant is positioned not as a full protocol-transcription system but as a mechanism for targeted rationale completion.

## 4. Empirical evaluation with professional designers

The user study is a within-subjects evaluation involving \(N=12\) professional UI/UX designers. Each participant performed two 30-minute design tasks, one with ClearFairy and one without it under a baseline think-aloud condition, and the order was counterbalanced [2509.14537].

The principal quantitative outcome is a shift in the proportion of CDSes with strong explanations. In the baseline condition, only \(13.9\%\) of CDSes had strong explanations. With ClearFairy, strong explanations rose to \(83.2\%\). For rationale inference specifically, \(85\%\) of inferred rationales were accepted.

Workload was assessed with NASA-TLX. Question workload did not significantly increase under ClearFairy according to the Wilcoxon signed-rank test with \(p>0.05\). At the same time, measures of explanation clarity, articulation of implicit knowledge, and metacognitive insight all improved significantly with \(p<0.01\). The statistical formulation is reported as

$$
Z = \mathrm{WilcoxonSignedRank}\bigl(\text{Baseline\_scores},\;\text{CF\_scores}\bigr),\quad p<0.01.
$$

These results bear directly on a common concern in workflow-capture systems: that richer documentation may impose additional cognitive demand. The study reports improved explanatory outcomes without a significant increase in question workload. A plausible implication is that the combination of lightweight prompting and candidate rationale inference is intended to shift part of the articulation burden from the human participant to the system.

## 5. Use in generative AI agents and design simulation

The captured CDS records are not only documentation artifacts; they are also used as grounding for generative AI agents in Figma. One application is **next-step prediction**. The evaluation compares three kinds of LLM-generated next-step suggestions per scenario: task-prompt only, CDS history without rationale, and full CDS plus rationale. Across 60 trials, participants preferred the variant using full CDS plus rationale in \(66.7\%\) of cases, compared with \(16.7\%\) for CDS history without rationale and \(16.6\%\) for task-prompt only [2509.14537].

A second application is **design simulation via a tool-calling agent**. The system uses a ReAct-style Figma agent with 49 tool calls and conditions it on documented CDS to extract style guidelines such as spacing \(=32\) px, font \(=\) Inter-Bold, and color \(=\#000000\), which are prepended to the prompt. Outputs were judged more coherent and professionally aligned than a baseline without CDS in a level-wise evaluation spanning component, section, and page.

The paper explicitly states that overall automation remains a research prototype. At the same time, qualitative feedback reports that CDS grounding led to fewer hallucinations and more consistent styling. This frames ClearFairy less as a fully autonomous design system than as an infrastructure for human knowledge-grounded creative AI agents. The significance lies in the claim that rationale-bearing workflow traces are more useful to downstream agents than action history alone.

## 6. Dataset, records, and research uses

The released dataset contains 417 cognitive decision steps captured from the 12 participants. Each record contains an ID, transcript excerpt, action log entries with timestamps and element IDs, artifact screenshots, summary text, clarifying Q&A, and an inferred or user-confirmed rationale [2509.14537].

| Record component | Description |
|---|---|
| Transcript excerpt | Self-explanation segment associated with the CDS |
| Action log entries | Interaction events with timestamps and element IDs |
| Artifact screenshots | Visual snapshots after actions |
| Summary text | Human-readable CDS documentation |
| Clarifying Q&A | In-situ question and response record |
| Inferred or user-confirmed rationale | Final rationale attached to the decision step |

The stated potential uses include training workflow segmentation models, rationale classification, LLM-grounded planning for UI agents, educational resources for novices, and large-scale analysis of design practices. Because the dataset is organized around CDSes rather than raw logs alone, it offers aligned supervision for problems that combine temporal segmentation, language grounding, and action-context modeling.

More broadly, the dataset embodies the paper’s formalization of workflow capture:

$$
\mathrm{CDS}_{t}=\langle E_{t},A_{t},\Art_{t}\rangle,\quad
f_{\mathrm{eval}}(\mathrm{CDS}_{t})\in\{\mathrm{S},\mathrm{W},\mathrm{E}\},\quad
\hat R_{t}=\arg\max_r P(r\mid H_{t-1},\mathrm{Transcript}).
$$

This suggests a research program in which creative workflows are represented as sequences of decision units with explicit rationale quality labels and progressively refined explanatory content. Within that framing, ClearFairy is notable for linking in-situ capture, rationale completion, and agent grounding in a single system rather than treating them as separate problems.

Source: https://www.emergentmind.com/topics/clearfairy