ReFinE: Streamlining UI Mockup Iteration with Research Findings

Abstract: Although HCI research papers offer valuable design insights, designers often struggle to apply them in design workflows due to difficulties in finding relevant literature, understanding technical jargon, the lack of contextualization, and limited actionability. To address these challenges, we present ReFinE, a Figma plugin that supports real-time design iteration by surfacing contextualized insights from research papers. ReFinE identifies and synthesizes design implications from HCI literature relevant to the mockup's design context, and tailors this research evidence to a specific design mockup by providing actionable visual guidance on how to update the mockup. To assess the system's effectiveness, we conducted a technical evaluation and a user study. Results show that ReFinE effectively synthesizes and contextualizes design implications, reducing cognitive load and improving designers' ability to integrate research evidence into UI mockups. This work contributes to bridging the gap between research and design practice by presenting a tool for embedding scholarly insights into the UI design process.

• The paper presents a Figma plugin that automates extracting, translating, and visualizing HCI research into actionable UI design iterations.
• It integrates multimodal LLMs for context extraction, semantic retrieval with cosine similarity, and clustered synthesis to generate design recommendations.
• Technical evaluations show significant workload reduction and faster, higher-quality design edits, validated through controlled user studies.

ReFinE: Systematically Translating HCI Research into Actionable UI Mockup Iteration

Introduction and Motivation

The persistent disconnect between research findings in human-computer interaction (HCI) and real-world UI design practice undermines the practical uptake of scholarly design implications. While HCI research routinely produces robust, actionable insights relevant for UI design, designers face significant challenges in locating, interpreting, and contextually applying these findings within their digital prototyping workflows. Problems include poor discoverability of relevant literature, difficulties in apprehending technical jargon, the lack of actionable guidance, and minimal contextual translation between research and specific design instances.

ReFinE (Research Findings for Evidence-informed UI design iteration) directly targets this translational science bottleneck. Presented as a Figma plugin, ReFinE automates the pipeline from evidence retrieval, research implication synthesis, and context-aware actionable recommendation, to direct instantiation and iterative visualization in the designer's current mockup. The contribution lies in a tightly integrated multimodal LLM pipeline that spans extraction of design context, semantic retrieval from preprocessed HCI paper repositories, analogical translation of design implications, cluster-based synthesis of guidance, and visual application of recommendations through robust HTML mockup reconstructions. This architecture supports real-time, evidence-backed iteration during design prototyping.

System Architecture and Pipeline

ReFinE is composed of several modular components that enable end-to-end translational support:

1. Design Context Extraction: The plugin analyzes selected Figma mockups, inferring six key design context dimensions (target user, domain, modality, pain point, client, and evaluation metric) via visual prompting of a multimodal LLM.
2. Scholarly Paper Indexing: The backend preprocesses a curated ACM CHI paper corpus, extracting the same dimensions from each paper via GROBID/XML parsing and LLM-based context inference. Vector embeddings are created for robust dimension-centric retrieval.
3. Design Implication Extraction: A rule-driven LLM parses each paper to identify and extract granular design implications, derived from the core study sections and mapped to precise HCI taxonomies [sas2014generating].
4. Semantic Retrieval and Clustering: The system retrieves topically relevant papers by cosine similarity between aggregated context embeddings and forms clusters of aligned design implications via hierarchical clustering, optimizing cluster count using mean silhouette scores.
5. Cluster-level Analogy and Translation: For each cluster, ReFinE synthesizes similarities/differences between the paper contexts and the designer's scenario, then generates analogical, context-specific actionable insights, which are formulated into concise calls-to-action.
6. Action Item Generation and Visualization: Using the designer's mockup image and cluster output, ReFinE produces targeted action items, each mapped to exact screen(s). For representable edits, the system reconstructs a manipulable HTML/CSS version of the mockup and applies DOM edits using an edit-only approach, yielding highly efficient and semantically faithful visualizations.

   Figure 1: Pipeline for generating the components provided by ReFinE.

Technical Evaluation

Efficiency and Fidelity

Key technical choices are grounded in quantitative benchmarking:

• HTML Reconstruction: Among Gemini 2.0 Flash, Claude 3.5 Sonnet, and GPT-4o, Gemini achieves optimal trade-off between latency (~11s per screen) and high visual fidelity ($0.79$ cosine similarity via ViT embeddings).
• Input Modalities: Raw mockup images yield higher reconstruction fidelity and lower latency than JSON or image+JSON combinations.
• Visualization: The edit-only DOM modification strategy reduces mean action item visualization latency from $10.25$ to $2.91$ seconds without statistically significant change in output accuracy.
• Context Extraction: The system achieves $\approx$95% accuracy in extracting dimensions from both papers and mockups, with strong inter-rater reliability ($\kappa$ in $[0.82,0.88]$).

Action Item Relevance

• Annotator ratings on a 5-point scale indicate that 86% of generated action items are at least moderately relevant to their source cluster implications and are accurately mapped to the appropriate screens.

Example Workflow and Outputs

A prototypical interaction with ReFinE begins with the designer selecting mockup screens in Figma; the system then presents clustered evidence-backed insights, highlights similarities/differences with the current context, translates relevant HCI implications, surfaces actionable action items, and visualizes edit proposals directly over a reconstructed HTML version of the mockup. Designers can rapidly toggle between before/after views and bookmark action items for later application.

Figure 2: Contents of the cluster and their formulation.

Figure 3: Example of mockup reconstruction in our pipeline, demonstrating semantic and structural fidelity to the original design, suitable for action item visualization.

Figure 4: Visualization of an action item application, supporting instant designer comprehension and iterative application.

Figure 5: Design edit examples showing the extracted design context, retrieved design implications, action items, and the resultant before/after screen pairs.

User Study: Effects on Designer Behavior

A within-subjects study ($N=12$ professional/students) compared ReFinE to a baseline of manual literature search (ACM DL and LLM-based summaries). Quantitative and behavioral data, combined with semi-structured interviews, produced the following empirical findings:

• Workload Reduction: ReFinE reduces cognitive, temporal, and physical workload across all NASA-TLX dimensions with high statistical significance ($p<.01$, $p<.001$).
• Quality of Insights: Communication of design implications in ReFinE scores significantly higher on relevance, generativity, inspirability, actionability, validity, and generalizability versus baseline, with no significant loss of perceived originality.
• Iteration Speed and Volume: Participants achieved more than double the design edits (mean 5.5 vs. 2.4) within equal time using ReFinE, driven by more rapid and productive evidence discovery.

  Figure 6: Perceived workload, communication quality, and iteration outcome metrics illustrating the impact of ReFinE vs. baseline design workflows. Error bars denote standard error.

• Interpretability and Learnability: Visual instantiation of action items was critical for rapid designer understanding and creative adaptation. Despite minor discrepancies in HTML reconstructions, all users reported semantic sufficiency for evaluating action suggestions.
• Clustered Insight Navigation: Users reported high trust in clustered implication groups and requested support for enhanced scannability (e.g., tags/keywords), as well as interface improvements like condensing or hiding sources by default.

  Figure 7: Example of a designer creatively reinterpreting an action item across screens and components.

  

  Figure 8: Sample mockups used in the controlled user study setting.

Implications and Limitations

Practical Impact

ReFinE significantly lowers barriers to the integration of scholarly HCI research into the practical design process, allowing for evidence-driven iteration without compromising designer autonomy or creativity. Visual, real-time evidence guidance is particularly impactful given designers’ preferences for visual over textual information.

Systemic Biases and Scope

A recurring bias was observed toward additive (rather than subtractive or simplification) recommendations. This likely reflects both the predominance of prescriptive, constructive design implications in the literature and LLMs' transformation tendencies. System design may need to balance or explicitly surface reductionist insights.

Generalization

Though evaluated in the mobile UI context and with a CHI-constrained corpus, the architecture is extensible to other interface modalities (web, wearable, etc.) and can incorporate additional paper sources and design context dimensions.

Future Directions

Prospective enhancements include support for dynamic/interactivity-based edits, designer-in-the-loop feedback to iteratively refine action items, surfacing unresolved implication tensions within clusters, and leveraging organizational/design intent documents to further personalize recommendations. Integration with native WYSIWYG editors and direct HTML export is also a promising avenue.

Conclusion

ReFinE demonstrates robust efficacy in automating the translational pipeline from HCI research to UI design practice. It affords designers contextualized research discovery, actionable and visually instantiated recommendation, and accelerated prototyping grounded in empirical evidence. These results underline the feasibility and utility of AI-powered translational pipelines for bridging research-practice gaps in design-oriented disciplines.