PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation

Abstract: User interface (UI) and user experience (UX) evaluation is central to product development, yet reliable feedback still relies on recruiting human participants or running online A/B tests, making early-stage iteration slow and costly. In light of this, recent work has explored Multimodal LLMs as proxy evaluators. However, existing approaches either produce surface-level critiques or a judgment that reflects the model's own biases rather than the genuine response of a particular user. We introduce PerceptUI, a framework for persona-conditioned UI/UX evaluation that predicts how a specific user would answer interface-related questions and produces natural-language rationales. PerceptUI is trained in two stages: (i) contrastive reflection fine-tuning distills teacher-generated rationales by extracting lessons from human decisions, and (ii) a reflective prompt-evolution step from the model's own failure traces. Across multiple domains and datasets, PerceptUI achieves human-level realism, generalizes to unseen questions and personas, and yields population-level response distributions.

• The paper introduces PerceptUI, a framework that employs contrastive fine-tuning and reflective prompt evolution to simulate persona-based UI/UX evaluations.
• It demonstrates significant improvements in accuracy, calibration, and rationale quality over traditional MLLM approaches using targeted evaluations.
• Empirical results across multiple datasets validate its capacity to generate realistic user feedback and population-level insights for interface design.

PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation

Motivation and Framework Design

Effective evaluation of UI/UX design typically necessitates direct feedback from users, either through studies or online A/B tests. Current automatic approaches leveraging MLLMs either generate generic critiques or predict preferences that reflect the inherent biases of those models, failing to capture the contextual and user-dependent nature of interface perception. The "PerceptUI" framework introduces a persona-conditioned approach: given a UI screenshot, a user persona description, and an evaluation question, it predicts how that user would respond and generates a rationale for the answer. The system is trained in two stages: contrastive reflection fine-tuning and reflective prompt evolution.

Figure 1: Overview of PerceptUI, a framework for persona-conditioned UI/UX evaluation.

Training Pipeline: Contrastive Reflection Fine-Tuning

Conventional imitation learning only teaches models to mimic labels, lacking explicit supervision about the underlying reasons behind choice selection. PerceptUI incorporates a teacher model to generate contrastive rationales per example, decomposing explanations into UI evidence, persona relevance, and contrast against alternative options. The model is then trained (via QLoRA on a Qwen-VL backbone) to reproduce these rationales along with the answer, thereby encouraging the agent to internalize not only what to select but also comparative grounds for rejection and acceptability—critical in subjective and context-sensitive UI/UX assessment.

Figure 2: Contrastive reflection fine-tuning leverages teacher-generated rationales that contrast the selected answer with distractors, distinguishing UI and persona-specific factors.

Reflective Prompt Evolution: Prompt Optimization via Failure Traces

Despite improved context grounding from fine-tuning, residual adaptation is necessary for generalization to diverse surveys, answer scales, and question formats. PerceptUI utilizes a non-parametric reflective prompt-evolution mechanism, iteratively refining inference prompts based on model errors encountered on a held-out development set. At each iteration, deviations from human responses are analyzed by a symbolic evaluator and LLM-based analyzer, which recommend targeted edits to UI grounding, persona conditioning, or answer formatting in prompts. Successive candidate prompts are automatically audited before deployment.

Figure 3: Reflective prompt evolution process, iteratively updating prompts using symbolic feedback from observed prediction failures.

Empirical Results

Personalized UI/UX Evaluation and Generalization

PerceptUI is evaluated on six public and proprietary datasets covering design preference, critique, Likert-scale rating, and population-level distributional calibration, including WiserUI-Bench, UIClip/BetterApp, WebDevJudge, UICrit, LabintheWild, LabintheWild-UX, and UXcar. The system consistently attains higher accuracy, calibration (lower JS divergence), and rationale quality as rated by humans than strong MLLM-based and classical baselines, with pronounced gains in order-invariant selection metrics and rationale usefulness.

• On WiserUI-Bench: PerceptUI yields the highest consistent accuracy (CA), addressing the positional bias that plagues MLLM-based models.
• On UIClip/BetterApp: PerceptUI surpasses UIClip variants trained with large amounts of design-labeled data and improves accuracy from 75.12% (best baseline) to 79.28%.
• On LabintheWild-UX: Persona-aware PerceptUI improves rating prediction and alignment with empirical distributions, reducing JS divergence and mean absolute error, with accuracy and calibration outperforming task-specific fine-tuned baselines.
• Aggregation over personas provides close alignment between predicted and empirical answer distributions (Figure 4), especially when leveraging soft aggregation of probabilistic responses across sampled personas.

  Figure 4: Population-level calibration on UXcar, showing close match between predicted and empirical human response distributions.

UX Question Answering: Visual Grounding and Persona Sensitivity

The model achieves strong macro-F1 across question categories (comprehension, clarity, perception, spatial location) on UXcar, illustrating an increased ability to handle perception tasks requiring fine-grained attention and user context.

Figure 5: UX question answering on UXcar, showing macro-F1 by question group.

Performance drops when contrastive fine-tuning or reflective prompt evolution are ablated, underscoring their complementary contributions for generalization to unseen questions and participants. Notably, PerceptUI maintains strongest accuracy even for entirely novel persona–question combinations.

Rationale Quality and Visual Localization

Human evaluation of rationales finds PerceptUI's explanations both more contrastive and grounded (average rating 3.94/5) than alternatives. Visual evidence localization on UICrit shows gains in associating textual critiques with bounding box regions, though fine-tuned, patch-based methods yield the best performance for dense localization.

Prompt Evolution and Ablation

Reflective prompt evolution rapidly improves accuracy in early iterations, particularly for tasks with highly variable question or scale formats. Gains plateau as low-level prompt ambiguities are eliminated. Ablation studies confirm that persona context must be specific and meaningfully connected to the question to yield improvements; shuffled or generic profiles are weak signals.

Figure 6: Accuracy during reflective prompt evolution across four UI-evaluation benchmarks, highlighting improvement dynamics.

Implications and Future Prospects

PerceptUI advances the simulation of persona-conditioned UI/UX response, providing both individualized predictions and population-level calibration. The methodology highlights the value of contrastive rationale supervision, symbolic prompt iteration, and explicit separation of user and UI evidence factors in LMM-based evaluation agents.

Practically, the framework reduces per-UI evaluation cost compared to repeated large-scale LLM querying per persona through amortized offline fine-tuning. This enables feasible early-stage screening of interface variants across many simulated user profiles. Theoretically, the approach demonstrates that explicit reasoning about answer rejection, option comparison, and context grounding translates to improvements in user-aligned simulation fidelity.

However, the framework inherits biases from model and persona data, does not currently incorporate multi-step interactive traces, and produces explanations that are still model-based rather than actual causal attributions. Addressing dynamic interaction, richer persona acquisition and auditing, and incorporation of behavioral priors remain key directions. Evaluation against further tasks bridging static preference, temporal adaptation, and accessibility is warranted.

Conclusion

PerceptUI establishes a two-stage pipeline for persona-conditional UI/UX evaluation by LMM-based agents, combining contrastive reflection fine-tuning and reflective prompt evolution. The system delivers enhanced user-aligned answer prediction, rationale generation, and response distribution calibration for a broad set of UI/UX evaluation tasks. The results validate that explicit contrastive supervision and prompt evolution are crucial for overcoming the inherent limitations of generic MLLMs in simulating human behavioral diversity and perception in design. Further research is necessary on extending persona simulation fidelity, modeling interactivity, and safeguarding against bias propagation (2606.05697).