DRACULA: Hunting for the Actions Users Want Deep Research Agents to Execute

Abstract: Scientific Deep Research (DR) agents answer user queries by synthesizing research papers into multi-section reports. User feedback can improve their utility, but existing protocols only score the final report, making it hard to study and learn which intermediate actions DR agents should take to improve reports. We collect DRACULA, the first dataset with user feedback on intermediate actions for DR. Over five weeks, nineteen expert CS researchers ask queries to a DR system that proposes actions (e.g., "Add a section on datasets"). Our users select actions they prefer, then judge whether an output report applied their selections successfully, yielding 8,103 action preferences and 5,230 execution judgments. After confirming a DR agent can execute DRACULA's actions, we study the predictability of user-preferred actions via simulation-how well LLMs predict the actions users select-a step toward learning to generate useful actions. We discover: (1) LLM judges initially struggle to predict action selections, but improve most when using a user's full selection history, rather than self-reported or extrapolated user context signals; (2) Users' selections for the same query differ based on unstated goals, bottlenecking simulation and motivating affordances that let users steer reports; and (3) Our simulation results inform an online intervention that generates new actions based on the user's past interactions, which users pick most often in follow-up studies. Overall, while work extensively studies execution, DRACULA reveals a key challenge is deciding which actions to execute in the first place. We open-source DRACULA's study design, user feedback, and simulation tasks to spur future work on action feedback for long-horizon agents.

• The paper introduces the DRACULA framework and dataset comprising 13,333 query-action-judgment points to systematically capture granular user feedback on intermediate actions.
• It demonstrates that agent utility issues stem primarily from suboptimal action selection rather than execution, with personalized history improving prediction performance.
• The study’s predictive modeling and simulation-informed interventions highlight the importance of dynamic, user-specific feedback for tuning adaptive deep research agents.

DRACULA: User Feedback on Intermediate Action Selection in Deep Research Agents

Motivation and Contributions

The DRACULA framework addresses a central limitation in current Deep Research (DR) agent evaluation: the dearth of user feedback on the intermediate actions agents take while generating long-form research syntheses. DR systems, which orchestrate chained LLM-based and retrieval-augmented operations for scientific report generation, have previously been assessed mainly via user judgments on the final report output. This approach obscures which decisions—retrieval foci, section planning, explanation strategies—users actually desire, hampering principled alignment and learning of effective agent policy.

DRACULA introduces the first open-source dataset and methodology for collecting granular, action-level user feedback in DR. Across 902 unique queries from 19 practicing computer science researchers, the protocol decouples action proposal from execution: users select desired actions from LLM-generated candidates per query, then assess execution quality after a DR agent attempts to realize those selections. This results in over 8,100 action preferences and 5,200 execution judgments, building a dataset for systematic study of which intermediate decisions users endorse and how accurately systems realize them.

Key technical contributions include:

• The DRACULA dataset: 13,333 granular query-action-judgment points annotated with rationales and user context.
• User simulation framework for predicting action preferences, establishing a new evaluation paradigm for agentic action proposal aligned with real user goals.
• Detailed ablation of context and modeling strategies for preference prediction, highlighting the importance of user-specific interaction history.
• Empirical evidence that deficiencies in agent utility stem primarily from suboptimal action selection rather than poor execution.
• Demonstrated utility of simulation-informed interventions for real-time action generation, yielding higher user endorsement rates.

Analysis of Action Selection and Execution

Empirical measurements reveal systematic discrepancies between DR agent capability to execute specified actions and the ability to autonomously propose actions that users prefer. Once an action is chosen by the user, the agent achieves high execution success (up to ≈0.75), while action selection rates are substantially lower (typically <0.65), indicating LLM-based action generators are the primary bottleneck.

Qualitative analysis of user rationales for action selection and rejection reveal several dominant patterns. Users overwhelmingly select actions that enhance report comprehension and comparison—summaries, structured comparisons, clarifying definitions—while rejecting actions that miscalibrate scope (narrowing or over-expanding reports), misalign with user intent, or introduce tangential or redundant content. Importantly, system limitations are cited extremely rarely as the reason for rejection; the central issue is action intent alignment, not execution capability. Fine-grained action clusters further refine this picture, showing that personalized, context-aware actions can increase selection rates but also risk drift or overfitting when not carefully managed.

Predictive Modeling of User Preferences

The authors formalize action prediction as a binary classification task: given a (query, action) pair and available user or session context, predict whether the user will select the action for execution. Several strong LLM-based baselines (e.g., GPT-5, Claude-4.6, Gemini-3.0 Pro) are evaluated in both zero-shot and few-shot (with user context/history) regimes.

Findings include:

• Baseline F1 for action preference prediction is only marginally better than random for generic actions; personalized/paper-informed actions are slightly more predictable.
• Incorporating user-specific historical selection context yields the largest improvements (ΔF1≈0.1–0.15 over non-personalized baselines), highlighting the necessity of user-level behavioral modeling.
• Alternative context regimes (user profile papers, stated preferences, rationales, intents) and input compression (using retrieved or stratified action histories) offer little incremental benefit over raw selection history.
• Stated user rules and model-inferred preference rules perform weaker than direct history, reflecting the implicit, dynamic, and at times unarticulable nature of user intent.

A follow-up consistency experiment establishes the upper bound for predictability of user preferences: even with access to all prior user actions, user intent can drift such that past interaction does not guarantee future desirability. The F1 for predicting a user's new selection after several months is ≈0.67, close to model-based results but far from perfect. This underscores the need for dynamically steerable interfaces and real-time clarification.

Personalized Action Generation and Intervention

Leveraging the strong explanatory power of user-specific interaction history, the study evaluates candidate personalization schemes for action generation: generic, profile (paper)-informed, user-stated preferences, and revealed (historical) preferences. Actions generated with context from user history and previously revealed preferences achieve the highest selection rates (≈0.86), surpassing both generic and research-paper-based contextualization.

These offline insights are translated into online interventions: DR agents conditioned on revealed preferences are empirically shown to elicit a broader and more highly-endorsed action set by users, introducing personalized variation mitigating the cold-start problem.

Implications and Future Directions

The results have significant theoretical and practical ramifications for the design of research-assistant agents and, more broadly, task-oriented long-horizon systems reliant on user preference realization. The findings support the following positions:

• Fine-grained, action-level user feedback is far more informative for agent modeling than summary output ratings, enabling targeted improvements, interpretability, and more robust alignment.
• Predictive user modeling should prioritize revealed preferences and actual behavior over static, profile-based context—addressing the inherent plasticity and situatedness of real user objectives.
• Simulation-driven agent tuning and intervention strategies offer promising mechanisms for bootstrapping personalization, but the ultimate ceiling is inherently limited by the unpredictability of individual human goals.
• System utility is currently bounded by action inferral, not execution; future work must close this gap via more sample-efficient, adaptive, and interactively clarifying generation approaches.

Ethical considerations are also addressed: optimizing for user-selected actions may reinforce existing biases, privacy risks, or echo undesirable user tendencies. DRACULA's methodology constrains personalization inputs to non-sensitive, system-derived contexts and recommends rigorous deployment qualification for target domains.

Conclusion

DRACULA advances the methodology for agentic research-assistant evaluation by introducing the first large-scale dataset and framework for collecting user feedback on intermediate actions in Deep Research systems. It demonstrates that preference prediction, not execution, is the primary challenge in aligning DR agent output with user needs—a challenge that is empirically tractable but inherently user- and context-dependent. The actionable insights provided by DRACULA enable more personalized, adaptive, and effective DR agents, and the dataset sets a foundation for future research in fine-grained user feedback, action alignment, and agentic task completion at scale.

(2604.23815)