LLARS: Enabling Domain Expert & Developer Collaboration for LLM Prompting, Generation and Evaluation

Abstract: We demonstrate LLARS (LLM Assisted Research System), an open-source platform that bridges the gap between domain experts and developers for building LLM-based systems. It integrates three tightly connected modules into an end-to-end pipeline: Collaborative Prompt Engineering for real-time co-authoring with version control and instant LLM testing, Batch Generation for configurable output production across user-selected prompts $\times$ models $\times$ data with cost control, and Hybrid Evaluation where human and LLM evaluators jointly assess outputs through diverse assessment methods, with live agreement metrics and provenance analysis to identify the best model-prompt combination for a given use case. New prompts and models are automatically available for batch generation and completed batches can be turned into evaluation scenarios with a single click. Interviews with six domain experts and three developers in online counselling confirmed that LLARS feels intuitive, saves considerable time by keeping everything in one place and makes interdisciplinary collaboration seamless.

• The paper introduces LLARS, an integrated open-source platform that unifies prompt engineering, batch generation, and hybrid human–LLM evaluation.
• It details a modular system with real-time collaborative editing, scalable generation across models and data, and rigorous evaluation metrics including Krippendorff’s α.
• Empirical results in online counselling demonstrate reduced turnaround time, enhanced interdisciplinary collaboration, and robust model selection.

LLARS: A Unified Platform for Domain Expert and Developer Collaboration in LLM Prompting, Generation, and Evaluation

Overview and Motivation

The LLARS (LLM Assisted Research System) platform addresses a critical bottleneck in deploying LLM systems within specialized domains, such as online counselling, law, and medicine, where domain expertise and technical capability must be tightly integrated. Existing workflows tend to segregate domain experts and developers, impeding iterative prompt engineering, systematic output generation, and rigorous evaluation—especially on proprietary or in-domain datasets that defy standardized benchmarks. The paper presents LLARS as an open-source, end-to-end web-based platform that unifies collaborative prompt engineering, extensible batch output generation, and hybrid human–LLM evaluation, thereby radically streamlining interdisciplinary workflows.

System Architecture and Capabilities

LLARS comprises three integrated modules:

1. Collaborative Prompt Engineering:

The platform features a real-time, multi-user prompt editor. Prompts are decomposed into ordered blocks with independent version control, enabling granular diffing and selective rollbacks. Template variables can be inserted to accommodate dynamic data insertion, and the shared variable palette maintains dataset schema consistency. Instant model feedback closes the prompt iteration loop, and each prompt (with version/context) is exportable and immediately propagates to subsequent modules.

2. Batch Generation Across Prompts, Models, and Data:

Batch generation operationalizes prompt testing at scale by constructing the full Cartesian product of prompts, models, and data instances. The batch module previews the generation matrix, implements real-time provenance tracking (including token count, cost, and version trace), and imposes budget-based controls. Generated outputs seamlessly feed into evaluation scenarios, ensuring strict chain-of-custody for every item.

3. Hybrid Human–LLM Evaluation:

Integrated campaigns allow for flexible evaluation schema: Likert, ranking, categorical, and pairwise paradigms are supported, with configurable roles and assignment policies for evaluators. Critically, LLARS treats LLM evaluators as peers to human annotators, permitting live calculation of agreement metrics such as Krippendorff’s $\alpha$ and advanced provenance analytics that directly surface optimal (prompt, model) pairs according to empirical metrics. Evaluations are constructed to blind evaluators regarding generation provenance, minimizing systematic bias.

Comparative Analysis with Existing Tooling

A systematic review of 12 established GUI-first tools reveals that while some platforms offer subsets of LLARS’s functionality (prompt versioning, batch generation, human or LLM-based evaluation), none consolidate all three in a single, open-source, real-time collaborative system with live analytics and provenance. For example, tools like Agenta [agenta2024] and ChainForge [arawjo2024chainforge] emphasize prompt management and multi-model comparison but lack robust support for structured, multi-evaluator, hybrid assessment. Label Studio and Argilla provide strong annotation pipelines but disconnect prompt development from evaluation. Commercial observability tools (LangSmith, Maxim, Braintrust, Vellum) restrict extensibility and collaborative authoring. LLARS thus advances the state-of-the-art by centralizing the full LLM experimentation and validation pipeline in a domain-agnostic platform.

Numerical Results and Qualitative Findings

A detailed use case in online counselling [steigerwald2025subjectline] illustrates the platform conducting a 253-way subject line generation task over 11 distinct LLMs, with professional counsellors and LLM evaluators producing 1,518 discrete quality assessments. This enabled fine-grained model selection against human quality thresholds without external data leakage or context loss. Semi-structured interviews with six domain experts and three developers indicated that the consolidated pipeline reduced turnaround time, eliminated inter-disciplinary translation overhead, and was rated as intuitive by practitioners. Notably, feedback highlighted that domain experts could independently author prompts and carry evaluations without technical guidance, marking a departure from siloed development paradigms.

Practical and Theoretical Implications

LLARS’s architecture directly responds to emerging regulatory and safety requirements for LLM-based applications in critical domains such as mental health, healthcare, and law [chen2024critical, euaiact2024], where transparent, auditable, and collaborative development lifecycles are now mandatory. The inclusion of hybrid evaluation scenarios anticipates mounting evidence of LLM evaluation inconsistencies [zheng2024judging] and provides practitioners with mechanisms to calibrate (and scrutinize) LLM-as-judge models in parallel with human raters.

On a theoretical level, the system’s approach to provenance tracking and interleaving human/LLM assessment dovetails with best-practice recommendations for robustly benchmarking LLMs in high-stakes or specialized settings.

Limitations and Future Work

Current system limitations include:

• Exclusive support for single-turn generation; multi-turn conversational evaluation is pending.
• LLM evaluators are limited by the reasoning/reliability bounds of the underlying LLM.
• The current ecosystem assumes text-based tasks; extension to multimodal domains would require non-trivial engineering.

Future work proposed includes multi-turn evaluation workflows with nested context tracking, automated calibration pipelines comparing LLM and human evaluation output for systematic bias detection, and HTTP APIs to enable closed-loop model fine-tuning based on evaluation outcomes.

Conclusion

LLARS introduces a rigorously integrated open-source platform for LLM prompt engineering, batch generation, and hybrid evaluation, engineered to streamline collaboration between domain specialists and AI developers. The platform's demonstrated impact in online counselling and its domain-agnostic design indicate strong potential for adoption in any sector requiring systematically validated, data- and prompt-centric LLM system development. Its methodological integration of human and LLM assessment, combined with provenance-first analytics, positions it as a significant enabler for future research on high-reliability, regulated, and context-sensitive LLM deployments.

Reference:

"LLARS: Enabling Domain Expert & Developer Collaboration for LLM Prompting, Generation and Evaluation" (2605.10593)