LLARS: LLM Assisted Research System

• LLARS is an open-source LLM research platform that integrates collaborative prompt engineering, batch generation, and hybrid evaluation into a single linear workflow.
• Its collaborative prompt engineering module enables real-time co-authoring with version control, providing immediate LLM testing and detailed provenance tracking.
• The batch generation module computes combinations of prompts, models, and data with explicit cost estimation, while hybrid evaluation uses both human and LLM raters for rigorous assessment.

LLARS, short for LLM Assisted Research System, is an open-source platform for building LLM-based systems through structured collaboration between domain experts and developers. It is organized as an end-to-end pipeline with three tightly connected modules—Collaborative Prompt Engineering, Batch Generation, and Hybrid Evaluation—so that prompts, generated outputs, and evaluation results move directly from one stage to the next. The system is designed to support real-time co-authoring, configurable large-scale generation across prompts, models, and data, and joint human–LLM assessment with live agreement metrics and provenance analysis for selecting a model–prompt combination for a given use case (Steigerwald et al., 11 May 2026).

1. Concept and scope

LLARS is presented as a platform that “bridges the gap between domain experts and developers for building LLM-based systems.” Its central design choice is full integration: prompt authoring, batch execution, and evaluation are not separate tools but parts of a single linear workflow. In the high-level architecture, domain experts and developers co-author prompts in the first module; those prompts flow into Batch Generation, which materializes the Cartesian product of prompts, models, and data; the resulting outputs then feed directly into Hybrid Evaluation, where human and LLM evaluators jointly rate and compare outputs (Steigerwald et al., 11 May 2026).

At each transition point, LLARS exposes explicit handoff artifacts. Prompt blocks with version metadata become template definitions for batch generation; batch results with provenance records become evaluation scenarios; evaluation outcomes, including agreement statistics and provenance analyses, are then used to select the best model–prompt pair. Prompts, LLM outputs, and evaluation results can be exported as JSON or CSV, and completed work can be handed off to the next stage with a single click (Steigerwald et al., 11 May 2026).

This arrangement suggests that LLARS is not merely a prompt editor or benchmarking interface, but a workflow system for iterative LLM development in which authoring, execution, and assessment are tightly coupled. A plausible implication is that the platform targets organizational bottlenecks associated with fragmented tooling, especially in settings where non-developers must participate directly in prompt design and qualitative evaluation.

2. Pipeline architecture and data movement

The overall pipeline consists of three core modules arranged in sequence:

Module	Primary function	Downstream handoff
Collaborative Prompt Engineering	Real-time co-authoring, version control, instant LLM testing	Prompt blocks with version metadata become template definitions
Batch Generation	Configurable output production across prompts, models, and data	Batch results with full provenance become evaluation scenarios
Hybrid Evaluation	Human and LLM assessment with agreement metrics and provenance analysis	Outcomes support selection of the best model–prompt pair

Within this pipeline, LLARS treats prompt definitions, generated outputs, and evaluation results as explicit data artifacts rather than ephemeral interface state. The data flow described in the system architecture is: prompt authoring followed by storage of prompts and versions; on generation, the backend expands $P \times M \times D$, streams requests to LLM APIs, and stores responses with provenance; on evaluation, the client instantiates a scenario, distributes items, collects ratings, and computes analytics (Steigerwald et al., 11 May 2026).

Batch generation is formalized through sets of prompts $P = \{p_1,\dots,p_{|P|}\}$, models $M = \{m_1,\dots,m_{|M|}\}$, and data items $D = \{d_1,\dots,d_{|D|}\}$, yielding

$\mathrm{Outputs} = P \times M \times D$

with cardinality

$|\mathrm{Outputs}| = |P| \times |M| \times |D|.$

The explicit use of the Cartesian product is operationally significant because it converts prompt experimentation into a reproducible combinatorial design over prompt variants, model choices, and input data. In LLARS, that design is exposed in a “generation matrix” interface that previews combinations before execution (Steigerwald et al., 11 May 2026).

3. Collaborative Prompt Engineering

The first module provides real-time co-authoring in a web-based prompt editor. Every keystroke is synchronized instantly across connected users via a CRDT framework, specifically citing Yjs and the underlying CRDT theory. Prompts are composed of ordered “blocks”; one block may be designated as the system prompt, while the remaining blocks form the user prompt (Steigerwald et al., 11 May 2026).

Version control operates at block granularity. Each block maintains its own version history, and insertions and deletions are tracked finely enough to support views such as “+100 / –0” in a diff display. This allows rollback or diff-based comparison without affecting other blocks. The system description states that block-level state is persisted in a Git-like object store, and that diff and rollback operations are exposed through the user interface (Steigerwald et al., 11 May 2026).

LLARS also includes template variables and a shared Variable Palette. Blocks can contain placeholders of the form {variable_name}, and the palette lists variables, sample values, and data types. That same variable specification is then consumed by Batch Generation, where users upload tabular data whose columns map to those variables. The prompt editor therefore acts both as a collaborative writing environment and as a template authoring system for subsequent large-scale runs.

The instant LLM testing workflow is integrated directly into this module. When “Test Prompt” is activated, the assembled prompt—with sample values substituted for variables—is sent to a selected LLM via an API connector, and the response is streamed back in real time into the editor panel. New prompts and prompt versions then become immediately available in Batch Generation (Steigerwald et al., 11 May 2026).

This module addresses a recurrent difficulty in prompt development: qualitative iteration often happens in documents or notebooks detached from execution environments. LLARS instead binds authoring, versioning, and immediate model probing into one collaborative surface.

4. Batch Generation and provenance

Batch Generation is the execution layer that expands selected prompts, models, and data into a full set of outputs. Its primary interface is a matrix view that previews every combination before execution. This makes explicit which prompt variants are run on which models and which data rows, and it exposes configuration at the level of both slices of the matrix and individual rows (Steigerwald et al., 11 May 2026).

A notable feature is explicit cost control. During preview, estimated cost is computed as

$$\mathrm{Cost}_{\mathrm{estimated}} = \sum_{(p,m,d)\in P\times M\times D} \left(\frac{\mathrm{tokens\_generated}(p,m,d)}{1000}\right)\times \mathrm{Rate}_m.$$

Users can set a budget cap $B$, and generation halts automatically if accumulated cost is at least $B$. For each model $m$, standard decoding parameters are configurable, specifically temperature, top-k, and max_tokens. These settings propagate across the selected slice of the matrix or can be overridden per row (Steigerwald et al., 11 May 2026).

Each generated item carries a provenance record of the form

$P = \{p_1,\dots,p_{|P|}\}$5

Completed batches can then be exported to CSV or JSON, or forwarded to Hybrid Evaluation with a single click. The same technical overview states that all API calls are logged for cost auditing and provenance, and that new models or providers can be registered via a simple JSON manifest so that they appear immediately in both Prompt Engineering and Batch Generation (Steigerwald et al., 11 May 2026).

The emphasis on provenance distinguishes LLARS from ad hoc prompt experimentation. Developers in the reported interviews explicitly valued “end-to-end provenance (no more ad hoc spreadsheets),” indicating that provenance is not only a backend logging property but also a collaboration feature that supports reproducibility and auditability (Steigerwald et al., 11 May 2026).

5. Hybrid Evaluation and model–prompt selection

Hybrid Evaluation bundles generated outputs into evaluation scenarios. A scenario includes assignment of evaluators, who may be human and/or LLM; a choice of assessment method; and a distribution strategy, either assigning all items to all evaluators or splitting subsets across raters. LLARS treats LLM evaluators as first-class participants, using the same user interface, the same randomization, and the same protocols as for human raters, and their judgments feed into the same analytics engine for side-by-side comparison (Steigerwald et al., 11 May 2026).

The available assessment methods are diverse: configurable multi-dimensional Likert scales, bucket ranking with ordinal categories, traditional pairwise comparison, categorical labels, mail assessment, and authenticity detection. “Mail Assessment” is described as a specialized preset for assessing email-style counselling replies. Bucket ranking is connected in the technical overview to Best-Worst and ordinal bucket formulations (Steigerwald et al., 11 May 2026).

Agreement statistics are computed on demand, including Krippendorff’s $P = \{p_1,\dots,p_{|P|}\}$0:

$P = \{p_1,\dots,p_{|P|}\}$1

where $P = \{p_1,\dots,p_{|P|}\}$2 is observed disagreement and $P = \{p_1,\dots,p_{|P|}\}$3 is expected disagreement by chance. Agreement can be filtered to “humans only,” “LLMs only,” or “both combined.” This matters because LLARS is designed not only to collect ratings but also to interrogate alignment or divergence between human and automated evaluators (Steigerwald et al., 11 May 2026).

For model–prompt selection, LLARS computes a per-pair “top-bucket hit rate”:

$P = \{p_1,\dots,p_{|P|}\}$4

A per-pair distribution of ratings across all buckets is displayed in a bar chart, and the system surface-ranks combinations by HitRate to recommend the best performer (Steigerwald et al., 11 May 2026).

This evaluation design combines scalar agreement metrics with provenance-aware analysis. Rather than treating evaluation as a terminal judgment, LLARS uses evaluation outputs to close the loop back to model and prompt selection. A plausible implication is that the platform is optimized for comparative decision-making under multiple raters and multiple candidate system configurations, rather than for single-model benchmarking alone.

6. Implementation, case studies, and observed effects

The described implementation is a containerized web application using Docker or Kubernetes. The backend API is specified as Python/Flask or Node.js, orchestrating prompt storage, batch jobs, evaluation campaigns, and analytics. Real-time collaboration is implemented with Yjs CRDT over WebSocket channels. LLM integration is modular, with connectors to OpenAI, Anthropic, HuggingFace, and related providers loaded dynamically. Prompt documents and version history are stored in a Git-style store, while batch outputs and evaluations reside in a relational or NoSQL database (Steigerwald et al., 11 May 2026).

The user interface is partitioned into three corresponding surfaces. The Prompt Editor includes a block-based canvas, diff view, variable palette, and a “Test” button with streaming output panel. The Batch Matrix View is a spreadsheet-like grid with checkboxes, a cost estimator slider, a “Run” button, and a live stream of results. The Evaluation Dashboard contains scenario overview, evaluator assignment, randomized rating UI, and live aggregated charts and agreement statistics (Steigerwald et al., 11 May 2026).

The paper reports case studies in online counselling. These include the Virtual Client project, used for prompt development for simulated client dialogues, and the CAIA system, an email-counselling assistant that used LLARS for prompt iteration and quality evaluation. A further Subject-Line Generation Study generated 253 candidate subject lines across 11 LLMs and collected 1,518 assessments by five professionals plus one LLM evaluator; the study identified minimal model complexity meeting quality thresholds (Steigerwald et al., 11 May 2026).

Semi-structured interviews were conducted with six domain experts and three developers. Reported findings include that LLARS “saves considerable time by keeping everything in one place,” that domain experts found the evaluation UI intuitive and appreciated instant feedback on prompt tests, that developers valued end-to-end provenance, and that both groups stated, “Finally we can work together rather than passing documents back and forth.” Quantitative self-reports indicated an approximately 30–40% reduction in cycle time from prompt draft to evaluated output (Steigerwald et al., 11 May 2026).

7. Benefits, limitations, and terminological distinctions

The benefits identified in the technical overview are Full E2E Integration, True Collaboration, Rigorous Provenance, and Hybrid Evaluation. In that account, LLARS eliminates tool-chaining overhead, enables domain experts to self-serve on prompt design and evaluation, ensures that every output and rating is traceable to prompt version and model, and combines human judgment with automated LLM evaluators for scale (Steigerwald et al., 11 May 2026).

The documented limitations are equally specific. First, LLARS has a Single-Turn Focus: the current modules assume atomic, one-off prompt-to-response interactions and do not include built-in support for multi-turn dialogue. Second, LLM Evaluator Bias remains a concern: the quality of LLM ratings depends on the reasoning properties of the underlying model, and systematic calibration is needed. The proposed future extensions are multi-turn conversational evaluation, automated calibration of LLM evaluators against human baselines in real time, and an HTTP API for closed-loop fine-tuning that would feed evaluation outcomes directly into model-training pipelines (Steigerwald et al., 11 May 2026).

Because the acronym resembles several unrelated research names, terminological distinction is useful. LARG refers to “Language-based Automatic Reward and Goal Generation,” a method that converts natural-language task descriptions into reward or goal-generation functions for robotic manipulation (Perez et al., 2023). LLARs denotes “Long-Lived Active Regions” in solar physics, defined as active regions observed during at least two consecutive Carrington rotations without significant successive flux emergence after decay begins (Mason et al., 28 Oct 2025). LLaRS refers to a language-conditioned foundation model for multi-modal remote sensing image restoration and fusion (Cui et al., 7 Apr 2026). These are separate topics; the collaborative LLM engineering platform is specifically LLARS (Steigerwald et al., 11 May 2026).

Taken together, the available description positions LLARS as infrastructure for interdisciplinary LLM system development in which authoring, execution, and evaluation are operationally unified. This suggests a broader research significance: the platform formalizes prompt iteration as a provenance-rich, evaluator-aware, and collaboration-centered engineering process rather than an informal sequence of disconnected experiments.