DRAMA-Bench: A Multifaceted Benchmark

• DRAMA-Bench is a collection of benchmarks spanning open-domain analytic queries, dynamic multi-agent coordination, and drama script continuation with distinct evaluation protocols.
• In the multi-agent setting, it uses metrics like success rate, average steps, and total steps to assess performance under task reallocation and agent churn scenarios.
• For open-domain analytics, DRAMA-Bench tests systems on data retrieval, transformation, and analytic reasoning using real-world, temporally fresh evidence.

DRAMA-Bench is not a single uniformly defined benchmark name across current arXiv literature. In one usage, it is an explicit benchmark for open-domain analytic queries, introduced alongside the DRAMA paradigm for unifying data retrieval, transformation, and analysis, and consisting of 100 claim verification and 100 question answering tasks (Hu et al., 31 Oct 2025). In another usage, the term is best treated as an implicit benchmark concept derived from the multi-agent systems paper “DRAMA: A Dynamic and Robust Allocation-based Multi-Agent System for Changing Environments,” whose experiments already function as a prototype evaluation suite for dynamic, failure-prone, changing environments, even though the paper does not define an explicit benchmark called DRAMA-Bench (Wang et al., 6 Aug 2025). A further source of ambiguity is the separate benchmark “DramaBench”, which targets drama script continuation across six independent dimensions (Ma et al., 22 Dec 2025).

1. Terminological scope and disambiguation

The name “DRAMA-Bench” sits in a crowded nomenclature space. The open-domain analytics paper formally introduces DRAMA-Bench as a benchmark name (Hu et al., 31 Oct 2025). By contrast, the multi-agent systems paper explicitly states that it does not define a benchmark called DRAMA-Bench, but its architecture and evaluation methodology strongly motivate one centered on adaptation under changing team composition and interrupted-task recovery (Wang et al., 6 Aug 2025). In drama-generation research, several nearby names appear but refer to different objects: Short-Drama-Bench for one-sentence-to-short-drama generation, DramaBoard for plot-to-short-drama generation, and DramaBench for drama script continuation (Shi et al., 21 May 2026).

Label	Domain	Status in the literature
DRAMA-Bench	Open-domain analytic queries	Explicit benchmark
DRAMA-Bench	Dynamic multi-agent systems	Implicit benchmark concept
DramaBench	Drama script continuation	Explicit benchmark

This naming overlap matters because the benchmark target, evaluation protocol, and failure modes differ substantially across these usages. In one case the core problem is analytic grounding over web data; in another it is online task reallocation under agent churn; in a third it is screenplay-quality continuation.

2. DRAMA-Bench as a dynamic multi-agent benchmark concept

In the multi-agent systems line of work, the motivating formalization begins with a standard MAS consisting of agents and tasks in a shared environment,

$$\mathcal{A} = \{a_1, a_2, ..., a_N\}, \qquad \mathcal{Q} = \{\mathtt{q}_1, \mathtt{q}_2, ..., \mathtt{q}_M\},$$

with each agent represented as

$a_i = \langle S_i, O_i, A_i, \pi_i \rangle,$

and task assignment over time represented by a function $f_t: \mathcal{T} \rightarrow \mathcal{A}$. The paper contrasts this with static systems where assignment is fixed,

$f_t = f_0, \quad \forall t \in [1, T],$

and argues that such a static assumption is unsuitable because real environments involve agent arrival, departure, capability changes, task reprioritization, failures, delays, and workload fluctuations (Wang et al., 6 Aug 2025).

The benchmark-relevant systems contribution is the split between a control plane and a worker plane. The control plane performs state monitoring and centralized planning. The worker plane comprises autonomous agents with local reasoning, execution, collaboration, and the ability to take over unfinished tasks. The monitor aggregates global resource state as

$$\mathcal{X}_t = \left\{ \mathcal{X}_{r_k, t} \mid r_k \in \mathcal{R} \right\},$$

and the assignment is recomputed by an event-triggered planner-critic mechanism,

$f_t = \mathrm{Planner\mbox{-}Critic}(\mathcal{X}_t,\, \mathcal{E}_t).$

The paper also unifies agents and tasks as resource objects,

$\mathcal{R} = \{r_1, r_2, \ldots, r_{N+M}\},$

so scheduling operates over structured attributes such as capabilities, requirements, workload, location, status, and availability.

As a benchmark concept, this implies that evaluation should not center only on one-shot planning quality. It should instead stress dynamic reassignment, agent churn, partial-progress handover, abnormal task progress, and event-triggered reallocation. The benchmark target is therefore adaptation of $f_t$ under disturbances, rather than static optimality of $f_0$.

3. Prototype evaluation dimensions in the dynamic MAS setting

The experiments in the DRAMA multi-agent paper already define a de facto benchmark template. The environment is based on Communicative Watch-And-Help (C-WAH), itself built on the VirtualHome-Social environment, and the authors deliberately increase the number of objects in each task because default C-WAH tasks were too simple to distinguish advanced methods well (Wang et al., 6 Aug 2025). This establishes an important design principle: dynamic coordination benchmarks require enough cooperative complexity that reallocation quality and interrupted-task recovery become observable.

The comparison set spans multiple coordination paradigms: CoELA, MCTS / MASTER, AgentVerse-Static (AV-Static), AgentVerse-Dynamic (AV-Dynamic), and ProAgent. The metrics are explicitly defined as Success Rate (SR), Average Steps (AS), and Total Steps (TS). Static scenarios use 2, 3, or 4 agents. Dynamic scenarios are initialized with 3 agents and then apply one of two perturbations: Agent Dropout, where one agent is randomly removed at a random step between 5 and 10, and Agent Addition, where one new agent is added at a random step between 5 and 10.

The reported findings are sharply benchmark-relevant. In the abstract, DRAMA achieves a 17% improvement in runtime efficiency and a 13% reduction in resource consumption relative to existing frameworks. In the main experimental section, the reported reductions are 4.8% to 13.5% in AS and 4.6% to 17.3% in TS relative to the strongest baselines. The robustness table is especially discriminative: all methods succeed in static scenarios; all succeed in agent addition scenarios; but in dynamic dropout scenarios, only DRAMA succeeds. The paper also reports a stability evaluation on two C-WAH tasks over 100 independent trials, where DRAMA achieves lower medians and tighter distributions. In dynamic 3→2 and 3→4 scenarios, DRAMA instantiated with GPT-4.1, Qwen-max, Deepseek-v3, and GPT-4o-mini achieves 100% SR in all cases, indicating that the framework behavior is not tied to a single LLM backbone.

Taken together, these experiments define what a DRAMA-Bench in this sense should test: dynamic perturbations, reassignment latency, continuity after interruption, robustness to worker failure, efficiency-resource tradeoffs, and stability under randomized event timing.

4. DRAMA-Bench as a benchmark for open-domain analytic queries

The most explicit use of the name appears in “DRAMA: Unifying Data Retrieval and Analysis for Open-Domain Analytic Queries,” which formally introduces DRAMA-Bench to evaluate systems on tasks that require open-domain data collection, structured data transformation, and analytic reasoning in one end-to-end pipeline (Hu et al., 31 Oct 2025). The paper formalizes the paradigm as

$collect(Q) \rightarrow D,$

$a_i = \langle S_i, O_i, A_i, \pi_i \rangle,$0

$a_i = \langle S_i, O_i, A_i, \pi_i \rangle,$1

The benchmark contains 200 task instances: 100 claim verification and 100 question answering. Given a claim or question and a blacklist of domains that must not be used, the system must output the final answer, the transformed structured table, the analytic code, and the source traces.

The curation rules are strict. Tasks are derived from real-world applications that have gained significant public attention and use content dated on or after January 1, 2024, later than the knowledge cutoffs of state-of-the-art LLMs. Claim verification tasks come from PolitiFact and Twitter/X Community Notes, while question answering tasks come from USAFacts. For every task, the authors manually transform the original source data into a structured table and annotate SQL scripts that reproduce the result. The annotation team consists of 5 researchers with substantial SQL experience, and the remaining four authors independently cross-verify each transformation and SQL pair. The paper also states that for all 200/200 tasks, the required data ultimately comes from a single domain, though it may span multiple files or tables within that domain.

Evaluation uses both system-level and stage-level metrics. System-level metrics include accuracy, data-grounded accuracy (DG accuracy), and cost. Stage-level metrics cover data validity, multiple forms of data similarity, code execution success rate, and multiple forms of code similarity. The baselines are Deep Research, AutoGPT, WebVoyager, OpenAI Research Agent, and Search + TAG. The central result is that DramaBot achieves 86.5% overall accuracy, 82.5% DG accuracy, and about \$0.05/task, outperforming all baselines. By category, it reaches 88% on verification and 85% on QA. This is the only benchmark in the supplied literature that unambiguously formalizes and names DRAMA-Bench.

5. Relation to adjacent drama-generation benchmarks

Several neighboring benchmarks are often conflated with DRAMA-Bench but target different problems. Short-Drama-Bench evaluates end-to-end short-drama generation from a single-sentence user idea. It contains 50 story prompts, spans 7 popular categories with 17 fine-grained subcategories, and evaluates outputs using benchmark-specific dimensions such as Opening Hook, End Hook, Escalation Effect, Narrative Coherence, Character Spatial Continuity, Environment Layout Continuity, Music-Emotion Alignment, and Transition Naturalness (Shi et al., 21 May 2026).

DramaBoard is a benchmark and dataset for plot-to-short-drama generation. It is built from 35 live-action short dramas, 2,807 episodes, and 81,283 segmented shots, and supports evaluation of storyboard narrative quality, instruction following, and intrinsic video quality. Its continuation corpus uses an 8:1:1 train/validation/test split (Zhou et al., 23 Jun 2026).

DramaBench, despite the near-identical spelling, is a different benchmark for drama script continuation. It contains 1,103 professionally structured English drama scripts, reports 8,824 total model-script evaluations, and evaluates six independent dimensions: Format Standards, Narrative Efficiency, Character Consistency, Emotional Depth, Logic Consistency, and Conflict Handling (Ma et al., 22 Dec 2025).

Benchmark	Task	Scale or core scope
Short-Drama-Bench	One-sentence-to-short-drama generation	50 prompts
DramaBoard	Plot-to-short-drama generation	35 dramas, 2,807 episodes, 81,283 shots
DramaBench	Drama script continuation	1,103 scripts, six dimensions

The disambiguation is consequential. Short-Drama-Bench and DramaBoard evaluate generated dramatic media products; DramaBench evaluates screenplay continuation quality; DRAMA-Bench in the analytics paper evaluates open-domain data reasoning; and the DRAMA multi-agent paper contributes a benchmark concept for dynamic coordination rather than a named benchmark artifact.

6. Limitations, controversies, and open design questions

Across its different meanings, DRAMA-Bench remains methodologically unsettled. In the dynamic MAS usage, the paper itself acknowledges that centralized control creates a potential single point of failure and may limit scalability; it also notes reliance on LLM-based reasoning for agent-task affinity evaluation, which introduces inference latency. The same paper does not provide explicit lifecycle state machines, explicit affinity formulas, scheduling pseudocode, detailed optimization constraints, or ablations isolating monitor, planner, critic, heartbeat mechanism, or worker takeover ability (Wang et al., 6 Aug 2025).

In the open-domain analytics usage, the benchmark is intentionally restricted in several ways: all tasks draw data from a single domain, the transformed artifact is a single structured table, and QA answers are limited to a single string or numeric value. These choices strengthen evaluation reliability, but they also simplify multi-domain integration, multi-table workflows, and richer analytic outputs. The benchmark’s temporal freshness is a strength, yet it also implies an ongoing maintenance problem as public sources change over time (Hu et al., 31 Oct 2025).

In the screenplay continuation usage, DramaBench reports substantial or moderate human agreement on only 3 of 5 LLM-evaluated dimensions, with weak or non-significant agreement on Narrative Efficiency and Character Consistency. It is also restricted to English and assumes Fountain screenplay format (Ma et al., 22 Dec 2025). These limitations do not invalidate the framework, but they show that evaluator reliability in creative-generation benchmarks remains uneven.

This suggests that “DRAMA-Bench” is best understood not as a single benchmark standard, but as a family of benchmark designs organized around difficult forms of structured reasoning under domain-specific constraints: dynamic task allocation in changing multi-agent environments, open-domain analytic querying over heterogeneous web data, and drama-aware generation or continuation. The main unresolved question is therefore not merely naming, but which latent capability should be primary: adaptation, grounded analysis, or dramatic coherence.