DynaBench Server

• DynaBench Server is a cloud-based evaluation-as-a-service infrastructure that automates and standardizes NLP model assessment using containerized workflows and multi-metric benchmarking.
• It features a modular microservice architecture with dynamic scheduling, metrics collection, and perturbation services to ensure fairness and reproducibility.
• Its Dynascore metric aggregates diverse evaluation dimensions like accuracy, robustness, fairness, and efficiency, enabling real-time leaderboard updates and API integration.

DynaBench Server is a cloud-based evaluation-as-a-service (EaaS) infrastructure designed to standardize, automate, and enhance the evaluation of NLP models via real-time, containerized assessment across multiple metrics and perturbation axes. Integrated deeply within the Dynabench platform, the server supports holistic multi-metric benchmarking, enables dynamic leaderboards, and enforces rigorous reproducibility, accessibility, and backwards compatibility in model evaluation methodologies. The DynaBench Server replaces self-reported model performance with platform-collected results, mitigating prevalent issues in reproducibility and comparability in NLP.

1. System Architecture and Core Components

The DynaBench Server implements a modular, microservice-oriented EaaS architecture. Model evaluation is fully containerized, based on the following principal subsystems (Ma et al., 2021):

• Model Submission & Packaging: Developers implement a standardized "model handler" (input JSON → output JSON), supply a Dynalab configuration, and submit the model via the CLI (dynalab), which uses TorchServe or TF-Serving as the serving backend. Both the containerized model and an accompanying model card (YAML/JSON: training data, license, hyperparameters, etc.) are uploaded to a cloud registry.
• Cloud Deployment & Orchestration: An Evaluation Orchestrator (microservice) retrieves the container image and deploys it to identically provisioned, CPU-only VMs (Docker-based workers) to ensure fairness across hardware. Each worker exposes a /predict endpoint (gRPC/HTTP) conforming to a fixed input-output specification.
• Scheduling and Execution: A Scheduler sequences model evaluation jobs, iterating over all (task, dataset, metric) combinations and invoking the /predict endpoint per example.
• Metrics and Perturbation Services: Metrics Collector services stream system- and inference-logs (CPU/GPU utilization, latency, memory), while a Perturbation Service uses TextFlint and fairness-perturbation libraries to generate adversarial or fairness-modified examples for robustness and bias testing.
• Results Persistence and API Layer: A Results Database records all prediction outputs, resource usages, perturbation results, and aggregate metrics. A WebSocket/REST API feeds leaderboards and supports UI interactivity, including on-the-fly re-ranking with custom metric weights.

2. Evaluation Workflow

The live, end-to-end evaluation workflow proceeds as follows (Ma et al., 2021):

(a) Model Submission:

The developer packages and submits a model by executing: $T_1$8 The CLI builds the container, validates on a “sanity” set, and pushes both code and metadata to the registry.

(b) Scheduling:

Upon new submission or re-evaluation request, the Scheduler enqueues model×dataset×metric jobs.

(c) Model Execution:

Each worker node pulls the model, launches TorchServe, and exposes /predict. For each dataset example $x$:

• Records wall-clock time ($T_1$, $T_2$) and memory usage ($M_1$, $M_2$).
• Throughput: $1/(T_2-T_1)$ examples/s.
• Memory: $\max(M_2-M_1)$ gigabytes. Logs are streamed to the Metrics Collector.

(d) Perturbation and Fairness Evaluation:

The Perturbation Service produces fairness (attribute swaps) and robustness (TextFlint: typos, OCR noise) variants:

• For $x'$, the fairness-perturbed example: check label flip.
• For robustness, compute the fraction of predictions unchanged under perturbation.

(e) Metric Aggregation:

Once evaluation completes:

• Raw scores (accuracy/F1 on each test set)
• Throughput, memory, fairness-flip rate, robustness-preservation rate are stored in the Results DB.

(f) Dynascore Computation and Leaderboard Update:

Frontend server accesses the Results DB, enables user-customizable weights for all metrics, computes the final Dynascore, and updates rankings on demand.

3. Dynascore: Multi-Metric Utility Aggregation

Dynascore is a utility-based aggregation metric reflecting holistic model quality over heterogeneous evaluation axes (Ma et al., 2021). All base metrics are normalized to "goods" (higher = better), and custom weighting is supported.

• Marginal Rate of Substitution (MRS):

For metric $M$ and canonical perf metric $perf$, let $T_1$0.

$T_1$1

for all pairs with $T_1$2.

• Average MRS (AMRS):

$T_1$3

• Dynascore Calculation:

$T_1$4

where the canonical performance metric receives $T_1$5 and all other metrics (throughput, memory-saved, fairness, robustness) share $T_1$6. User-specified weights $T_1$7 are supported; the front end instantly re-calculates Dynascore upon changes.

This suggests that users can flexibly emphasize robustness, memory efficiency, throughput, or fairness in ranking models, supplementing accuracy-based evaluations.

4. Reproducibility, Accessibility, and Compatibility Guarantees

DynaBench Server enforces several critical properties for scientific benchmarking (Ma et al., 2021):

• Reproducibility:

All models are run in identical containerized environments (same TorchServe/TF-Serving version, system libraries, CPU VM type). No self-reported metrics are accepted; results (including predictions and resource usage) are platform-collected.

• Accessibility:

Models are hosted in the cloud and made accessible via a simple prediction API. Researchers and practitioners can interactively query any registered model (via browser or CLI) without needing local compute resources.

• Backwards/Forwards Compatibility:

New datasets or metrics (e.g., future stress sets or newly released automated metrics) can be retroactively applied: the scheduler re-enqueues prior models for batch evaluation. Submitted models are immutable with version tags (container hash, model card), and datasets are versioned; historical evaluation data remains accessible, supporting longitudinal comparisons.

5. API Interfaces and Interoperability

The platform exposes an inferred public API, enabling integration with external tools and continuous benchmarking workflows. While a full REST/OpenAPI schema is not provided, the following endpoints are documented (Ma et al., 2021):

Endpoint	Purpose	Example Payload/Result
POST /models	Register model, submit container and card	{ "model_name":..., "handler_image":..., ... }
GET /models/{model_id}	Retrieve model metadata and evaluation status	{ "model_id":"abc123", "status":"pending" }
POST /evaluations	Initiate evaluation batch (model, task, datasets, metrics)	{ "model_id":"abc123", "task":"NLI", ... }
GET /evaluations/{eval_id}/results	Retrieve scores and Dynascore for evaluation	{ ... "dynascore": 45.8, ... }
GET /leaderboard	Query/re-rank leaderboard by weights/datasets/tasks	[{"model_id": ..., "dynascore": ..., "rank": ...}]

A plausible implication is that users and automated pipelines can submit new models, trigger evaluations, and integrate results without direct UI interaction, similar to EaaS platforms such as CodaLab and EvalAI.

6. Context, Significance, and Broader Impacts

DynaBench Server addresses limitations of traditional benchmarking, which relies on static test sets, single metrics, and self-reporting, by enabling dynamic, platform-controlled, and multi-dimensional evaluation workflows. The infrastructure-level standardization of hardware, serving, and metrics collection mitigates artifacts caused by inconsistent local experiments. The adoption of utility-based aggregation (Dynascore) foregrounds practical concerns—such as efficiency, fairness, and robustness—that are increasingly relevant to NLP system deployment and research.

Its integration of adversarial robustness and fairness perturbation as first-class evaluation axes enables systematic auditing and research replication. The inherent extensibility—inserting new datasets or metrics and retroactively re-scoring published models—supports evolving community standards and continual benchmarking, an increasingly critical requirement as NLP tasks and evaluation desiderata diversify (Ma et al., 2021).