---
title: DynaBench Server
url: https://www.emergentmind.com/topics/dynabench-server
type: topic
---

# DynaBench Server

DynaBench Server is a cloud-based evaluation-as-a-service (EaaS) infrastructure designed to standardize, automate, and enhance the evaluation of NLP models via real-time, containerized assessment across multiple metrics and perturbation axes. Integrated deeply within the Dynabench platform, the server supports holistic multi-metric benchmarking, enables dynamic leaderboards, and enforces rigorous reproducibility, accessibility, and backwards compatibility in model evaluation methodologies. The DynaBench Server replaces self-reported model performance with platform-collected results, mitigating prevalent issues in reproducibility and comparability in NLP.

## 1. System Architecture and Core Components

The DynaBench Server implements a modular, microservice-oriented EaaS architecture. Model evaluation is fully containerized, based on the following principal subsystems [2106.06052]:

- **Model Submission & Packaging**: Developers implement a standardized "model handler" (input JSON → output JSON), supply a Dynalab configuration, and submit the model via the CLI (`dynalab`), which uses TorchServe or TF-Serving as the serving backend. Both the containerized model and an accompanying model card (YAML/JSON: training data, license, hyperparameters, etc.) are uploaded to a cloud registry.
- **Cloud Deployment & Orchestration**: An Evaluation Orchestrator (microservice) retrieves the container image and deploys it to identically provisioned, CPU-only VMs (Docker-based workers) to ensure fairness across hardware. Each worker exposes a `/predict` endpoint (gRPC/HTTP) conforming to a fixed input-output specification.
- **Scheduling and Execution**: A Scheduler sequences model evaluation jobs, iterating over all (task, dataset, metric) combinations and invoking the `/predict` endpoint per example.
- **Metrics and Perturbation Services**: Metrics Collector services stream system- and inference-logs (CPU/GPU utilization, latency, memory), while a Perturbation Service uses TextFlint and fairness-perturbation libraries to generate adversarial or fairness-modified examples for robustness and bias testing.
- **Results Persistence and API Layer**: A Results Database records all prediction outputs, resource usages, perturbation results, and aggregate metrics. A WebSocket/REST API feeds leaderboards and supports UI interactivity, including on-the-fly re-ranking with custom metric weights.

## 2. Evaluation Workflow

The live, end-to-end evaluation workflow proceeds as follows [2106.06052]:

(a) **Model Submission:**  
The developer packages and submits a model by executing:
```bash
dynalab deploy my_model_handler.py --config model_card.yaml
```
The CLI builds the container, validates on a “sanity” set, and pushes both code and metadata to the registry.

(b) **Scheduling:**  
Upon new submission or re-evaluation request, the Scheduler enqueues model×dataset×metric jobs.

(c) **Model Execution:**  
Each worker node pulls the model, launches TorchServe, and exposes `/predict`. For each dataset example $x$:
- Records wall-clock time ($T_1$, $T_2$) and memory usage ($M_1$, $M_2$).
- Throughput: $1/(T_2-T_1)$ examples/s.
- Memory: $\max(M_2-M_1)$ gigabytes.
Logs are streamed to the Metrics Collector.

(d) **Perturbation and Fairness Evaluation:**  
The Perturbation Service produces fairness (attribute swaps) and robustness (TextFlint: typos, OCR noise) variants:
- For $x'$, the fairness-perturbed example: check label flip.
- For robustness, compute the fraction of predictions unchanged under perturbation.

(e) **Metric Aggregation:**  
Once evaluation completes:
- Raw scores (accuracy/F1 on each test set)
- Throughput, memory, fairness-flip rate, robustness-preservation rate  
are stored in the Results DB.

(f) **Dynascore Computation and Leaderboard Update:**  
Frontend server accesses the Results DB, enables user-customizable weights for all metrics, computes the final Dynascore, and updates rankings on demand.

## 3. Dynascore: Multi-Metric Utility Aggregation

Dynascore is a utility-based aggregation metric reflecting holistic model quality over heterogeneous evaluation axes [2106.06052]. All base metrics are normalized to "goods" (higher = better), and custom weighting is supported.

- **Marginal Rate of Substitution (MRS):**  
For metric $M$ and canonical perf metric $perf$, let $perf(x_1) ≥ perf(x_2) ≥ … ≥ perf(x_n)$.  
$$
MRS(M, perf) = \left| \frac{M(x_i) - M(x_{i+1})}{perf(x_i) - perf(x_{i+1})} \right|
$$
for all pairs with $perf(x_i) - perf(x_{i+1}) > \epsilon$.

- **Average MRS (AMRS):**  
$$
AMRS(M, perf) = \frac{1}{n-1} \sum_i MRS(M, perf)
$$

- **Dynascore Calculation:**  
$$
Dynascore(x_i) = \sum_{M} w_M \cdot \frac{M(x_i)}{AMRS(M, perf)}
$$
where the canonical performance metric receives $w_{perf} = 0.5$ and all other metrics (throughput, memory-saved, fairness, robustness) share $w_M = 0.5/(m-1)$.  
User-specified weights $z_M$ are supported; the front end instantly re-calculates Dynascore upon changes.

*This suggests that users can flexibly emphasize robustness, memory efficiency, throughput, or fairness in ranking models, supplementing accuracy-based evaluations.*

## 4. Reproducibility, Accessibility, and Compatibility Guarantees

DynaBench Server enforces several critical properties for scientific benchmarking [2106.06052]:

- **Reproducibility:**  
All models are run in identical containerized environments (same TorchServe/TF-Serving version, system libraries, CPU VM type). No self-reported metrics are accepted; results (including predictions and resource usage) are platform-collected.
- **Accessibility:**  
Models are hosted in the cloud and made accessible via a simple prediction API. Researchers and practitioners can interactively query any registered model (via browser or CLI) without needing local compute resources.
- **Backwards/Forwards Compatibility:**  
New datasets or metrics (e.g., future stress sets or newly released automated metrics) can be retroactively applied: the scheduler re-enqueues prior models for batch evaluation. Submitted models are immutable with version tags (container hash, model card), and datasets are versioned; historical evaluation data remains accessible, supporting longitudinal comparisons.

## 5. API Interfaces and Interoperability

The platform exposes an inferred public API, enabling integration with external tools and continuous benchmarking workflows. While a full REST/OpenAPI schema is not provided, the following endpoints are documented [2106.06052]:

| Endpoint                       | Purpose                                                    | Example Payload/Result                                  |
|---------------------------------|------------------------------------------------------------|---------------------------------------------------------|
| POST /models                   | Register model, submit container and card                  | `{ "model_name":..., "handler_image":..., ... }`        |
| GET /models/{model_id}         | Retrieve model metadata and evaluation status              | `{ "model_id":"abc123", "status":"pending" }`           |
| POST /evaluations              | Initiate evaluation batch (model, task, datasets, metrics) | `{ "model_id":"abc123", "task":"NLI", ... }`            |
| GET /evaluations/{eval_id}/results | Retrieve scores and Dynascore for evaluation            | `{ ... "dynascore": 45.8, ... }`                        |
| GET /leaderboard               | Query/re-rank leaderboard by weights/datasets/tasks        | `[{"model_id": ..., "dynascore": ..., "rank": ...}]`    |

*A plausible implication is that users and automated pipelines can submit new models, trigger evaluations, and integrate results without direct UI interaction, similar to EaaS platforms such as CodaLab and EvalAI.*

## 6. Context, Significance, and Broader Impacts

DynaBench Server addresses limitations of traditional benchmarking, which relies on static test sets, single metrics, and self-reporting, by enabling dynamic, platform-controlled, and multi-dimensional evaluation workflows. The infrastructure-level standardization of hardware, serving, and metrics collection mitigates artifacts caused by inconsistent local experiments. The adoption of utility-based aggregation (Dynascore) foregrounds practical concerns—such as efficiency, fairness, and robustness—that are increasingly relevant to NLP system deployment and research.

Its integration of adversarial robustness and fairness perturbation as first-class evaluation axes enables systematic auditing and research replication. The inherent extensibility—inserting new datasets or metrics and retroactively re-scoring published models—supports evolving community standards and continual benchmarking, an increasingly critical requirement as NLP tasks and evaluation desiderata diversify [2106.06052].

Source: https://www.emergentmind.com/topics/dynabench-server