RISK-Bench: E-commerce Risk Benchmark

• RISK-Bench is a domain-specific evaluation suite featuring 802 single-step and 320 multi-step trajectories to assess realistic e-commerce risk management workflows.
• It utilizes DOM-indexed tool actions and algorithmically determined difficulty levels to rigorously test GUI agents’ multi-step reasoning and process fidelity.
• It underpins reinforcement fine-tuning in RISK-R1 and set the methodological groundwork for subsequent benchmarks like RiskWebWorld in risk evaluation.

Searching arXiv for papers specifically discussing “RISK-Bench” in e-commerce risk management and closely related follow-up work. RISK-Bench is the benchmark component of the RISK framework for GUI agents in e-commerce risk management. It is a domain-specific evaluation suite built around standardized web-interaction trajectories rather than generic grounding or consumer web navigation tasks, and it is intended to measure how well an agent can perform realistic risk-management workflows involving information search and website verification in real browser environments (Chen et al., 26 Sep 2025). Within the broader framework, RISK-Data serves as the training corpus, RISK-Bench serves as the held-out benchmark, and RISK-R1 provides the reinforcement fine-tuning framework evaluated against it (Chen et al., 26 Sep 2025). Later work positions RiskWebWorld as an interactive successor that operationalizes similar e-commerce risk-control domains in a larger environment-backed setting, which suggests that RISK-Bench occupies an important earlier role in structuring standardized evaluation for this application area (Chen et al., 15 Apr 2026).

1. Position within the RISK framework

RISK-Bench is introduced as a benchmark with 802 single-step and 320 multi-step trajectories for standardized evaluation of GUI agents in e-commerce risk management (Chen et al., 26 Sep 2025). Its function is explicitly separable from the other two components of the framework. RISK-Data contains 8,492 single-step and 2,386 multi-step trajectories used for training, whereas RISK-Bench is the reserved benchmark subset deliberately excluded from training in order to support clean offline and online evaluation (Chen et al., 26 Sep 2025).

The benchmark is motivated by limitations in prior GUI and web-agent benchmarks. Existing resources are described as focusing on generic web or desktop grounding, often with synthetic webpages, short horizons, or evaluation criteria centered on click coordinates rather than the DOM-indexed tool abstractions used by frameworks such as Browser Use (Chen et al., 26 Sep 2025). RISK-Bench instead targets complex, domain-specific, stateful workflows characteristic of e-commerce risk management, such as information aggregation across sites, website verification, and multi-step decision processes (Chen et al., 26 Sep 2025).

This positioning is reinforced by later work on RiskWebWorld, which adopts the taxonomy from prior work on RISK and expands it into a larger interactive benchmark for production risk-control pipelines across 8 business domains (Chen et al., 15 Apr 2026). That relationship suggests that RISK-Bench established the earlier benchmarking template for standardized evaluation in this vertical, while RiskWebWorld extends it toward more realistic, long-horizon, environment-backed assessment (Chen et al., 15 Apr 2026).

2. Benchmark construction and trajectory types

RISK-Bench is built from the same collection and post-processing pipeline as RISK-Data. The source trajectories are collected via Browser Use together with Qwen-VL-Max in real browser environments, with screenshots, DOM trees, and tool-based actions recorded over multi-round interactions (Chen et al., 26 Sep 2025). Post-processing includes trajectory filtering, step cleaning, information refinement, data augmentation, and the chaining of steps into multi-step “think-action-observation” loops (Chen et al., 26 Sep 2025).

The benchmark has two structural units.

First, single-step trajectories consist of one question, one GUI state, and one ground-truth tool call list. These evaluate webpage perception and element manipulation at a single decision point (Chen et al., 26 Sep 2025). RISK-Bench contains 802 such trajectories (Chen et al., 26 Sep 2025).

Second, multi-step trajectories consist of a sequence of steps, each with a current subgoal, screenshot, DOM, and ground-truth tool list. These evaluate task-level process understanding, planning, and correction capability over longer workflows (Chen et al., 26 Sep 2025). RISK-Bench contains 320 multi-step trajectories (Chen et al., 26 Sep 2025).

The benchmark samples are selected so that they span the same kinds of scenarios seen in the training corpus but with more difficult multi-step compositions. This is reflected in the difficulty distribution: single-step benchmark cases are 47% easy, 25% moderate, and 28% difficult, whereas multi-step cases are 30% easy, 17% moderate, and 53% difficult (Chen et al., 26 Sep 2025). A plausible implication is that RISK-Bench is intentionally stress-oriented on longer-horizon reasoning rather than being a simple held-out sample of the training distribution.

3. Difficulty grading and benchmark semantics

A distinctive feature of RISK-Bench is that its difficulty labels are defined algorithmically using the empirical accuracy of an advanced MLLM rather than heuristic properties such as tool count or bounding-box size (Chen et al., 26 Sep 2025). For each question, Qwen-VL-Max is queried 5 times, and the resulting empirical accuracy determines whether the item is labeled easy, moderate, or difficult (Chen et al., 26 Sep 2025).

The grading rule is:

$$a(q) = \frac{\#\text{correct responses among 5}}{5}$$

and

$$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$

This design is used consistently in both RISK-Data and RISK-Bench (Chen et al., 26 Sep 2025). The paper contrasts it with a rule-based alternative using the number of tools, and reports that the heuristic alternative degrades performance when used for training reweighting, which suggests that the benchmark’s notion of difficulty is intended to approximate semantic and procedural hardness rather than superficial interaction complexity (Chen et al., 26 Sep 2025).

Each benchmark trajectory includes the task description, the sequence of GUI states, the ground-truth tool calls including tool types and DOM indices, and the difficulty level (Chen et al., 26 Sep 2025). Because the benchmark is aligned with the Browser Use interaction model, the semantics of correctness are expressed in terms of tool selection and DOM-targeted execution rather than pixel-level interaction (Chen et al., 26 Sep 2025).

4. Task domains and GUI interaction structure

RISK-Bench covers two broad capability categories: Information Search and Website Verification (Chen et al., 26 Sep 2025). Within these, the benchmark represents several e-commerce risk-management subdomains.

In Information Search, tasks include Product Risk Profile, Merchant Risk Profile, Client Risk Profile, Logistics and Supply Chain Tracking, and Customs Declaration and Clearance Audits (Chen et al., 26 Sep 2025). These tasks require agents to retrieve and verify information from search engines, business registries, logistics portals, customs websites, and related sources (Chen et al., 26 Sep 2025).

In Website Verification, tasks include Transaction Laundry Detection, Website Accessibility and Identity Verification, Content Consistency Assurance, and Secure Payment Channel Validation (Chen et al., 26 Sep 2025). These tasks simulate website authenticity checks, payment-channel validation, cross-page consistency inspection, and the discovery of suspicious or illicit content embedded in otherwise legitimate web properties (Chen et al., 26 Sep 2025).

The benchmark’s interaction model is rooted in DOM-based tool usage. Actions include tools such as click_element_by_index, search_google, and done, with arguments expressed in DOM-index form rather than coordinate form (Chen et al., 26 Sep 2025). The benchmark also includes form filling, table interpretation, dynamic content handling, and multi-page navigation across real websites (Chen et al., 26 Sep 2025).

A representative multi-step benchmark episode proceeds from a search engine to an official registry site, then through a search form, result table, and terminal verification step, ending with a structured done action summarizing whether risk indicators are present (Chen et al., 26 Sep 2025). This suggests that RISK-Bench is fundamentally organized around procedural web reasoning rather than isolated perceptual subproblems.

5. Offline and online evaluation protocol

RISK-Bench supports both offline replay-based evaluation and online live-site evaluation (Chen et al., 26 Sep 2025). The two settings use related but distinct metrics.

For offline single-step evaluation, the metric is accuracy of tool calls. Let $T_i$ be the predicted tool list for trajectory $i$, and $T_i^{\text{gt}}$ the ground-truth tool list. A prediction is counted correct if the tool-list F1 exceeds 0.5 (Chen et al., 26 Sep 2025):

$$\text{Correct}(T_i) = \begin{cases} 1 & \text{if } F_1(T_i, T_i^{\text{gt}}) > 0.5,\ 0 & \text{otherwise}. \end{cases}$$

Single-step accuracy is then:

$$\text{Acc}_{\text{single}} = \frac{1}{N_{\text{single}}} \sum_{i=1}^{N_{\text{single}}}\text{Correct}(T_i).$$

For offline multi-step evaluation, correctness is strict at the trajectory level. Each step is judged by the same tool-list F1 threshold, and a multi-step trajectory is successful only if all steps are correct (Chen et al., 26 Sep 2025):

$$\text{CorrectStep}(j,k) = \begin{cases} 1 & \text{if } F_1(T_{j,k}, T_{j,k}^{\text{gt}}) > 0.5,\ 0 & \text{otherwise}. \end{cases}$$

$$\text{Success}(j) = \begin{cases} 1 & \text{if } \prod_{k=1}^{n_j}\text{CorrectStep}(j,k)=1,\ 0 & \text{otherwise}. \end{cases}$$

$$\text{SuccessRate}_{\text{multi}} = \frac{1}{N_{\text{multi}}} \sum_{j=1}^{N_{\text{multi}}}\text{Success}(j).$$

For online evaluation, tasks are executed against live websites using Browser Use. Two metrics are used: task completion rate and task success rate (Chen et al., 26 Sep 2025). Completion requires finishing within a 20-step limit, whereas success additionally requires that the tool calls and final state satisfy all correctness criteria (Chen et al., 26 Sep 2025). Formally:

$$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$0

$$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$1

Offline evaluation is deterministic with temperature $$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$2 via vLLM, while online evaluation uses real-time page loads in the live browser environment (Chen et al., 26 Sep 2025). This distinction is important because the benchmark is intended to test both idealized replay competence and robustness under deployment-like web variability.

6. Relationship to RISK-R1 and empirical results

RISK-Bench is tightly coupled to RISK-R1, the reinforcement fine-tuning framework in the same paper. The reward structure in RISK-R1 is designed directly around the benchmark’s output format, step-level correctness, multi-step process structure, and difficulty labels (Chen et al., 26 Sep 2025).

At each step, RISK-R1 uses a structured output with fields such as think, evaluation_previous_goal, memory, next_goal, and action (Chen et al., 26 Sep 2025). It defines a format reward $$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$3, a stepwise accuracy reward $$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$4, a process reweight $$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$5 that gives later steps larger weight, and a level reweight $$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$6 tied to the easy/moderate/difficult grading shared with RISK-Bench (Chen et al., 26 Sep 2025).

The process reweight is:

$$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$7

and the overall step reward is:

$$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$8

with default $$L(q) = \begin{cases} \text{easy} & \text{if } a(q) = 1.0,\ \text{moderate} & \text{if } 0.2 \le a(q) < 1.0,\ \text{difficult} & \text{if } a(q) < 0.2. \end{cases}$$9 and $T_i$0 (Chen et al., 26 Sep 2025). Difficulty reweighting is:

$T_i$1

The benchmark results show that RISK-R1-7B achieves the strongest reported performance on RISK-Bench (Chen et al., 26 Sep 2025). On offline single-step evaluation, it reaches 88.3% overall accuracy, compared with 83.5% for RISK-SFT-7B, 81.5% for GPT-4o, and 80.6% for Qwen2.5-VL-72B (Chen et al., 26 Sep 2025). Its gains are concentrated in harder items: it improves from 83.2% to 90.1% on moderate single-step tasks and from 52.5% to 65.5% on difficult single-step tasks relative to the SFT baseline (Chen et al., 26 Sep 2025).

On offline multi-step evaluation, RISK-R1-7B reaches 82.8% task success rate, compared with 75.3% for RISK-SFT-7B, 74.0% for GPT-4o, and 67.8% for Qwen2.5-VL-72B (Chen et al., 26 Sep 2025). This indicates that the benchmark is not merely detecting perceptual improvement; it is responsive to training interventions aimed at longer-horizon process fidelity.

On online RISK-Bench tasks, RISK-R1-7B attains the highest reported task success rate of 70.5%, with task completion rate 87.6% (Chen et al., 26 Sep 2025). Qwen2.5-VL-72B attains a slightly higher completion rate of 88.7% but lower success rate of 68.9%, which suggests that RISK-Bench’s online protocol distinguishes between merely finishing a workflow and finishing it correctly (Chen et al., 26 Sep 2025).

7. Subsequent development and broader significance

RISK-Bench is best understood as an early, domain-specific benchmark for GUI agents in e-commerce risk management, organized around realistic web tasks, DOM-grounded action semantics, and difficulty-aware evaluation (Chen et al., 26 Sep 2025). Its importance lies less in scale than in the way it operationalizes a specialized professional workflow that generic web benchmarks do not capture.

RiskWebWorld extends this line of work by introducing 1,513 interactive Web UI tasks across 8 core business domains sourced from production risk-control pipelines, with uncooperative websites, environmental hijacks, and a Gymnasium-compliant environment for agentic reinforcement learning (Chen et al., 15 Apr 2026). That later benchmark is described as “the first highly realistic interactive benchmark for evaluating GUI agents in e-commerce risk management,” and it explicitly adopts the taxonomy from prior work on RISK (Chen et al., 15 Apr 2026). This suggests a developmental trajectory in which RISK-Bench provides the benchmark backbone for offline and online trajectory evaluation, while RiskWebWorld pushes the same research program toward more realistic, environment-native, long-horizon assessment.

A plausible implication is that RISK-Bench’s main historical significance is methodological. It established a standardized, held-out benchmark structure for a domain where evaluation had previously been fragmented across generic GUI tasks and ad hoc demonstrations. Its use of real webpages, DOM-based actions, multi-step trajectories, and difficulty labels tied to advanced-model performance gives it a clear identity as an evaluation instrument for e-commerce risk-management agents (Chen et al., 26 Sep 2025).

8. Limitations and interpretation

The framework paper identifies several limitations that directly affect RISK-Bench. First, reinforcement fine-tuning in the reported setup uses only single-step trajectories due to GPU memory limits, so multi-step competence is evaluated more strongly than it is directly optimized (Chen et al., 26 Sep 2025). Second, process reweighting is used to simulate multi-step importance in offline training, but this does not fully reproduce the dynamics of live multi-step interaction (Chen et al., 26 Sep 2025). Third, the authors explicitly note the need to collect more diverse and complex multi-step instances, implying that the current benchmark, while substantial for its niche, is not exhaustive (Chen et al., 26 Sep 2025).

The paper also does not state that RISK-Bench is publicly released, in contrast to some other benchmarks in the broader risk-evaluation landscape. It is described as an internal benchmark within the RISK framework at Ant International (Chen et al., 26 Sep 2025). This distinguishes it from later open environment-backed systems such as RiskWebWorld, whose Gymnasium-compliant infrastructure is described in public terms (Chen et al., 15 Apr 2026).

These constraints do not diminish the benchmark’s role, but they do clarify how it should be interpreted. RISK-Bench is a carefully constructed in-domain evaluation suite, not a universal GUI benchmark and not yet a full interactive environment in the later RiskWebWorld sense. Its strongest contribution is the disciplined alignment between benchmark structure, task semantics, reward design, and deployment-oriented evaluation for e-commerce risk management (Chen et al., 26 Sep 2025).