---
title: 'RSBench: Evaluating Safe Reasoning Paths'
url: https://www.emergentmind.com/topics/reasoning-path-benchmark-rsbench
type: topic
---

# RSBench: Evaluating Safe Reasoning Paths

Searching arXiv for the benchmark paper and related reasoning-path benchmarks to ground the article in current literature.
The **Reasoning Path Benchmark (RSBench)** is a benchmark introduced to evaluate not only the final answer of a multimodal large language model (MLLM), but the **quality of its intermediate chain-of-thought (CoT) reasoning path** under safety-sensitive multimodal inputs. In the formulation associated with cross-modal safety, RSBench is presented as the evaluation counterpart to the **Safe-Semantics-but-Unsafe-Interpretation (SSUI)** setting: image-text pairs in which each modality is safe in isolation, but the combined semantics can be unsafe. The benchmark is motivated by the claim that prior multimodal safety benchmarks “predominantly focus on assessing the final output of the model” and “often overlook the quality of the intermediate CoTs of the safety reasoning process,” thereby leaving the integrity of the reasoning process underexamined [2509.12060]. Related work on reasoning-path evaluation in language-only, visual, and logical settings reinforces this broader shift from final-answer scoring toward path-sensitive assessment, though those benchmarks target different problem domains and use different protocols [2410.03117], [2503.11557], [2602.21044].

## 1. Definition and benchmark motivation

RSBench is introduced as a benchmark for the **safety of the reasoning path itself**, the **effectiveness of the reasoning path**, and their **joint satisfaction**. Its central premise is that safety evaluation should not stop at the endpoint. A model may produce a safe final refusal or safe-looking answer while nevertheless traversing unsafe, fragile, or semantically incorrect intermediate reasoning. RSBench is therefore intended to benchmark the **integrity of the thought process**, not only the endpoint [2509.12060].

The motivating failure mode is **implicit reasoning risk**. In this setting, the image and text are safe in their own modal, but the semantic combination of them is potentially unsafe, which tends to generate harmful output of MLLM. The benchmark is specifically designed for such cases because they require long, fragile reasoning chains in which an error at an intermediate step can redirect the model toward unsafe or ineffective conclusions. This places RSBench within a broader research trend that treats reasoning trajectories as first-class evaluation objects rather than latent by-products of answer generation. ProcBench, for example, evaluates whether a model can follow an explicit multi-step procedure step by step, while VERIFY evaluates multimodal reasoning fidelity via human-annotated reasoning paths; both similarly reject final-answer-only evaluation as insufficient, though neither addresses cross-modal safety in the SSUI sense [2410.03117], [2503.11557].

A necessary clarification is that the label **“RSBench”** is not unique across arXiv-indexed research. In addition to the multimodal safety benchmark introduced in the SSUI/SRPO paper, the name or closely related labeling is also used for a robotic path-planning proof benchmark and for a benchmark suite on reasoning shortcuts written as **rsbench** [2603.19464], [2406.10368]. In the cross-modal safety literature, however, **Reasoning Path Benchmark (RSBench)** denotes the benchmark tied to SSUI and implicit reasoning risk [2509.12060].

## 2. RSBench within SSUI and cross-modal safety

RSBench is tightly coupled to the **SSUI dataset**. SSUI provides training and reference reasoning-path data: image-text pairs where each modality is safe alone, but their combination can imply unsafe behavior or unsafe outcomes. The dataset includes interpretable reasoning chains explaining why the combined input falls into implicit reasoning risk. In this formulation, **SSUI** supplies annotated safe and unsafe reasoning paths, whereas **RSBench** evaluates CoT reasoning safety and effectiveness on similar cross-modal safety cases [2509.12060].

The underlying safety problem is described as **cross-modal safety** rather than unimodal toxicity or direct unsafe prompting. The issue is not that either modality is overtly harmful. Rather, the dangerous semantic interpretation emerges only after multimodal integration. This makes the benchmark distinct from conventional refusal or moderation tests, which usually judge surface harmfulness of the query or safety of the final response. RSBench instead probes whether the model can remain aligned with human safety values throughout the reasoning path required to detect the latent risk [2509.12060].

This benchmark design has close conceptual affinities with other reasoning-path evaluations that ask whether intermediate reasoning is faithful, grounded, or complete. VERIFY, for instance, is built to expose cases in which an MLLM answers correctly for the wrong visual reasons, and it uses human-annotated reasoning paths to make such failures visible. LogicGraph similarly argues that standard evaluations score only the final answer and thereby underexplore the structure of valid derivations. The cross-modal safety version of RSBench differs in target phenomenon, but it belongs to the same methodological family: benchmarks that assess reasoning quality directly rather than inferring it from answer correctness [2503.11557], [2602.21044].

## 3. Construction, task setup, and arbitration protocol

The paper gives only a high-level description of RSBench, but several structural features are explicit. RSBench is built using the same conceptual framework as SSUI: image-text pairs that require safety-aware reasoning. The benchmark is designed around **CoT reasoning paths** rather than only final answers. The task is to evaluate each model-generated reasoning path and determine whether it satisfies two conditions: **Safety**, meaning the reasoning path is safe, and **Effectiveness**, meaning the reasoning path is practically useful [2509.12060].

The benchmark introduces two binary indicators: **safe / unsafe** and **effective / ineffective**. A reasoning path can therefore be classified according to whether it is safe, effective, both, or neither. RSBench then combines these judgments into a third metric reflecting joint attainment of both properties. The evaluation is performed on reasoning paths, not only on final responses [2509.12060].

A distinctive protocol choice is that RSBench **“leverages GPT-4o as an arbitration model.”** GPT-4o is used to judge whether a response path is safe and whether it is effective. For each model and each response path, GPT-4o makes those two judgments, after which the benchmark computes aggregate rates. This makes RSBench an example of automated path arbitration rather than symbolic verification or purely human grading [2509.12060].

That choice also implies a methodological limitation. The paper does not provide a deep critique of the arbitration dependency, but the dependence of judgments on an automated judge model is an implied limitation of the benchmark protocol. This contrasts with the design of LogicGraph, which explicitly motivates solver-backed symbolic verification because LLM judges can produce false negatives on compressed but valid inferences and false positives by filling in missing premises themselves. A plausible implication is that RSBench prioritizes scalable evaluation of safety-sensitive multimodal reasoning, whereas LogicGraph prioritizes formal proof validation in logical derivation tasks [2602.21044].

## 4. Metrics and evaluation target

RSBench defines three metrics exactly as follows:

$$
SR = \frac{1}{N} \sum_{i=1}^{N_h} f_h(i) \ \mbox{,} \ ER = \frac{1}{N}\sum_{j=1}^{N_r} f_r(j),
$$

where \(N_h\), \(N_r\), and \(N\) represent the number of safe responses, effective responses, and total responses, respectively. \(f_h(i)\) and \(f_r(j)\) are indicator functions. \(f_h(i)=1\) if the \(i\)-th query yields a safe response and 0 otherwise. Similarly, \(f_r(j)=1\) if the \(j\)-th query yields an effective response and 0 otherwise [2509.12060].

The joint metric is:

$$
SER = \frac{1}{N} \sum_{k=1}^{N} [f_h(k) \cdot f_r(k)].
$$

Under this metric suite, **SR** measures how often the reasoning path is judged safe, **ER** measures how often it is judged practically effective, and **SER** measures how often both are simultaneously true. The benchmark’s central idea is therefore not generic task success, but path-level safety-quality assessment under multimodal risk [2509.12060].

These metrics place RSBench in contrast with benchmarks that score only the final answer or only exact path reproduction. ProcBench, for example, uses **Prefix Match Length (PML)**, **Prefix Accuracy (PA)**, **Sequential Match (SM)**, and **Final Match (FM)** to evaluate execution of explicit procedures step by step. VERIFY uses stage-level reasoning-fidelity measures over recognition, abstraction, and deduction, along with perception metrics. LogicGraph measures coverage over the complete proof set, including diversity and versatility. RSBench is more specialized: it operationalizes reasoning quality through the paired lenses of **safety** and **practical usefulness** in SSUI-style multimodal settings [2410.03117], [2503.11557], [2602.21044].

## 5. Empirical comparisons and reported performance

RSBench is used to evaluate **Qwen2.5-SRPO**, its base model **Qwen-2.5VL**, and **8 families of advanced MLLMs**, including open-source and closed-source systems. The broader experiments referenced in the paper include closed-source models such as **GPT-4o, Claude-3.5-Sonnet2, Gemini-1.5-Pro, Gemini-2.0-Flash**, and open-source models such as **DeepSeek-VL, VILA-1.5-7B, MiniGPT-v2, LLaVA-v1.5-7B, LLaVA-v1.6-mistral-7B, InternVL2.5-8B, MiniCPM-LLaMA3-V 2.5, MiniCPM-V-2.6, Qwen2-VL-7B, GLM-4v-9B, LLaVA-NeXT-LLaMA3, Qwen-2.5VL-7B** [2509.12060].

For RSBench specifically, the paper reports that **Qwen2.5-SRPO significantly outperforms its base model Qwen-2.5VL**, achieving **more than 20% absolute gain in both SR and ER**. It also reports that the SRPO-trained model shows **safer and more effective reasoning paths against the selected leading closed-source MLLMs**. The broader claim is that SRPO-trained models achieve **state-of-the-art results on key safety benchmarks, including RSBench**, and outperform both open-source and top-tier commercial MLLMs [2509.12060].

The significance of these results lies less in raw benchmark leadership than in what is being improved. RSBench is intended to measure whether the model’s intermediate reasoning remains aligned with human safety values. Accordingly, the reported gains are evidence that optimizing reasoning paths can materially affect both safety and usefulness, rather than only refusal frequency or answer filtering. This differs from the interpretive goal of VERIFY, where the key finding is that even the best MLLMs remain below random chance on hard visual reasoning problems and often exhibit imbalance between perception and deduction. In RSBench, the main result is not merely that the task is hard, but that targeted path optimization measurably improves path-level safety and effectiveness [2503.11557].

## 6. SRPO and optimization of intermediate branch decisions

The method used to improve RSBench performance is **Safety-aware Reasoning Path Optimization (SRPO)**. SRPO is designed to optimize the model not only on the final reference reasoning path, but also on **intermediate branch choices** in the reasoning chain. This is the methodological point most directly tied to RSBench: rather than rewarding only the eventual answer, SRPO attempts to shape the continuation chosen at each intermediate reasoning state so that favorable branches are encouraged and unfavorable branches are penalized [2509.12060].

The method has two stages. The first is **Generative Exploration**, in which reasoning branches are expanded step by step from reference CoT paths in SSUI, generating both favorable and unfavorable branches at intermediate states and using temperature sampling to explore the solution space. The second is **Path Optimization**, in which the model is trained on both the reference reasoning paths and the contrastive positive and negative branches [2509.12060].

For each intermediate state \(v_i\), SRPO constructs a **positive instance** \(\tau_i^+\) with a correct continuation and a **negative instance** \(\tau_i^-\) with an incorrect continuation, and applies the path-level contrastive objective

$$
\mathcal{J}_{\text{Align}, i}(\theta) = - k \cdot \log\sigma \left( \mathcal{L}(\tau_i^+ | \theta) - \mathcal{L}(\tau_i^- | \theta) \right),
$$

where \(k\) is a scaling factor [2509.12060].

The paper also states that the model learns the reference path through

$$
\mathcal{J}_{\text{Ref}(\theta) = - \mathbb{E}_{(v_{i-1}, v_i) \in \tau^*} [\log p_\theta(v_i | v_{i-1})],
$$

and that the final objective balances reference-path learning and contrastive alignment through a coefficient \(\lambda\) [2509.12060].

The conceptual import is explicit: SRPO optimizes the **entire chain of reasoning**, but especially the **intermediate transitions** between reasoning states. This directly matches the evaluation goal of RSBench, which measures whether those reasoning paths are both safe and useful. A plausible implication is that RSBench and SRPO form a tightly coupled benchmark-method pair: the benchmark exposes failures in path integrity, and the training framework directly targets those failures at branch points rather than only at terminal outputs [2509.12060].

## 7. Limitations, interpretive cautions, and relation to neighboring benchmarks

The paper does not provide a long dedicated RSBench failure-analysis section, but it reports a relevant trade-off involving the loss weight \(\lambda\). A **smaller \(\lambda\)** overemphasizes the reference safe path and underemphasizes exploration, whereas a **larger \(\lambda\)** overemphasizes explored branches; both extremes are suboptimal. This indicates that reasoning-path optimization requires a balance between grounding in the reference path and exploration of alternative branches. Because RSBench rewards both safety and effectiveness of the reasoning path, that balance is likely important for benchmark performance [2509.12060].

Another implied limitation is evaluator dependence. Since GPT-4o serves as the arbitration model, benchmark judgments inherit the properties and possible biases of the judge. The paper does not present a detailed analysis of this dependency. Related benchmark work makes such concerns more explicit. VERIFY reports that several strong LLM judges gave high similarity scores even when human evaluators found the answers wrong, revealing an overestimation bias and a bias toward completed choices. LogicGraph likewise argues that LLM-as-a-judge can mis-evaluate proof validity, motivating its Prover9-based symbolic verifier. These findings do not invalidate RSBench, but they situate its evaluation protocol within a broader debate about judge-model reliability in reasoning-path assessment [2503.11557], [2602.21044].

RSBench is also best understood as one instance of a wider benchmark design movement. ProcBench isolates **multi-step inference** by explicitly providing the procedure and scoring the predicted trajectory. VERIFY evaluates **visual reasoning fidelity** using human-written reasoning paths. LogicGraph evaluates **multi-path logical reasoning** by measuring coverage over an exhaustive set of minimal proofs. The multimodal safety RSBench differs from each of these in target phenomenon, data modality, and scoring criterion, but shares the same foundational methodological departure from final-answer-only evaluation: reasoning quality itself is the object of measurement [2410.03117], [2503.11557], [2602.21044].

A final point of disambiguation is terminological. The literature contains another benchmark described as **RSBench** for **research-level algorithm verification** in robotic path planning, where the task is to prove approximation optimality for algorithms over 34 tasks from 11 peer-reviewed robotics papers [2603.19464]. It also contains **rsbench**, a benchmark suite for studying **reasoning shortcuts** and concept quality in neuro-symbolic learning [2406.10368]. These are distinct from the multimodal safety **Reasoning Path Benchmark** introduced in the SSUI/SRPO framework [2509.12060].

Source: https://www.emergentmind.com/topics/reasoning-path-benchmark-rsbench