---
title: 'CIEBench: Multi-Domain Evaluation Benchmark'
url: https://www.emergentmind.com/topics/ciebench
type: topic
---

# CIEBench: Multi-Domain Evaluation Benchmark

CIEBench is an overloaded benchmark name in the arXiv literature. In one usage, it denotes a benchmark for complex, reasoning-based image editing introduced alongside CIELR, where each sample couples an input image with an implicit reasoning query and a ground-truth edit mask [2510.27335]. In another, it denotes a cross-platform benchmark for interval computation libraries, coupled with reference solutions computed with exact arithmetic to evaluate correctness, interval tightness, performance, and consistency across architectures, operating systems, and compilers [2110.06215]. A further source of ambiguity is CIAware-Bench, which is also referred to as CIEBench in places, although its title and main text use CIAware-Bench [2606.11063].

## 1. Terminological scope

The name “CIEBench” does not identify a single benchmark family. In the materials considered here, it is attached to at least two distinct artifacts and also appears as an alternate label in a third.

| Name in use | Domain | Paper |
|---|---|---|
| CIEBench | Complex, reasoning-based image editing | [2510.27335] |
| CIEBench | Cross-platform benchmark for interval computation libraries | [2110.06215] |
| CIAware-Bench, also referred to as CIEBench in places | Control intervention awareness across frontier LLMs | [2606.11063] |

This naming overlap matters because the underlying evaluation targets are unrelated. One benchmark tests whether an image editor can infer an implicit target before editing; another tests whether interval arithmetic implementations are correct, tight, fast, and portable; the third tests whether an acting model can detect that a control protocol altered its trajectory. A plausible implication is that citations by arXiv identifier are necessary whenever the bare name “CIEBench” appears in technical discussion.

## 2. CIEBench in reasoning-based image editing

In "Understanding the Implicit User Intention via Reasoning with Large Language Model for Image Editing" [2510.27335], CIEBench is introduced as a benchmark for reasoning-based image editing rather than simple instruction following. The motivating distinction is between explicit edits such as “replace the apple with an orange” and implicit edits such as “replace the food containing the most vitamin C with an orange.” In the latter case, a model must identify the correct object through reasoning and then edit it while preserving the rest of the image.

The benchmark contains **86 image-query-mask triplets**. Each sample includes an input image \(x\), an implicit reasoning query \(q\), and a binary ground-truth mask \(m_{gt}\) for the region to be edited. The dataset is built from **ReasonSeg** images and masks: the authors “carefully selected 86 high-quality images from ReasonSeg” and manually transformed the original segmentation instructions into implicit, reasoning-intensive editing queries. The paper presents CIEBench as a **test benchmark** rather than reporting a train/validation/test split.

CIEBench includes the three reasoning-editing types described in the paper: **source object identification**, **target object identification**, and **multi-step edits**. These task types are intended to expose failures that conventional image editing benchmarks can miss, especially failures in **multi-step inference**, **implicit target identification**, and **semantic correctness under reasoning**. The benchmark is therefore positioned as complementary to datasets such as **MagicBrush**, which contains 1053 samples but mainly explicit instructions, and the **SmartEdit Reasoning Scenario Set**, which contains 60 samples with implicit instructions but does not eliminate the broader scarcity of public resources for this setting [2510.27335].

## 3. Evaluation methodology and the CIELR framework

CIEBench was introduced together with **CIELR** (“Complex Image Editing via LLM Reasoning”) [2510.27335]. The framework decouples reasoning from editing by introducing a **structured semantic representation** of the image that includes **segmentation masks**, **object labels**, and **relative depth**. This representation is built in a zero-shot manner using **SAM2** for segmentation masks, **OWLv2** for semantic labels, and **DepthAnything** for depth estimation. An LLM then reasons over this structure, identifies the target region, and converts the implicit instruction into an explicit editing instruction and mask. When the initial representation is insufficient, CIELR performs iterative refinement through a **chain of structured semantic representation updates**.

The paper argues that conventional metrics such as **PSNR**, **SSIM**, **LPIPS**, and **CLIP similarity** are inadequate for reasoning-based editing because they do not reliably measure whether the edit satisfies the reasoning requirement. A visually plausible edit can therefore be semantically wrong. To address this, the authors propose **Image Difference Check Score (IDCS)**, an LLM-based metric defined in two stages:
\[
\mathrm{D}_\text{diff} = \mathcal{F}_\text{llm}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{llm}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})
\]
where \(\mathrm{x}\) is the original image, \(\mathrm{x}_\text{edit}\) is the edited image, \(\mathrm{q}\) is the original implicit query, \(\mathcal{F}_\text{llm}^\text{desc}\) describes the differences between the original and edited image, and \(\mathcal{F}_\text{llm}^\text{eval}\) judges whether those differences satisfy the query. The final score is \(\mathrm{S}_\text{RE} \in \{1,2,3,4,5\}\).

The design of IDCS makes the evaluation target explicit: it asks what changed in the image and whether those changes match the implicit instruction. This shifts measurement from low-level similarity to **semantic correctness**, which the paper treats as the central difficulty in reasoning-based editing.

## 4. Empirical results on the image-editing CIEBench

On CIEBench, the compared methods are **InstructPix2Pix**, **MagicBrush**, **InstructDiffusion**, **CIELR (I.P.)**, and **CIELR (I.A.)** [2510.27335]. The reported metrics are **PSNR**, **SSIM**, **LPIPS**, and **IDCS**. The table in the paper gives the following values: **InstructPix2Pix** reports \(9.794\) PSNR, \(0.337\) SSIM, \(0.517\) LPIPS, and \(1.200\) IDCS; **MagicBrush** reports \(9.632\), \(0.333\), \(0.509\), and \(1.380\); **InstructDiffusion** reports \(9.193\), \(0.312\), \(0.520\), and \(1.300\); **CIELR (I.P.)** reports \(25.096\), \(0.788\), \(0.095\), and \(1.988\); and **CIELR (I.A.)** reports **36.206**, **0.960**, **0.035**, and **2.366**. The paper specifically highlights the comparison between **CIELR (I.A.)** and **InstructPix2Pix**: PSNR **36.206** versus **9.794**, SSIM **0.960** versus **0.337**, and IDCS **2.366** versus **1.200**.

The ablation study isolates two components: **SSR** and **Chain**. On CIEBench, **SSR only** yields \(31.964\) PSNR, \(0.902\) SSIM, \(0.080\) LPIPS, \(1.686\) IDCS, and \(3.451\) seconds; **Chain only** yields \(33.287\), \(0.935\), \(0.056\), \(1.802\), and \(3.717\) seconds; **SSR + Chain** yields **36.206**, **0.960**, **0.035**, **2.366**, and **6.504** seconds. The factual interpretation given in the paper is that either component helps, but **both together** are best, while the cost is a longer runtime of about **6.5 seconds per image**.

The qualitative figure uses **blue boxes** for target regions, **red boxes** for incorrect or suboptimal edits, and **green boxes** for successful edits. The authors also note that **CLIP score does not reflect actual task success** in some examples. This supports their claim that CIEBench is specifically sensitive to reasoning failures rather than merely to visual quality degradation.

The benchmark’s limitations are also explicit. It has only **86 samples**; its implicit reasoning queries were **manually created** from ReasonSeg instructions; **IDCS** depends on LLM judgment; and the full method incurs nontrivial runtime. The conclusion mentions extending the framework to **video reasoning editing** and suggests **interactive creative workflows** as future directions [2510.27335].

## 5. CIEBench in interval computation libraries

In "A Cross-Platform Benchmark for Interval Computation Libraries" [2110.06215], CIEBench designates the **first benchmark for interval computations** with **reference solutions computed using exact arithmetic**. Its stated purpose is to compare interval arithmetic libraries on four properties: **correctness**, **interval size / tightness**, **performance**, and **consistency / portability**. The motivation is that interval arithmetic is widely used to certify floating-point computations, but practical implementations are not standardized and are sensitive to compiler, operating-system, and hardware behavior.

The benchmark compares four open-source C/C++ interval libraries: **filib**, **filib++**, **Boost**, and **BIAS**. The paper further treats the operational modes of some libraries as distinct interval types. For **filib++**, these are **native**, **pred**, and **multiplicative**. For **BIAS**, the paper considers **ROUND DOWN**, **ROUND UP**, and **ROUND NEAR** separately because the documentation does not clearly specify how these affect interval operations.

CIEBench is built around an expression suite with **28 expressions** authored by the paper’s authors and **104 expressions** taken from **FPBench**. The 28 hand-crafted expressions include the four basic arithmetic operations, four transcendental functions \(\sqrt{}\), \(\exp\), \(\sin\), and \(\cos\), ten composite expressions with only arithmetic, and ten composite expressions mixing arithmetic and transcendental functions. For each expression, the benchmark generates **one million valid inputs**. Inputs and outputs are converted to a rational representation using **GMP**, and the exact query is evaluated using **Mathematica**. Interval width is also computed exactly by rational subtraction, so the validation machinery does not inherit floating-point error from the evaluated libraries.

Performance is measured by generating **1,000 inputs per expression** and evaluating each expression **10,000 times per input**. The benchmark was run on four platforms: **Windows** on an Intel Core i7-8700K with **MSVC 14.27.29110**; **macOS Intel** on an Intel Core i9 with **Apple clang 12.0.0**; **macOS ARM** on **Apple M1** with **Apple clang 12.0.0**; and **Linux** on an **AMD EPYC 7452 32-Core Processor** with **GCC 9.2.1** [2110.06215].

## 6. Findings, recommendations, and naming overlap with CIAware-Bench

The interval-computation CIEBench yields several concrete findings [2110.06215]. All libraries were correct on the basic arithmetic expressions. **Boost failed on transcendental functions**, specifically for \(\exp\) and trigonometric functions. For composite expressions using only arithmetic, all libraries were correct; when transcendental functions were mixed into composite expressions, **BIAS produced incorrect intervals** on at least one expression. On the **FPBench** expressions, **filib** and **filib++ pred/multiplicative** were correct on all tests, while **filib++ native** failed on four FPBench expressions. The paper’s bottom line on correctness is that the only methods correct for **all tests** were **filib**, **filib++ pred**, and **filib++ multiplicative**.

On interval size, libraries using system rounding modes—**Boost**, **filib++ native**, and **BIAS**—tended to produce smaller intervals for pure arithmetic, while **filib++ multiplicative** produced the largest intervals on arithmetic-only expressions. On performance, the general ordering for the 28 main expressions is reported as **Boost** slowest overall, then **BIAS**, then **filib**, while **filib++ native** is highly optimized and much faster than the other two traditional rounding-mode libraries. For more complex workloads, the overall ordering is generally: **filib++ multiplicative**, **filib++ pred**, **filib++ native**, **filib**, **BIAS**, **Boost**. On portability and consistency, **filib** is described as the only library that was **correct**, **consistent**, **portable**, and **reasonably fast**, and the paper explicitly recommends **filib** as the best option among those tested.

The same section of the literature also contains a separate nomenclature issue. In "CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs" [2606.11063], the paper states that the benchmark is also referred to as **CIEBench** in places, but its **title and main text use CIAware-Bench**. That benchmark measures **control intervention (CI) awareness**: whether an acting model can recognize, from a trajectory alone, whether a step was produced by itself or by another model under a control intervention. It spans **essay writing**, **BigCodeBench**, **Bash Arena**, and **SHADE-Arena**; frames evaluation as a balanced binary classification problem with chance level **0.5**; and reports low to moderate CI awareness across **eleven frontier models** from four providers. This is not the same benchmark as either the reasoning-based image-editing CIEBench or the interval-computation CIEBench.

Taken together, these uses show that “CIEBench” is a domain-dependent label rather than a stable benchmark brand. In current arXiv usage, the most direct interpretations are either the **86-sample reasoning-based image editing benchmark** paired with **IDCS** [2510.27335] or the **exact-arithmetic-backed benchmark for interval computation libraries** [2110.06215], while **CIAware-Bench** should be cited under its canonical name even when alternate mentions of CIEBench appear [2606.11063].

Source: https://www.emergentmind.com/topics/ciebench