CIEBench: Multi-Domain Evaluation Benchmark

• CIEBench is a multi-domain benchmark label applied to distinct artifacts in reasoning-based image editing, interval arithmetic, and control intervention awareness.
• In image editing, it assesses semantic correctness through implicit reasoning queries, ground-truth masks, and novel metrics like IDCS.
• For interval arithmetic, it evaluates library correctness, interval tightness, performance, and consistency using exact arithmetic reference solutions.

CIEBench is an overloaded benchmark name in the arXiv literature. In one usage, it denotes a benchmark for complex, reasoning-based image editing introduced alongside CIELR, where each sample couples an input image with an implicit reasoning query and a ground-truth edit mask (Wang et al., 31 Oct 2025). In another, it denotes a cross-platform benchmark for interval computation libraries, coupled with reference solutions computed with exact arithmetic to evaluate correctness, interval tightness, performance, and consistency across architectures, operating systems, and compilers (Tang et al., 2021). A further source of ambiguity is CIAware-Bench, which is also referred to as CIEBench in places, although its title and main text use CIAware-Bench (Schaeffer et al., 9 Jun 2026).

1. Terminological scope

The name “CIEBench” does not identify a single benchmark family. In the materials considered here, it is attached to at least two distinct artifacts and also appears as an alternate label in a third.

Name in use	Domain	Paper
CIEBench	Complex, reasoning-based image editing	(Wang et al., 31 Oct 2025)
CIEBench	Cross-platform benchmark for interval computation libraries	(Tang et al., 2021)
CIAware-Bench, also referred to as CIEBench in places	Control intervention awareness across frontier LLMs	(Schaeffer et al., 9 Jun 2026)

This naming overlap matters because the underlying evaluation targets are unrelated. One benchmark tests whether an image editor can infer an implicit target before editing; another tests whether interval arithmetic implementations are correct, tight, fast, and portable; the third tests whether an acting model can detect that a control protocol altered its trajectory. A plausible implication is that citations by arXiv identifier are necessary whenever the bare name “CIEBench” appears in technical discussion.

2. CIEBench in reasoning-based image editing

In "Understanding the Implicit User Intention via Reasoning with LLM for Image Editing" (Wang et al., 31 Oct 2025), CIEBench is introduced as a benchmark for reasoning-based image editing rather than simple instruction following. The motivating distinction is between explicit edits such as “replace the apple with an orange” and implicit edits such as “replace the food containing the most vitamin C with an orange.” In the latter case, a model must identify the correct object through reasoning and then edit it while preserving the rest of the image.

The benchmark contains 86 image-query-mask triplets. Each sample includes an input image $x$, an implicit reasoning query $q$, and a binary ground-truth mask $m_{gt}$ for the region to be edited. The dataset is built from ReasonSeg images and masks: the authors “carefully selected 86 high-quality images from ReasonSeg” and manually transformed the original segmentation instructions into implicit, reasoning-intensive editing queries. The paper presents CIEBench as a test benchmark rather than reporting a train/validation/test split.

CIEBench includes the three reasoning-editing types described in the paper: source object identification, target object identification, and multi-step edits. These task types are intended to expose failures that conventional image editing benchmarks can miss, especially failures in multi-step inference, implicit target identification, and semantic correctness under reasoning. The benchmark is therefore positioned as complementary to datasets such as MagicBrush, which contains 1053 samples but mainly explicit instructions, and the SmartEdit Reasoning Scenario Set, which contains 60 samples with implicit instructions but does not eliminate the broader scarcity of public resources for this setting (Wang et al., 31 Oct 2025).

3. Evaluation methodology and the CIELR framework

CIEBench was introduced together with CIELR (“Complex Image Editing via LLM Reasoning”) (Wang et al., 31 Oct 2025). The framework decouples reasoning from editing by introducing a structured semantic representation of the image that includes segmentation masks, object labels, and relative depth. This representation is built in a zero-shot manner using SAM2 for segmentation masks, OWLv2 for semantic labels, and DepthAnything for depth estimation. An LLM then reasons over this structure, identifies the target region, and converts the implicit instruction into an explicit editing instruction and mask. When the initial representation is insufficient, CIELR performs iterative refinement through a chain of structured semantic representation updates.

The paper argues that conventional metrics such as PSNR, SSIM, LPIPS, and CLIP similarity are inadequate for reasoning-based editing because they do not reliably measure whether the edit satisfies the reasoning requirement. A visually plausible edit can therefore be semantically wrong. To address this, the authors propose Image Difference Check Score (IDCS), an LLM-based metric defined in two stages: $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$ where $\mathrm{x}$ is the original image, $\mathrm{x}_\text{edit}$ is the edited image, $\mathrm{q}$ is the original implicit query, $\mathcal{F}_\text{LLM}^\text{desc}$ describes the differences between the original and edited image, and $\mathcal{F}_\text{LLM}^\text{eval}$ judges whether those differences satisfy the query. The final score is $\mathrm{S}_\text{RE} \in \{1,2,3,4,5\}$.

The design of IDCS makes the evaluation target explicit: it asks what changed in the image and whether those changes match the implicit instruction. This shifts measurement from low-level similarity to semantic correctness, which the paper treats as the central difficulty in reasoning-based editing.

4. Empirical results on the image-editing CIEBench

On CIEBench, the compared methods are InstructPix2Pix, MagicBrush, InstructDiffusion, CIELR (I.P.), and CIELR (I.A.) (Wang et al., 31 Oct 2025). The reported metrics are PSNR, SSIM, LPIPS, and IDCS. The table in the paper gives the following values: InstructPix2Pix reports $q$0 PSNR, $q$1 SSIM, $q$2 LPIPS, and $q$3 IDCS; MagicBrush reports $q$4, $q$5, $q$6, and $q$7; InstructDiffusion reports $q$8, $q$9, $m_{gt}$0, and $m_{gt}$1; CIELR (I.P.) reports $m_{gt}$2, $m_{gt}$3, $m_{gt}$4, and $m_{gt}$5; and CIELR (I.A.) reports 36.206, 0.960, 0.035, and 2.366. The paper specifically highlights the comparison between CIELR (I.A.) and InstructPix2Pix: PSNR 36.206 versus 9.794, SSIM 0.960 versus 0.337, and IDCS 2.366 versus 1.200.

The ablation study isolates two components: SSR and Chain. On CIEBench, SSR only yields $m_{gt}$6 PSNR, $m_{gt}$7 SSIM, $m_{gt}$8 LPIPS, $m_{gt}$9 IDCS, and $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$0 seconds; Chain only yields $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$1, $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$2, $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$3, $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$4, and $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$5 seconds; SSR + Chain yields 36.206, 0.960, 0.035, 2.366, and 6.504 seconds. The factual interpretation given in the paper is that either component helps, but both together are best, while the cost is a longer runtime of about 6.5 seconds per image.

The qualitative figure uses blue boxes for target regions, red boxes for incorrect or suboptimal edits, and green boxes for successful edits. The authors also note that CLIP score does not reflect actual task success in some examples. This supports their claim that CIEBench is specifically sensitive to reasoning failures rather than merely to visual quality degradation.

The benchmark’s limitations are also explicit. It has only 86 samples; its implicit reasoning queries were manually created from ReasonSeg instructions; IDCS depends on LLM judgment; and the full method incurs nontrivial runtime. The conclusion mentions extending the framework to video reasoning editing and suggests interactive creative workflows as future directions (Wang et al., 31 Oct 2025).

5. CIEBench in interval computation libraries

In "A Cross-Platform Benchmark for Interval Computation Libraries" (Tang et al., 2021), CIEBench designates the first benchmark for interval computations with reference solutions computed using exact arithmetic. Its stated purpose is to compare interval arithmetic libraries on four properties: correctness, interval size / tightness, performance, and consistency / portability. The motivation is that interval arithmetic is widely used to certify floating-point computations, but practical implementations are not standardized and are sensitive to compiler, operating-system, and hardware behavior.

The benchmark compares four open-source C/C++ interval libraries: filib, filib++, Boost, and BIAS. The paper further treats the operational modes of some libraries as distinct interval types. For filib++, these are native, pred, and multiplicative. For BIAS, the paper considers ROUND DOWN, ROUND UP, and ROUND NEAR separately because the documentation does not clearly specify how these affect interval operations.

CIEBench is built around an expression suite with 28 expressions authored by the paper’s authors and 104 expressions taken from FPBench. The 28 hand-crafted expressions include the four basic arithmetic operations, four transcendental functions $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$6, $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$7, $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$8, and $$\mathrm{D}_\text{diff} = \mathcal{F}_\text{LLM}^\text{desc}(\mathrm{x}, \mathrm{x}_\text{edit}), \qquad \mathrm{S}_\text{RE} = \mathcal{F}_\text{LLM}^\text{eval}(\mathrm{D}_\text{diff}, \mathrm{q})$$9, ten composite expressions with only arithmetic, and ten composite expressions mixing arithmetic and transcendental functions. For each expression, the benchmark generates one million valid inputs. Inputs and outputs are converted to a rational representation using GMP, and the exact query is evaluated using Mathematica. Interval width is also computed exactly by rational subtraction, so the validation machinery does not inherit floating-point error from the evaluated libraries.

Performance is measured by generating 1,000 inputs per expression and evaluating each expression 10,000 times per input. The benchmark was run on four platforms: Windows on an Intel Core i7-8700K with MSVC 14.27.29110; macOS Intel on an Intel Core i9 with Apple clang 12.0.0; macOS ARM on Apple M1 with Apple clang 12.0.0; and Linux on an AMD EPYC 7452 32-Core Processor with GCC 9.2.1 (Tang et al., 2021).

6. Findings, recommendations, and naming overlap with CIAware-Bench

The interval-computation CIEBench yields several concrete findings (Tang et al., 2021). All libraries were correct on the basic arithmetic expressions. Boost failed on transcendental functions, specifically for $\mathrm{x}$0 and trigonometric functions. For composite expressions using only arithmetic, all libraries were correct; when transcendental functions were mixed into composite expressions, BIAS produced incorrect intervals on at least one expression. On the FPBench expressions, filib and filib++ pred/multiplicative were correct on all tests, while filib++ native failed on four FPBench expressions. The paper’s bottom line on correctness is that the only methods correct for all tests were filib, filib++ pred, and filib++ multiplicative.

On interval size, libraries using system rounding modes—Boost, filib++ native, and BIAS—tended to produce smaller intervals for pure arithmetic, while filib++ multiplicative produced the largest intervals on arithmetic-only expressions. On performance, the general ordering for the 28 main expressions is reported as Boost slowest overall, then BIAS, then filib, while filib++ native is highly optimized and much faster than the other two traditional rounding-mode libraries. For more complex workloads, the overall ordering is generally: filib++ multiplicative, filib++ pred, filib++ native, filib, BIAS, Boost. On portability and consistency, filib is described as the only library that was correct, consistent, portable, and reasonably fast, and the paper explicitly recommends filib as the best option among those tested.

The same section of the literature also contains a separate nomenclature issue. In "CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs" (Schaeffer et al., 9 Jun 2026), the paper states that the benchmark is also referred to as CIEBench in places, but its title and main text use CIAware-Bench. That benchmark measures control intervention (CI) awareness: whether an acting model can recognize, from a trajectory alone, whether a step was produced by itself or by another model under a control intervention. It spans essay writing, BigCodeBench, Bash Arena, and SHADE-Arena; frames evaluation as a balanced binary classification problem with chance level 0.5; and reports low to moderate CI awareness across eleven frontier models from four providers. This is not the same benchmark as either the reasoning-based image-editing CIEBench or the interval-computation CIEBench.

Taken together, these uses show that “CIEBench” is a domain-dependent label rather than a stable benchmark brand. In current arXiv usage, the most direct interpretations are either the 86-sample reasoning-based image editing benchmark paired with IDCS (Wang et al., 31 Oct 2025) or the exact-arithmetic-backed benchmark for interval computation libraries (Tang et al., 2021), while CIAware-Bench should be cited under its canonical name even when alternate mentions of CIEBench appear (Schaeffer et al., 9 Jun 2026).