CleanUpBench: Multi-Domain Evaluation

• CleanUpBench is a collection of benchmark frameworks evaluating cleanup tasks in code refactoring, robotics, outdoor environments, and language model decontamination.
• It uses controlled protocols with explicit metrics such as test pass rate, task completion ratio, and contamination rates to assess performance.
• Its modular design supports reproducibility and integration across diverse applications, from multi-robot cleaning tasks to decontamination pipelines.

CleanUpBench denotes several benchmark frameworks that address cleanup-related evaluation in distinct research areas: code refactoring, embodied indoor cleaning, outdoor public-space cleaning, and benchmark decontamination for LLMs. In the cited literature, the name is attached to systems that measure how agents detect and remove structural “code smells” in real-world code while preserving functionality, evaluate mobile robots that must both sweep loose debris and grasp larger objects, formalize stakeholder-driven outdoor cleaning scenarios, or produce a single “clean” evaluation set that can be fairly used to benchmark any collection of $K$ LLMs (Lin et al., 4 Jun 2026, Li et al., 7 Aug 2025, Memmesheimer et al., 2024, Liu et al., 20 May 2026).

1. Name, scope, and disambiguation

The shared label masks substantial heterogeneity. One line of work treats CleanUpBench as a refactoring benchmark inspired by SmellBench; another defines it as an embodied benchmark built on NVIDIA Isaac Sim; a third uses it as a framework for outdoor cleaning robots in public spaces; and a fourth uses it as a decontamination pipeline built around Joint Envelope Conformal Selection (JECS). A related multi-robot cleaning-allocation benchmark is explicitly proposed as a module that can be “directly dropped into CleanUpBench under a new ‘CleaningAllocation’ benchmark module” (Wang et al., 2023).

Variant	Domain	Primary object
CleanUpBench inspired by SmellBench	Code agents	Detect and remove structural “code smells”
CleanUpBench: Embodied Sweeping and Grasping Benchmark	Embodied AI	Sweep loose debris and grasp larger objects
CleanUpBench for public spaces	Outdoor robotics	Public-space debris collection under real constraints
CleanUpBench built around JECS	LLM evaluation	Single “clean” benchmark shared across audited models

A common misconception is to treat CleanUpBench as a single benchmark with a stable task definition. The cited material instead describes several independent artifacts that share an emphasis on cleanup, maintenance, or evaluation hygiene. This suggests that the term functions more as a recurring benchmark name than as a single canonical suite.

2. CleanUpBench as a code-refactoring benchmark

In the code-agent setting, CleanUpBench is described as a refactoring benchmark inspired by Fowler’s classic refactoring taxonomy and by “SmellBench: Towards Fine-Grained Evaluation of Code Agents on Refactoring Tasks” (Lin et al., 4 Jun 2026). It starts from seven mature, popular Python repositories—examples given are click, numpy, pandas, scikit-learn, jinja, seaborn, and xarray—with comprehensive test suites. From these repositories, it extracts “clean” code snippets, including classes, modules, and functions, that already pass their existing unit tests.

Dataset construction proceeds through controlled smell injection. For each snippet, specific insertion points are chosen via a two-stage “Candidate Discovery” process: coarse filtering based on file size and importance, followed by code-agent localization to pick methods or classes that satisfy a given smell specification without breaking tests. A powerful code agent is then prompted to introduce a single type of smell in a controlled way, producing a “smelly” version that still compiles and passes its tests. Expert engineers or skilled annotators subsequently produce a human-written refactoring that removes the smell while preserving behavior, and the benchmark retains both the smelly input and the clean, refactored output as paired examples.

The smell taxonomy contains seven types. Feature Envy is defined as a method that refers excessively to data or methods of another class. God Class is a class that accumulates too many responsibilities, with low cohesion and high coupling. Data Clumps are groups of parameters or fields that always appear together, suggesting a missing abstraction. Shotgun Surgery is a single conceptual change that must be applied in many scattered places. Dead Code Elimination targets unreachable or unused code elements. Interface Segregation concerns overly broad interfaces forcing clients to depend on unused methods. Deep Inlining collapses nested calls into a single method. These smells are stratified into Easy, Medium, and Hard by varying the number of involved files, the level of code indirection, and the camouflage of the smell. Two instruction settings are provided: Guided, in which “There is an X smell in file A,” and Targeted, in which “Method m in class C in file A exhibits smell X.”

Evaluation measures four complementary aspects of refactoring. Functional Correctness is the Test Pass Rate (TPR), under which a refactoring is valid only if the entire existing test suite still passes. Localization Ability (LA) asks whether the agent correctly identifies the class or method that must be refactored. Refactoring Quality is scored with an LLM-as-Judge rubric covering Code Quality (CQ), Structural Soundness (SS), Cross-File Coordination (CC), and Smell Elimination (SE). The overall elimination metric is also expressed as

$$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$

The benchmark contains 294 cases spanning 7 popular smell types, 3 difficulty levels, 2 instruction settings, and 7 real-world repositories. Experiments with two open-source agents, OpenHands and Qwen Code, each paired with six LLM backends—Qwen-Coder-30B, Qwen-Coder-480B, DeepSeek-V3.2, Gemini-2.5-Flash, GPT-5-Mini, and Claude-Sonnet-4.5—show that the best performer, Qwen Code + Claude Sonnet 4.5, achieves $\mathrm{TPR} \approx 0.94$, $\mathrm{LA} \approx 0.89$, and Smell Elimination $\approx 0.5034$ or 50.34%. Even the strongest combination removes only half of the injected smells on average. Localized edits such as Deep Inlining and Dead Code reach up to 70–80% elimination, whereas cross-file refactorings such as Shotgun Surgery and God Class fall below 30%. Guided versus targeted prompts yield a 25–30% boost in LA, which identifies self-localization as a major bottleneck.

The reported limitations are also methodological. Some repositories contain latent smells, such as God Classes, which can introduce noise. Using a single agent for smell injection can lead to stereotyped patterns that agents learn to recognize. LLM-as-Judge quality assessment may inherit the judge model’s preferences or blind spots. Future extensions include multilingual and multi-language code bases such as Java, JavaScript, and Go; additional smells such as Lazy Class and Message Chains; composite scenarios with multiple coexisting smells; static-analysis integration for localization and cross-file dependency resolution; and multi-round, agent-in-the-loop refinements. A central lesson is that functional correctness alone cannot distinguish refactoring skill.

3. CleanUpBench as an embodied indoor cleaning benchmark

“CleanUpBench: Embodied Sweeping and Grasping Benchmark” defines a reproducible and extensible benchmark for evaluating embodied agents in realistic indoor cleaning scenarios (Li et al., 7 Aug 2025). Its stated objective is to provide a task-centric testbed for indoor cleaning robots that must both sweep loose debris and grasp larger objects, thereby bridging the gap between low-level skill testing and full household simulations. Intended applications include autonomous home-service robots, warehouse or office cleaning with mixed debris, and multi-robot coordinated cleaning in commercial environments.

The benchmark is built on NVIDIA Isaac Sim and simulates a mobile service robot equipped with a sweeping mechanism and a six-degree-of-freedom robotic arm. It includes 20 manually designed layouts across five categories—Sparse Exploration, High-Density Sweeping, Narrow Corridors, Dynamic Interference, and Multi-Zone Coordination—together with one procedurally generated layout for generalization. The procedural generator uses Voronoi-based room geometries, Poisson disk sampling for obstacles, A* to ensure connectivity, and configurable target distributions that may be random, clustered, or linear. Sweepable debris consists of paper scraps, dust, and small particles with mass 0.01–0.05 kg; graspable items include bottles, toys, and remotes with mass 0.1–0.8 kg. Static obstacles include furniture, walls, and fixtures, and task zones include collection areas and restricted zones.

The Isaac Sim configuration specifies PhysX 5.0 rigid-body dynamics with Coulomb friction, a time step of $1/60\,\mathrm{s}$, and contact tolerance $0.01\,\mathrm{m}$. The sensor suite includes RGB-D at $1280\times720$ with $0.3\text{–}10\,\mathrm{m}$ range, semantic segmentation masks, $360^\circ$ LiDAR with 1440 points and $$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$0 resolution, and proprioception. Fidelity features include continuous collision detection and RTX-accelerated rendering. The robot platform has a mobile base weighing 25.5 kg with dimensions $$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$1, maximum linear velocity $$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$2, and maximum angular velocity $$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$3; a front brush of diameter $$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$4, 200 RPM, and width $$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$5; and a 6-DOF manipulator with reach $$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$6, payload 3.0 kg, and gripper span 0–0.08 m.

Task definitions are formalized over an MDP $$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$7. The benchmark supports Sweep-only, in which all sweepable objects $$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$8 must be cleaned; Grasp-only, in which all graspable objects $$\text{EliminationScore} = \frac{\#\text{smells removed}}{\#\text{smells injected}} \times 100\%.$$9 must be picked and stored; and Dual-mode, in which sweeping and grasping are interleaved. The robot is spawned at a random valid location, objects are randomly placed per scene specification, and an episode ends when both $\mathrm{TPR} \approx 0.94$0 and $\mathrm{TPR} \approx 0.94$1 are cleared or at timeout $\mathrm{TPR} \approx 0.94$2, given as, for example, 300 s, or on failure through excessive collisions. The control interface exposes a hybrid action space with $\mathrm{TPR} \approx 0.94$3, continuous navigation actions $\mathrm{TPR} \approx 0.94$4, and manipulator joint-velocity commands $\mathrm{TPR} \approx 0.94$5, with observations and actions streamed at 60 Hz.

Its evaluation suite is multi-level and includes Coverage Ratio (CR), Sweep Redundancy (SR), Task Completion Ratio (TCR), Motion Efficiency (ME), Collision count, Computation Time (CT), Finish Time (FT), and motion smoothness through average velocity, acceleration, and jerk. Baseline agents include Manhattan, Chebyshev, Vertical/Horizontal, and m-explore for sweep-only coverage; REMANI for grasp-only planning; and PRIMAL2 and IR2 for dual-mode or coordinated settings. On the excerpted performance summary averaged over 5 runs on all 20 scenes, Manhattan achieves TCR 0.12, CR 0.34, FT 250.2 s, and CT 0.08 s in Sweep mode; Chebyshev achieves TCR 0.15 and CR 0.42; REMANI achieves TCR 0.30 and CR 0.57 in Grasp mode; PRIMAL2 achieves TCR 0.60, ME 0.21, SR 0.01, CR 0.69, FT 287.7 s, and CT 0.04 s in Dual mode; and IR2 in Sweep mode reaches TCR 0.17, ME 0.01, SR 1.00, and CR 0.37. PRIMAL2 leads on TCR 0.60, while sweep-only heuristics have near-zero grasp scores and grasp-only methods lack coverage.

Reproducibility is treated as part of the benchmark definition. Environments are stored as JSON or YAML, the asset library is organized under assets/, agents under agents/, evaluation code under evaluation/, and reproduction uses fixed seeds, a Docker container with Isaac Sim 4.5 and Python dependencies, and the command python scripts/run_benchmark.py --scene All --agent PRIMAL2.

4. CleanUpBench for outdoor public-space cleaning

A separate CleanUpBench framework is proposed for outdoor cleaning robots in public spaces and is derived directly from seven in-depth stakeholder interviews with municipal cleaning departments and facility managers (Memmesheimer et al., 2024). The interviews identify the tasks and constraints that these stakeholders need automated. Collecting small, potentially hazardous debris of differing consistencies—broken glass, cigarette butts, syringes, and paper wrappers—is mentioned by 100% of stakeholders. Reaching hard-to-access locations under benches or cars, in seams, and near water edges is also mentioned by 100%. Vacuuming and sweeping sidewalks, squares, and paved surfaces are mentioned at 100% and 86%, respectively. Additional tasks include leaf removal, weed raking, seam cleaning, bin emptying, snow clearing, graffiti or chewing-gum removal, and mopping.

The same interviews define boundary conditions. Social acceptance, including interactions with passersby and risk of vandalism, is reported by 100% of stakeholders. Heterogeneous substrates such as asphalt, cobbles, and grass of varying heights; unpredictable weather and lighting including rain, wind, reflections, shadows, and night operation; and environmental protection, including avoiding harm to wildlife or vegetation, are each cited by 86%. Other constraints include unmapped or unknown areas, stairs, inclines, noise restrictions, limited on-robot storage, unknown objects, and removing debris without damaging surfaces.

The physical layout is a real park segment with a central paved pathway 3 m wide and a 1 m wayside of grass or gravel on each side, giving a total benchmark width of 5 m and a typical length of 10–15 m. Fixed obstacles include curbs, cycle-barriers, park benches, and parked car mock-ups. Pollution consists of twenty or more pieces of randomly distributed litter drawn from a standard trash kit that includes plastic cups, paper wrappers, glass shards, cigarette butts, and beverage cartons. Variation in size from 1–10 cm, material, moisture level, color contrast, and partial occlusion is used to stress both perception and manipulation. Difficulty levels are explicitly defined as Basic, which collects debris from the unobstructed central pathway; Intermediate, which adds the adjacent waysides up to 1 m from curbs; Advanced, which includes items under benches, behind barriers, or inside curb pockets for bonus points; and an Optional extension involving mopping or scrubbing an oil-based spill patch added mid-run.

The performance metrics and evaluation criteria are deliberately modest in formalization. The paper encourages reporting $\mathrm{TPR} \approx 0.94$6, the number of pieces of trash successfully retrieved; $\mathrm{TPR} \approx 0.94$7, the total mass in grams; $\mathrm{TPR} \approx 0.94$8, the time in seconds from start signal until completion or until a fixed trial duration elapses; and $\mathrm{TPR} \approx 0.94$9, bonus points for hard-to-reach items. Teams may also report Collision Count, Coverage Rate, Energy Efficiency $\mathrm{LA} \approx 0.89$0, and Cleanliness Score $\mathrm{LA} \approx 0.89$1. A notable caveat is explicit: the original paper does not prescribe exact formulas, and the sample scoring function shown in the summary is “not from the paper.” That caveat matters because it marks the framework as a scenario proposal rather than a finalized scoring standard.

Protocols include a technical acceptance inspection with robot mass $\mathrm{LA} \approx 0.89$2, footprint $\mathrm{LA} \approx 0.89$3, an emergency-stop button accessible from front and sides, and a basic obstacle-avoidance shakedown run. During trial execution, organizers set up the park segment, randomly place debris according to a predefined seed, release the robot on a go signal, and allow a fixed maximum runtime such as 5 min. Data recording includes an overview video, on-board logs of timestamps, positions, detections, battery voltage, and emergency-stop triggers, and a manual tally of collected items and bonus items. Reporting requires a run summary and a robot technical sheet covering chassis, powertrain, sensors, manipulation mechanism, onboard compute, and software stack. Each difficulty level is repeated three times with different random seeds, with mean $\mathrm{LA} \approx 0.89$4 standard deviation reported.

5. CleanUpBench as a decontamination pipeline

In benchmark hygiene, CleanUpBench is defined as a decontamination pipeline built around JECS, or Joint Envelope Conformal Selection, to produce a single “clean” evaluation set for multiple LLMs (Liu et al., 20 May 2026). The problem formulation assumes $\mathrm{LA} \approx 0.89$5 audited models and a pool of $\mathrm{LA} \approx 0.89$6 candidate examples. An example is contaminated if it was used in any model’s training, and the objective is to produce a single selected subset $\mathrm{LA} \approx 0.89$7 such that the global contamination rate is controlled at level $\mathrm{LA} \approx 0.89$8 while maximizing the fraction of truly pure examples.

JECS operates in three stages. First, it computes per-model conformal $\mathrm{LA} \approx 0.89$9-values from a shared calibration set of examples known to be members of all $\approx 0.5034$0 models. Second, it aggregates them through the max-$\approx 0.5034$1 statistic,

$\approx 0.5034$2

which is valid for the joint null that the item is contaminated by at least one model. Third, because $\approx 0.5034$3 is super-uniform under the null, it reconstructs a conservative envelope of the null CDF from right-tail observations above a data-driven threshold $\approx 0.5034$4, rescales the aggregated $\approx 0.5034$5-values by the fitted envelope, estimates the null proportion with Storey’s estimator, and applies the Storey-BH step-up rule. The theoretical guarantee stated in the benchmark description is Theorem 1: asymptotic GCR control, namely

$\approx 0.5034$6

The motivation is comparative fairness. Applying single-model conformal decontamination separately can produce model-specific benchmarks, undermining cross-model comparison. JECS is designed to produce one shared benchmark with provable global contamination rate control and higher power than the max-$\approx 0.5034$7 baseline. In the reported experiments, synthetic settings with $\approx 0.5034$8 show that naive union and intersection of per-model selections violate $\approx 0.5034$9, whereas JECS controls GCR at approximately 0.038. On WikiMIA and ArXivTection with $1/60\,\mathrm{s}$0 fine-tuned NeoX-20B, LLaMA-7B, and Pythia-6.9B models, JECS maintains realized GCP curves at or below the diagonal across $1/60\,\mathrm{s}$1, and on ArXivTection with Pythia-6.9B at $1/60\,\mathrm{s}$2, JMCS power is 0.094 versus JECS power 0.447.

A complementary line of work is provided by AntiLeak-Bench, which addresses contamination by constructing samples with explicitly new knowledge absent from models’ training sets and by using a fully automated workflow to build and update the benchmark without human labor (Wu et al., 2024). The relevant lessons for a CleanUpBench system are stated directly: strict cutoff-based novelty filtering using $1/60\,\mathrm{s}$3; automated document-history verification through revision histories; overlap or similarity-based contamination scoring; a fully scriptable end-to-end pipeline; multi-hop and distractor augmentation; and continuous monitoring of contamination rates over time. Whereas JECS addresses joint decontamination through conformal statistics, AntiLeak-Bench addresses contamination-free sample construction by enforcing post-cutoff novelty by construction. The two approaches target the same reliability problem from different directions.

6. Related extensions and recurrent design patterns

The most explicit extension path appears in multi-robot autonomous cleaning. “Towards Practical Multi-Robot Hybrid Tasks Allocation for Autonomous Cleaning” provides a dataset of 100 instances made from floor plans, each with 2D manually labeled images and a 3D model, and its summary states that all code, data, and baseline solvers can be directly dropped into CleanUpBench under a new “CleaningAllocation” benchmark module (Wang et al., 2023). The formulation uses a set of cleaning tasks $1/60\,\mathrm{s}$4, heterogeneous robots $1/60\,\mathrm{s}$5, zone locations, task types such as vacuum and mop, precedence constraints such as vacuum before mop in the same zone, and uncertainty in cleaning durations. It studies a deterministic MILP and a robust counterpart under box, convex-hull, and ellipsoidal uncertainty sets.

The released dataset is grouped into five subsets by size, ranging from 3–6 zones with average area 38 m$1/60\,\mathrm{s}$6 to 18–35 zones with average area 148 m$1/60\,\mathrm{s}$7. For each instance, the release includes a floor-plan PNG with semantic labels, a JSON file listing zone area $1/60\,\mathrm{s}$8, centroid $1/60\,\mathrm{s}$9, task types and precedence, ROS/Gazebo worlds and URDF, precomputed travel-time matrices, and robot profiles. Solvers include Gurobi 9.x, Simulated Annealing, Genetic Algorithm, Particle Swarm Optimization, and a deep reinforcement learning solver adapted from Nazari et al. for VRP. Reported deterministic results show Gurobi obtaining the smallest average makespan on the smallest subsets within a 10 min limit, while Simulated Annealing outperforms GA and PSO in makespan. Under uncertainty, the ellipsoidal set gives the smallest overhead for all deviations and instance sizes, whereas box sets are described as overly conservative.

The proposed CleanUpBench integration path is concrete: add the 100 instances under CleanUpBench/data/cleaning_allocation/{set1,…,set5}, use a unified JSON schema for tasks, robots, and travel times, provide launch files under CleanUpBench/launch/, wrap each solver in a Python interface, and implement a standard evaluation hook that records $0.01\,\mathrm{m}$0, runtime, and CSV or plot outputs. This module-oriented proposal aligns with the embodied benchmark’s modular control API and scene configuration structure, and with the decontamination pipeline’s scriptable multi-stage architecture.

Taken together, these materials suggest that the recurring function of CleanUpBench is not a fixed task family but a benchmark philosophy. In the code-agent variant, the central issue is long-term maintainability rather than functional correctness alone. In the indoor and outdoor robotics variants, the emphasis is on realistic cleaning tasks under clutter, physical constraints, and reporting discipline. In the decontamination variant, the objective is fair evaluation under training-data uncertainty. Across these uses, the shared design pattern is controlled construction, explicit metrics, reproducibility hooks, and a deliberate attempt to evaluate properties that simpler benchmarks tend to miss.