---
title: 'PandasBench: Pandas API Benchmark'
url: https://www.emergentmind.com/topics/pandasbench
type: topic
---

# PandasBench: Pandas API Benchmark

Searching arXiv for papers on PandasBench and closely related work.
{"query":"PandasBench benchmark pandas API RuleFlow arXiv", "max_results": 10}
I found relevant recent papers on PandasBench and its use in evaluating Pandas optimization frameworks, including the benchmark paper itself [2506.02345] and RuleFlow’s evaluation on PandasBench [2602.09051].
PandasBench is a large, executable benchmark designed specifically to evaluate techniques that accelerate or transform programs written against the pandas API. In the benchmark-paper sense, it comprises 102 Kaggle notebooks and 3,721 cells, cleaned and adapted for repeatable execution, and augmented with non-uniform input scaling so that end-to-end notebook behavior can be measured under realistic exploratory data analysis (EDA) conditions rather than only on synthetic kernels [2506.02345]. In later optimizer research, the same benchmark serves as a held-out test suite for notebook-level Pandas optimization, most notably in RuleFlow, which uses PandasBench to evaluate reusable rewrite rules on real-world multi-cell workloads [2602.09051].

## 1. Definition and benchmark criteria

PandasBench was introduced to fill a gap left by benchmarks from relational databases, numerical computing, and small pandas micro-evaluations. The benchmark paper argues that those alternatives do not capture the distinctive properties of pandas workloads, including heterogeneous data, string-heavy processing, schema inference at runtime, host-language interleaving, bursty EDA, and the interaction between lazy or eager semantics and I/O behavior [2506.02345]. The benchmark is therefore intended to evaluate techniques on code that resembles notebook practice rather than on isolated dataframe kernels.

The benchmark is organized around four explicit requirements. These requirements define what the authors regard as necessary for a practical pandas API benchmark.

| Requirement | Meaning |
|---|---|
| **RWC** | Real-world coverage |
| **REL** | Relevant code |
| **APPR** | Appropriate for benchmarking |
| **DSC** | Effective data scaling |

RWC requires executable, randomly sampled real notebooks at sufficiently large scale. REL requires notebooks to be cleaned so that pandas-relevant computations are preserved while irrelevant code is removed. APPR requires adaptation for repeatability, environment repair, and forced evaluation of intermediate results for lazy systems. DSC requires non-uniform per-notebook scaling rather than a single uniform scale factor [2506.02345].

The benchmark’s construction reflects these requirements. Notebooks were drawn from Kaggle, restricted to Jupyter + Python, required to have at least 10 cells, and required heavy pandas usage, defined as at least 20% of static calls. The authors also excluded the 20 Dias notebooks to avoid bias, and then randomly sampled 102 notebooks for the suite [2506.02345]. This design makes PandasBench a benchmark of notebook workflows, not merely of isolated dataframe operators.

## 2. Notebook composition and workload characteristics

PandasBench is explicitly notebook-centric. Each workload is a multi-cell EDA workflow that includes ad hoc manipulations, mixed Pandas/NumPy/native Python code, and data-dependent control flow [2602.09051]. The benchmark paper further emphasizes that real notebooks interleave dataframe processing with I/O, inspections, and other host-language activity, which is one reason benchmarks such as TPC-H cannot simply be repurposed for the pandas API [2506.02345].

The benchmark’s input and API behavior are intentionally heterogeneous. Dynamic logging of top-level pandas calls shows that the notebook subset invokes 32.9% of pandas.*, Series.*, and DataFrame.* members at least once [2506.02345]. The input columns concentrate on four dtypes, with bool rare and str common. Eleven files, or about 10%, trigger inference challenges such as mixed-type warnings, encoding issues, or bad-line skipping. One notebook transposes data to 4,446,966 columns before calling `np.corrcoef`, illustrating the benchmark’s inclusion of transformation patterns that are atypical for relational benchmarks but occur in notebook practice [2506.02345].

The covered patterns span data cleaning and column management, selection and indexing, grouping and aggregation, plotting calls embedded in analysis code, scalar and array operations, and dtype-sensitive behavior. The RuleFlow paper identifies representative operations in the benchmark and its induced rules, including `DataFrame.rename`, `DataFrame.drop` versus `pop`, bracket-based slicing and subsetting, `.loc`, `.iloc`, `groupby(...).agg(...)`, `plot(kind=...)`, `columns.get_indexer`, `isnull().sum()`, boolean checks, and selection chaining [2602.09051]. It also notes that the observed optimizations often eliminate unnecessary copies, replace slower idioms with vectorized Pandas-native operations, and fuse chained transformations.

This composition is central to the benchmark’s difficulty. Several systems cannot execute the full suite end-to-end because of unsupported API coverage, runtime errors, or mismatches between bulk-parallel dataframe systems and representative EDA idioms. Compiler-style rewriters face a different difficulty: they must preserve semantics under common notebook conditions such as dtype changes, NaN handling, and aliasing or reference effects [2602.09051].

## 3. Construction, cleaning, and scaling methodology

PandasBench was not assembled by merely collecting notebooks. The benchmark paper describes a curation pipeline that fixes environment drift, updates deprecated APIs, repairs file paths, and resolves column-name or column-size mismatches. It also removes irrelevant code, such as plotting or machine-learning code when that code is outside the target pandas computation, excises functions and control flow that depend on removed libraries, eliminates IPython magics, and preserves transformations when possible by substituting original columns [2506.02345].

For benchmarking, notebooks are adapted so that repeated execution is invariant. Cache or write-once behavior is removed, and evaluator functions are injected to force materialization of intermediate results. This is particularly important for Dask, Modin, and Koalas, whose lazy or deferred execution could otherwise produce misleading timings by postponing work until a later cell [2506.02345].

A distinctive feature of PandasBench is non-uniform input scaling. Rather than applying a single TPC-H-style scale factor to every notebook, the benchmark chooses a per-notebook scale $s_i$ so that the baseline pandas runtime satisfies

$$
T_{\mathrm{pandas},i}(s_i) \in [X, 1.33X].
$$

Here, $X \in \{5,10,20\}$ seconds, and the benchmark also reports results for the default unscaled setting [2506.02345]. The paper states that uniform scaling is too coarse because notebooks exhibit large runtime variance, diverse file formats, and notebook-specific scale-sensitive constraints such as deduped indices and hardcoded row fixes. The reported variance is 1946.256 seconds, which motivates the notebook-specific scaling procedure [2506.02345].

The execution environment used in the benchmark paper is a dual-socket Intel Xeon Gold 6348 server with 1 TB RAM and 6 TB NVMe local storage, running Ubuntu 22.04.5 LTS and Python 3.10.16, with pandas 1.5.1 as the baseline. The evaluated alternatives are Modin 0.17.0 with the Ray backend, Dask 2024.4.1 with the threaded scheduler, Koalas 3.5.1 with Spark and Java 17 where needed, and Dias 0.1.2. All alternatives are configured to 4 CPU cores to match single-machine notebook contexts [2506.02345].

## 4. Measurement protocol, metrics, and empirical findings

PandasBench evaluates notebooks end-to-end. Timing is recorded at notebook granularity, intermediate results are forced to materialize for lazy systems, and memory is measured after each cell as Proportional Set Size for the main Python process plus worker processes. Spill to disk is tracked for Modin via the Ray CLI and for Koalas via Spark REST metrics after the run [2506.02345].

The benchmark paper defines speedup for notebook $i$ as

$$
S_i = \frac{T_{\mathrm{base},i}}{T_{\mathrm{sys},i}},
$$

percent speedup as

$$
\Delta\% = \frac{T_{\mathrm{base},i} - T_{\mathrm{sys},i}}{T_{\mathrm{base},i}} \times 100\%,
$$

and failure rate as

$$
F = 1 - \frac{\mathrm{runs\_successful}}{\mathrm{total\_runs}}.
$$

Coverage is also reported for API members and at notebook or cell granularity [2506.02345].

The benchmark’s baseline evaluation shows that real-world coverage is a primary bottleneck. The paper reports that Modin runs 72/102 notebooks, Dask 4 in the abstract and 3 at the default target runtime in the main text, Koalas 10, and Dias 97 [2506.02345]. It also reports that the most notebooks that got a speedup were 8/102 for Modin and 0 for both Koalas and Dask, whereas Dias showed speedups in up to 55 notebooks, approximately 54%, though with correctness issues [2506.02345].

| System | Successful notebooks | Speedup summary |
|---|---:|---|
| **Modin** | 72/102 | 8/102 speedups at 20-second target runtime; max 2.10× |
| **Dask** | 3–4/102 | 0 speedups across target runtimes |
| **Koalas** | 10/102 | 0 speedups across target runtimes |
| **Dias** | 97/102 | Speedups in up to 55 notebooks |

The paper’s qualitative analysis identifies several failure categories: Incompatible API, Invalid Output, Bad Type Inference, and Scaling Error. Example failure modes include missing methods, missing arguments such as unsupported `inplace`, unsupported uses such as transpose on string columns, incorrect type inference in `read_csv`, and failures that appear only at larger scale. Dias is singled out for incorrect rewrites that cross semantic boundaries; the paper gives an example where `Series.str.split(' ')` is rewritten into elementwise Python string splitting, which fails if the column contains floats [2506.02345].

The memory results are mixed rather than uniformly unfavorable to alternatives. Dask and Koalas use less memory than pandas on average as scale increases, whereas Modin’s peak memory can increase sharply, with worst per-notebook peaks up to about 451.93× relative to pandas on the compared notebooks. Dias remains near pandas, with geometric mean relative memory about 1.00× and max about 1.08× [2506.02345]. This suggests that PandasBench is not only a runtime benchmark; it is also a stress test for compatibility, semantic fidelity, and resource behavior.

## 5. PandasBench as RuleFlow’s evaluation backbone

RuleFlow uses PandasBench as its held-out evaluation suite. In that work, rule discovery is performed offline on a separate corpus of 199 Kaggle notebooks, while the 102 PandasBench notebooks form the unseen test suite on which rules are applied automatically and evaluated end-to-end [2602.09051]. This separation is central to RuleFlow’s claim that reusable notebook optimizations can be discovered once and deployed repeatedly without invoking an LLM during evaluation.

RuleFlow is organized as a three-stage pipeline: Discovery (`SnippetGen`), Bridge (`RuleGen`), and Deployment (`CodeGen`). Discovery proposes optimized program variants, which are tested by `EquivCheck` on randomized dataframes and by `OptCheck` under an absolute improvement threshold of $\tau = 150\ \mathrm{ms}$ and a relative improvement threshold of $\epsilon = 2$. Bridge generalizes successful origin-to-optimized pairs into rewrite rules in Dias’s DSL, including variable and constant generalization, AST type resolution, rule construction, and runtime precondition synthesis. Deployment then applies validated rules by deterministic static matching, inserts runtime preconditions, and falls back to the original code when those conditions fail [2602.09051].

On PandasBench, RuleFlow reports 101/102 notebooks executed, compared with 97 for Dias, 72 for Modin, 3 for Dask, and 10 for Koalas. Its reported notebook-level speedups over these baselines are mean 1.54×, median 1.13×, max 4.3×, min 0.80× versus Dias; mean 112.79×, median 32.78×, max 1914.89×, min 0.48× versus Modin; mean 12.32×, median 6.45×, max 28.85×, min 1.68× versus Dask; and mean 140.15×, median 100.36×, max 377.09×, min 7.25× versus Koalas [2602.09051]. The paper summarizes these results as “up to 4.3× over Dias” and “up to 1914.9× over Modin.”

The reuse statistics help characterize how PandasBench functions as an optimization benchmark rather than merely as a performance suite. RuleFlow reports that 88 of 102 notebooks have at least one rule application, that rules collectively apply to 87.13% of test notebooks, and that the notebooks use 18.68% of the proposed optimizations when reapplied. Of 120 generated rules, 88 were retained after qualitative filtering, 24 hit at least one notebook, and 17 were selected for deployment. The discovery yield is reported as about 3.79%, corresponding to 157 accepted origin–optimized pairs out of 3,981 candidates [2602.09051].

The RuleFlow paper also documents both strengths and failure modes on PandasBench. It attributes gains largely to rules that avoid copies or use faster accessors and indexing APIs, with examples such as `rename(..., inplace=True)` to avoid copies and replacing slower single-column deletion with `pop`. It also notes that runtime precondition checks can introduce overhead when preconditions fail, producing slowdowns such as 0.60× for one rule. Rules that were found to be semantically fragile, especially rules assuming no NaNs or relying on in-place mutation, were excluded from the reported evaluation [2602.09051].

## 6. Scope, ambiguity of the name, and prospective development

In current arXiv usage, the name *PandasBench* is not fully unambiguous. Besides the pandas-API optimization benchmark, the paper “Drawing Pandas” introduces a separate benchmark formally named *PandasPlotBench* and states that it is also referred to as *PandasBench* for brevity. That benchmark contains 175 human-curated plotting tasks for evaluating LLMs that generate plotting code for tabular data across Matplotlib, Seaborn, and Plotly [2412.02764]. The two benchmarks address different problems: one evaluates execution, compatibility, and optimization of pandas notebooks; the other evaluates natural-language-to-plotting-code generation.

Within the pandas-API literature, PandasBench is positioned as the first large benchmark for its target problem. The benchmark paper states that there was previously no benchmark for the pandas API and argues that existing benchmarks from other domains cannot be repurposed for it [2506.02345]. Earlier single-machine studies of dataframe libraries evaluated real datasets and TPC-H-style workloads, but not a dedicated benchmark of executable pandas notebooks with notebook-specific scaling and cleaning methodology [2312.11122].

The benchmark paper also identifies several limitations. Kaggle notebooks provide real code and data, but they reflect competition and EDA styles rather than all enterprise or scientific pipelines. Fixing, cleaning, and scaling were performed with documented rules but were not fully automated. The scope is single-machine rather than distributed. Variance and caching effects are mitigated by evaluator functions, but warmup runs and multiple repetitions are not detailed [2506.02345]. These limitations are methodological rather than incidental: they reflect the difficulty of constructing a benchmark that is both realistic and repeatable.

The published future directions are correspondingly practical. The authors propose adding more notebooks across domains, expanding API subsets such as time series and I/O formats, improving constraint-aware per-notebook scaling, integrating stronger correctness tooling such as golden-output comparisons and differential testing, and collaborating with pandas accelerators on recurrent failure modes such as `inplace` support, dtype handling, and categorical operations [2506.02345]. A plausible implication is that PandasBench is evolving less as a static scorecard than as infrastructure for studying the interaction among coverage, correctness, and end-to-end notebook performance.

Source: https://www.emergentmind.com/topics/pandasbench