PandasBench: Pandas API Benchmark
- The paper introduces PandasBench, a benchmark with 102 Kaggle notebooks and 3,721 cells for realistic evaluation of pandas API optimizations.
- PandasBench is defined by its cleaning, curation, and non-uniform scaling methodology to capture heterogeneous, string-heavy, and dynamic EDA workloads.
- The benchmark serves as the evaluation backbone for optimizers like RuleFlow, measuring speedups, memory usage, and failure modes in notebook-level performance.
Searching arXiv for papers on PandasBench and closely related work. {"query":"PandasBench benchmark pandas API RuleFlow arXiv", "max_results": 10} I found relevant papers on PandasBench and its use in evaluating Pandas optimization frameworks, including the benchmark paper itself (Broihier et al., 3 Jun 2025) and RuleFlow’s evaluation on PandasBench (Singh et al., 6 Feb 2026). PandasBench is a large, executable benchmark designed specifically to evaluate techniques that accelerate or transform programs written against the pandas API. In the benchmark-paper sense, it comprises 102 Kaggle notebooks and 3,721 cells, cleaned and adapted for repeatable execution, and augmented with non-uniform input scaling so that end-to-end notebook behavior can be measured under realistic exploratory data analysis (EDA) conditions rather than only on synthetic kernels (Broihier et al., 3 Jun 2025). In later optimizer research, the same benchmark serves as a held-out test suite for notebook-level Pandas optimization, most notably in RuleFlow, which uses PandasBench to evaluate reusable rewrite rules on real-world multi-cell workloads (Singh et al., 6 Feb 2026).
1. Definition and benchmark criteria
PandasBench was introduced to fill a gap left by benchmarks from relational databases, numerical computing, and small pandas micro-evaluations. The benchmark paper argues that those alternatives do not capture the distinctive properties of pandas workloads, including heterogeneous data, string-heavy processing, schema inference at runtime, host-language interleaving, bursty EDA, and the interaction between lazy or eager semantics and I/O behavior (Broihier et al., 3 Jun 2025). The benchmark is therefore intended to evaluate techniques on code that resembles notebook practice rather than on isolated dataframe kernels.
The benchmark is organized around four explicit requirements. These requirements define what the authors regard as necessary for a practical pandas API benchmark.
| Requirement | Meaning |
|---|---|
| RWC | Real-world coverage |
| REL | Relevant code |
| APPR | Appropriate for benchmarking |
| DSC | Effective data scaling |
RWC requires executable, randomly sampled real notebooks at sufficiently large scale. REL requires notebooks to be cleaned so that pandas-relevant computations are preserved while irrelevant code is removed. APPR requires adaptation for repeatability, environment repair, and forced evaluation of intermediate results for lazy systems. DSC requires non-uniform per-notebook scaling rather than a single uniform scale factor (Broihier et al., 3 Jun 2025).
The benchmark’s construction reflects these requirements. Notebooks were drawn from Kaggle, restricted to Jupyter + Python, required to have at least 10 cells, and required heavy pandas usage, defined as at least 20% of static calls. The authors also excluded the 20 Dias notebooks to avoid bias, and then randomly sampled 102 notebooks for the suite (Broihier et al., 3 Jun 2025). This design makes PandasBench a benchmark of notebook workflows, not merely of isolated dataframe operators.
2. Notebook composition and workload characteristics
PandasBench is explicitly notebook-centric. Each workload is a multi-cell EDA workflow that includes ad hoc manipulations, mixed Pandas/NumPy/native Python code, and data-dependent control flow (Singh et al., 6 Feb 2026). The benchmark paper further emphasizes that real notebooks interleave dataframe processing with I/O, inspections, and other host-language activity, which is one reason benchmarks such as TPC-H cannot simply be repurposed for the pandas API (Broihier et al., 3 Jun 2025).
The benchmark’s input and API behavior are intentionally heterogeneous. Dynamic logging of top-level pandas calls shows that the notebook subset invokes 32.9% of pandas., Series., and DataFrame.* members at least once (Broihier et al., 3 Jun 2025). The input columns concentrate on four dtypes, with bool rare and str common. Eleven files, or about 10%, trigger inference challenges such as mixed-type warnings, encoding issues, or bad-line skipping. One notebook transposes data to 4,446,966 columns before calling np.corrcoef, illustrating the benchmark’s inclusion of transformation patterns that are atypical for relational benchmarks but occur in notebook practice (Broihier et al., 3 Jun 2025).
The covered patterns span data cleaning and column management, selection and indexing, grouping and aggregation, plotting calls embedded in analysis code, scalar and array operations, and dtype-sensitive behavior. The RuleFlow paper identifies representative operations in the benchmark and its induced rules, including DataFrame.rename, DataFrame.drop versus pop, bracket-based slicing and subsetting, .loc, .iloc, groupby(...).agg(...), plot(kind=...), columns.get_indexer, isnull().sum(), boolean checks, and selection chaining (Singh et al., 6 Feb 2026). It also notes that the observed optimizations often eliminate unnecessary copies, replace slower idioms with vectorized Pandas-native operations, and fuse chained transformations.
This composition is central to the benchmark’s difficulty. Several systems cannot execute the full suite end-to-end because of unsupported API coverage, runtime errors, or mismatches between bulk-parallel dataframe systems and representative EDA idioms. Compiler-style rewriters face a different difficulty: they must preserve semantics under common notebook conditions such as dtype changes, NaN handling, and aliasing or reference effects (Singh et al., 6 Feb 2026).
3. Construction, cleaning, and scaling methodology
PandasBench was not assembled by merely collecting notebooks. The benchmark paper describes a curation pipeline that fixes environment drift, updates deprecated APIs, repairs file paths, and resolves column-name or column-size mismatches. It also removes irrelevant code, such as plotting or machine-learning code when that code is outside the target pandas computation, excises functions and control flow that depend on removed libraries, eliminates IPython magics, and preserves transformations when possible by substituting original columns (Broihier et al., 3 Jun 2025).
For benchmarking, notebooks are adapted so that repeated execution is invariant. Cache or write-once behavior is removed, and evaluator functions are injected to force materialization of intermediate results. This is particularly important for Dask, Modin, and Koalas, whose lazy or deferred execution could otherwise produce misleading timings by postponing work until a later cell (Broihier et al., 3 Jun 2025).
A distinctive feature of PandasBench is non-uniform input scaling. Rather than applying a single TPC-H-style scale factor to every notebook, the benchmark chooses a per-notebook scale so that the baseline pandas runtime satisfies
Here, seconds, and the benchmark also reports results for the default unscaled setting (Broihier et al., 3 Jun 2025). The paper states that uniform scaling is too coarse because notebooks exhibit large runtime variance, diverse file formats, and notebook-specific scale-sensitive constraints such as deduped indices and hardcoded row fixes. The reported variance is 1946.256 seconds, which motivates the notebook-specific scaling procedure (Broihier et al., 3 Jun 2025).
The execution environment used in the benchmark paper is a dual-socket Intel Xeon Gold 6348 server with 1 TB RAM and 6 TB NVMe local storage, running Ubuntu 22.04.5 LTS and Python 3.10.16, with pandas 1.5.1 as the baseline. The evaluated alternatives are Modin 0.17.0 with the Ray backend, Dask 2024.4.1 with the threaded scheduler, Koalas 3.5.1 with Spark and Java 17 where needed, and Dias 0.1.2. All alternatives are configured to 4 CPU cores to match single-machine notebook contexts (Broihier et al., 3 Jun 2025).
4. Measurement protocol, metrics, and empirical findings
PandasBench evaluates notebooks end-to-end. Timing is recorded at notebook granularity, intermediate results are forced to materialize for lazy systems, and memory is measured after each cell as Proportional Set Size for the main Python process plus worker processes. Spill to disk is tracked for Modin via the Ray CLI and for Koalas via Spark REST metrics after the run (Broihier et al., 3 Jun 2025).
The benchmark paper defines speedup for notebook as
percent speedup as
and failure rate as
Coverage is also reported for API members and at notebook or cell granularity (Broihier et al., 3 Jun 2025).
The benchmark’s baseline evaluation shows that real-world coverage is a primary bottleneck. The paper reports that Modin runs 72/102 notebooks, Dask 4 in the abstract and 3 at the default target runtime in the main text, Koalas 10, and Dias 97 (Broihier et al., 3 Jun 2025). It also reports that the most notebooks that got a speedup were 8/102 for Modin and 0 for both Koalas and Dask, whereas Dias showed speedups in up to 55 notebooks, approximately 54%, though with correctness issues (Broihier et al., 3 Jun 2025).
| System | Successful notebooks | Speedup summary |
|---|---|---|
| Modin | 72/102 | 8/102 speedups at 20-second target runtime; max 2.10× |
| Dask | 3–4/102 | 0 speedups across target runtimes |
| Koalas | 10/102 | 0 speedups across target runtimes |
| Dias | 97/102 | Speedups in up to 55 notebooks |
The paper’s qualitative analysis identifies several failure categories: Incompatible API, Invalid Output, Bad Type Inference, and Scaling Error. Example failure modes include missing methods, missing arguments such as unsupported inplace, unsupported uses such as transpose on string columns, incorrect type inference in read_csv, and failures that appear only at larger scale. Dias is singled out for incorrect rewrites that cross semantic boundaries; the paper gives an example where Series.str.split(' ') is rewritten into elementwise Python string splitting, which fails if the column contains floats (Broihier et al., 3 Jun 2025).
The memory results are mixed rather than uniformly unfavorable to alternatives. Dask and Koalas use less memory than pandas on average as scale increases, whereas Modin’s peak memory can increase sharply, with worst per-notebook peaks up to about 451.93× relative to pandas on the compared notebooks. Dias remains near pandas, with geometric mean relative memory about 1.00× and max about 1.08× (Broihier et al., 3 Jun 2025). This suggests that PandasBench is not only a runtime benchmark; it is also a stress test for compatibility, semantic fidelity, and resource behavior.
5. PandasBench as RuleFlow’s evaluation backbone
RuleFlow uses PandasBench as its held-out evaluation suite. In that work, rule discovery is performed offline on a separate corpus of 199 Kaggle notebooks, while the 102 PandasBench notebooks form the unseen test suite on which rules are applied automatically and evaluated end-to-end (Singh et al., 6 Feb 2026). This separation is central to RuleFlow’s claim that reusable notebook optimizations can be discovered once and deployed repeatedly without invoking an LLM during evaluation.
RuleFlow is organized as a three-stage pipeline: Discovery (SnippetGen), Bridge (RuleGen), and Deployment (CodeGen). Discovery proposes optimized program variants, which are tested by EquivCheck on randomized dataframes and by OptCheck under an absolute improvement threshold of and a relative improvement threshold of . Bridge generalizes successful origin-to-optimized pairs into rewrite rules in Dias’s DSL, including variable and constant generalization, AST type resolution, rule construction, and runtime precondition synthesis. Deployment then applies validated rules by deterministic static matching, inserts runtime preconditions, and falls back to the original code when those conditions fail (Singh et al., 6 Feb 2026).
On PandasBench, RuleFlow reports 101/102 notebooks executed, compared with 97 for Dias, 72 for Modin, 3 for Dask, and 10 for Koalas. Its reported notebook-level speedups over these baselines are mean 1.54×, median 1.13×, max 4.3×, min 0.80× versus Dias; mean 112.79×, median 32.78×, max 1914.89×, min 0.48× versus Modin; mean 12.32×, median 6.45×, max 28.85×, min 1.68× versus Dask; and mean 140.15×, median 100.36×, max 377.09×, min 7.25× versus Koalas (Singh et al., 6 Feb 2026). The paper summarizes these results as “up to 4.3× over Dias” and “up to 1914.9× over Modin.”
The reuse statistics help characterize how PandasBench functions as an optimization benchmark rather than merely as a performance suite. RuleFlow reports that 88 of 102 notebooks have at least one rule application, that rules collectively apply to 87.13% of test notebooks, and that the notebooks use 18.68% of the proposed optimizations when reapplied. Of 120 generated rules, 88 were retained after qualitative filtering, 24 hit at least one notebook, and 17 were selected for deployment. The discovery yield is reported as about 3.79%, corresponding to 157 accepted origin–optimized pairs out of 3,981 candidates (Singh et al., 6 Feb 2026).
The RuleFlow paper also documents both strengths and failure modes on PandasBench. It attributes gains largely to rules that avoid copies or use faster accessors and indexing APIs, with examples such as rename(..., inplace=True) to avoid copies and replacing slower single-column deletion with pop. It also notes that runtime precondition checks can introduce overhead when preconditions fail, producing slowdowns such as 0.60× for one rule. Rules that were found to be semantically fragile, especially rules assuming no NaNs or relying on in-place mutation, were excluded from the reported evaluation (Singh et al., 6 Feb 2026).
6. Scope, ambiguity of the name, and prospective development
In current arXiv usage, the name PandasBench is not fully unambiguous. Besides the pandas-API optimization benchmark, the paper “Drawing Pandas” introduces a separate benchmark formally named PandasPlotBench and states that it is also referred to as PandasBench for brevity. That benchmark contains 175 human-curated plotting tasks for evaluating LLMs that generate plotting code for tabular data across Matplotlib, Seaborn, and Plotly (Galimzyanov et al., 2024). The two benchmarks address different problems: one evaluates execution, compatibility, and optimization of pandas notebooks; the other evaluates natural-language-to-plotting-code generation.
Within the pandas-API literature, PandasBench is positioned as the first large benchmark for its target problem. The benchmark paper states that there was previously no benchmark for the pandas API and argues that existing benchmarks from other domains cannot be repurposed for it (Broihier et al., 3 Jun 2025). Earlier single-machine studies of dataframe libraries evaluated real datasets and TPC-H-style workloads, but not a dedicated benchmark of executable pandas notebooks with notebook-specific scaling and cleaning methodology (Mozzillo et al., 2023).
The benchmark paper also identifies several limitations. Kaggle notebooks provide real code and data, but they reflect competition and EDA styles rather than all enterprise or scientific pipelines. Fixing, cleaning, and scaling were performed with documented rules but were not fully automated. The scope is single-machine rather than distributed. Variance and caching effects are mitigated by evaluator functions, but warmup runs and multiple repetitions are not detailed (Broihier et al., 3 Jun 2025). These limitations are methodological rather than incidental: they reflect the difficulty of constructing a benchmark that is both realistic and repeatable.
The published future directions are correspondingly practical. The authors propose adding more notebooks across domains, expanding API subsets such as time series and I/O formats, improving constraint-aware per-notebook scaling, integrating stronger correctness tooling such as golden-output comparisons and differential testing, and collaborating with pandas accelerators on recurrent failure modes such as inplace support, dtype handling, and categorical operations (Broihier et al., 3 Jun 2025). A plausible implication is that PandasBench is evolving less as a static scorecard than as infrastructure for studying the interaction among coverage, correctness, and end-to-end notebook performance.