PolyBench: A Unified Multi-Domain Benchmark Suite
- PolyBench is a collection of benchmark suites covering scientific loop optimization, GPU kernel tuning, polymer design, and polyphonic audio reasoning.
- It standardizes performance evaluation by providing diverse kernels and tasks for autotuning, compiler optimization, and AI-driven system assessments.
- PolyBench enables rigorous cross-domain comparisons, fostering reproducibility while exposing bottlenecks in LLM reasoning and system integration.
PolyBench refers to several distinct, high-impact benchmark suites and datasets sharing the same name but targeting fundamentally different research domains: scientific loop kernel optimization, LLM–driven material science (polymer design), and polyphonic audio reasoning. Each constitutes a reference standard for evaluation, algorithmic innovation, and empirical comparison within its respective community.
1. PolyBench in Scientific Kernel Optimization
PolyBench/C is the canonical benchmark suite for evaluating loop transformation systems, static polyhedral compilers, and auto-scheduling techniques. It consists of 30 loop-nest kernel programs, yielding 150 instances when evaluated over five input sizes (MINI, SMALL, MEDIUM, LARGE, XLARGE) (Merouani et al., 1 Nov 2025). The kernels cover representative scientific computing motifs:
- Linear algebra: matrix multiplications and reductions (2mm, 3mm, GEMM, syrk, trmm, cholesky, lu, ludcmp, mvt, nussinov, trisolv)
- Stencil computations: 2D/3D stencils (seidel2d, jacobi2d, heat-3d)
- Vector reductions/statistics: covariance, correlation, mvt
These kernels exhibit a wide range of nest depths (from 2 in vector–matrix kernels up to 5 in 3D stencils), data-access patterns (perfectly regular to highly strided or triangular), and complex dependence structures, often encapsulating multiple computations per nest. This diversity makes PolyBench/C a stringent “stress test” for evaluating correctness and performance of loop transformations, including interchange, tiling, fusion, distribution, unrolling, skewing, and parallelization.
A typical example is the evaluation of agentic LLM-based loop scheduling: ComPilot, leveraging Tiramisu as the polyhedral backend, achieves geometric mean speedups of 2.66× (single run) and 3.54× (best-of-5 runs) against baseline code, outperforming the state-of-the-art Pluto scheduler on most instances (Merouani et al., 1 Nov 2025). Similarly, PolyBench/C is used for end-to-end LLVM-based in-memory compilation workflows, for autotuning with Bayesian optimization, and as the principal testbed in low-level accelerator design studies (Vadivel et al., 2020, Wu et al., 2020).
| Domain | Variant | Focus | Representative Work |
|---|---|---|---|
| HPC Compilers | PolyBench/C | Loop transformation, tiling, etc. | ComPilot (Merouani et al., 1 Nov 2025), TDO-CIM (Vadivel et al., 2020), Clang/Polly Autotune (Wu et al., 2020) |
2. PolyBench in LLM-Guided GPU Kernel Optimization
GPU-focused LLM frameworks adopt PolyBench to span a variety of compute- and memory-bound motifs when benchmarking systems for kernel extraction, autonomous feedback-driven tuning, and portability (Chu et al., 15 Dec 2025). Here, PolyBench targets include 20 distinct GPU-oriented kernels ranging from dense GEMM-like operations to sparse stencils and statistical reductions. In these studies, each kernel is extracted, encapsulated as a Minimal Executable Program (MEP), and subjected to iterative optimization (block dimension tuning, memory layout transformations, shared-memory buffering), with feedback loops enforcing correctness and guiding variant generation.
Key findings include strong correlation between standalone MEP and integrated PolyBench harness performance, and average speedups of 5.05× (NVIDIA CUDA) and 7.77× (Haiguang DCU/HIP) compared to baseline (Chu et al., 15 Dec 2025). Kernel types respond differently: memory-bound operations (correlation, covariance) benefit most from shared-memory and layout transformations, while compute-bound GEMMs yield consistent 2–4× gains using tiling.
| Platform | # Kernels | Average Speedup | Optimization Scheme |
|---|---|---|---|
| NVIDIA | 13 | 5.05× | LLM+MEP, perf feedback, error repair |
| Haiguang | 13 | 7.77× | LLM+MEP, perf feedback, error repair |
3. PolyBench for Polymer Design (AI4Science)
A distinct PolyBench benchmark serves as a large-scale training and evaluation dataset for LLM–mediated polymer design (Mohanty et al., 22 Jan 2026). This PolyBench comprises 125,000+ tasks structured into six analytical–synthetic categories, built atop a 13M+ experimental/synthetic polymer knowledge base. Tasks progress from structural parsing to multi-constraint generative design, with data preprocessed to encode molecular descriptors and property annotations.
Knowledge-augmented chain-of-thought (CoT) reasoning is central; fine-tuning open-source LLMs such as Qwen-2.5-14B on PolyBench yields up to 80% absolute improvement on property reasoning and design tasks versus unaligned models, even surpassing some closed-source “frontier” LLMs on specialized metrics (ROUGE, SMILES similarity, and Tanimoto Validity). The dataset explicitly supports diagnostic decomposition and compositional generalization tests, identifying both atomic skill mastery and persistent compositionality gaps in current LLMs.
| Tasks | Data Sources (examples) | Model Impact | |
|---|---|---|---|
| PolyBench | 125K+ | PolymersML, ChEMBL, OMG, RDKit | SLMs > chem. LLMs, CoT > base |
4. PolyBench for Polyphonic Audio Reasoning
A separate PolyBench benchmark targets compositional reasoning in polyphonic audio for evaluating Large Audio LLMs (LALMs) (Chen et al., 5 Mar 2026). This PolyBench includes 728 clips sampled across major environmental and instrument-rich datasets (DataSED, DESED, MAESTRO-Real) with controlled overlap cardinalities (2–4 sources per clip) and supports five task categories: counting sound events, duration estimation, concurrency, overlapped classification, and temporal overlap localization. All tasks are posed as MCQA (multiple-choice question answering) with Chain-of-Thought prompting and exact-match/semantic scoring.
State-of-the-art LALMs achieve only 55–60% accuracy on counting and overlap detection, revealing fundamental bottlenecks in compositional scene parsing under masking and temporal overlap. Concurrency and classification tasks permit higher performance (up to 90.4% ACC for some models in monophonic settings), but performance degrades systematically as polyphony increases or distractors are introduced. PolyBench’s confusion matrix and error analyses show biases toward default “yes/no” answers based on prior-task statistics, underscoring the need for improved polyphonic event separation and temporal structure modeling.
| Task | Model ACC (Best) | Key Challenge |
|---|---|---|
| Counting | 57.5% | Masking, event fusion |
| Duration Estimation | 70.2% | Temporal integration |
| Concurrency Detection | 93.7%/85.2% | Default bias, perceptual |
| Classification | 77.9% | Overlap discrimination |
| Overlap Detection | 63.4% | Localization under overlap |
5. PolyBench as an Autotuning and Compiler Optimization Benchmark
PolyBench/C is widely used as the principal workload for evaluating autotuning frameworks and interactive compiler systems (Wu et al., 2020). For instance, Bayesian optimization autotuners operating over Clang/Polly loop transformation pragmas demonstrate consistent performance improvements over –O3 and default Polly-optimized baselines across complex kernels such as syr2k, 3mm, lu, heat-3d, covariance, and Floyd-Warshall. Key autotuning choices include tiling, unrolling, interchange, array packing, and vectorization, with parameter spaces reaching up to 170,368 configurations (e.g., 10-dimensional for 3mm).
Measured speedup factors for autotuned kernels over –O3 reach 1.29× for syr2k, 1.19× for 3mm, and 1.12× for covariance. Notably, default Polly heuristics sometimes degrade performance, as with Floyd-Warshall, but a hybrid of explicit pragma control and autotuning can outstrip hand-tuned settings. The impact of autotuning is most pronounced for compute-bound, cache-intensive kernels, with diminishing returns for memory-bound or low-arithmetic-intensity cases.
| Kernel | Speedup (Autotuned vs –O3) |
|---|---|
| syr2k | 1.29× |
| 3mm | 1.19× |
| lu | 1.12× |
| heat-3d | 1.08× |
| covariance | 1.12× |
6. Role in Emerging Architectures and End-to-End Toolflows
PolyBench remains central in end-to-end evaluations of novel architectural paradigms, such as in-memory computing with compiler support (Vadivel et al., 2020). TDO-CIM demonstrates transparent LLVM-based detection, optimization, and offload of PolyBench/C kernels to memristor-based accelerators. GEMM-like kernels (matrix–matrix multiply, 2mm, 3mm) on PolyBench yield accelerator-host speedups ≈50× and energy reductions ≈120×. In contrast, GEMV-like or low-intensity kernels underperform due to poor compute/write amortization. Architectural inferences (e.g., the necessity of crossbar reuse, fusion, and tiling to maximize endurance and efficiency) stem directly from PolyBench/C outcome analyses.
7. Synthesis: PolyBench Across Research Fronts
PolyBench, although multiply realized across scientific computing, AI4Science, and polyphonic audio, maintains a unified role in each domain: as a stress test, performance yardstick, and catalyst for research progress in optimization technology, learning systems, and compositional reasoning. Its open availability (for all major variants), well-defined kernel/task structure, and established community practices underpin rigorous comparison and reproducibility.
A plausible implication is that future expansions of PolyBench—particularly multimodal and cross-domain variants—will continue to set the standard for benchmarking advanced AI and compiler systems, while serving as a crucible for exposing bottlenecks in reasoning, optimization, and low-level system integration.