I-Bench: Benchmarking Interactive Data Exploration

• I-Bench is a benchmarking framework for interactive data exploration that evaluates database systems on speed, accuracy, and progressive result quality.
• The framework simulates realistic user workflows such as zooming, filtering, and drill-down, emphasizing time-to-insight and iterative query refinement.
• It supports diverse engine paradigms—including blocking, approximate query processing, and progressive sampling—using metrics like MRE, latency violations, and bias.

I-Bench

I-Bench, formally "IDEBench," is a benchmarking framework for the evaluation of database systems in the context of interactive data exploration (IDE) workflows. Unlike traditional decision-support benchmarks such as TPC-H and TPC-DS, which focus on static, pre-defined SQL queries over stationary datasets, I-Bench explicitly targets the unique demands of ad-hoc, incrementally built, and latency-sensitive queries arising in visual analytics and exploration interfaces. The framework addresses both the time-to-insight and accuracy trade-offs for a spectrum of database engine architectures, including traditional blocking DBMSs, approximate query processing (AQP) systems, and progressive engines (Eichmann et al., 2018).

1. Motivation and Benchmarking Paradigm

IDEBench was motivated by the inadequacy of legacy benchmarks to characterize performance in IDE settings. In IDE, users typically interact through visual interfaces, iteratively constructing and refining queries via actions such as zooming, filtering, drill-down, and cross-filtering. These workflows require sub-second response, enable acceptance of approximate answers or progressive refinements, and involve interactions whose effects propagate across multiple coordinated views. Classic benchmarks (TPC-H/DS) do not account for:

• Time-to-insight and interactivity: Speed constraints for incremental, user-driven tasks.
• Approximation/progressiveness: Quality of answers that evolve over time or are expressed with confidence intervals.
• Workflow realism: The branching, stateful nature of visual exploration, where user actions are sequential and conditional.

Thus, I-Bench simulates common IDE interaction patterns and introduces metrics capturing both runtime and the evolving quality of intermediate or approximate results (Eichmann et al., 2018).

2. Data Model and Workload Construction

IDEBench employs a data generator seeded on real-world datasets, with the canonical instance based on U.S. domestic flights (Bureau of Transportation Statistics). Key properties:

• Schema and scale: The default is a denormalized fact table, with optional vertical partitioning into star or snowflake schemas. Scale factors include S = 100M, M = 500M, L = 1B rows.
• Synthetic data generation: The generator computes empirical covariances Σ from the seed data, performs Cholesky decomposition (Σ = AᵀA), and samples from $X \sim \mathcal{N}(0,1)$, then maps distributions back to match empirical marginals and correlations via CDF transformations.
• Workflows: Generated to simulate realistic exploration, drawing on a Markov model over possible interactions (create visualization, filter, link, discard), and incorporating user-like "think time" (default 1–10 s) between actions for more realistic pacing and enabling speculation.

A typical workload reflects four core IDE workflow types: independent browsing, sequential linking, 1→N linking, and N→1 linking. This architecture supports data and workload customizability to maintain both realism and reproducibility.

3. Evaluation Metrics

I-Bench is characterized by a multi-dimensional evaluation scheme that rigorously quantifies both speed and result quality:

• Interactive Latency (Time Requirement, TR): The proportion of queries violating a user-specified latency budget (e.g., 0.5 s, 1 s, 3 s) is computed.
• Result Completeness and Accuracy: For queries returning binned or grouped aggregates, completeness and error are assessed by:
  • $$\text{MissingBins} = \frac{|\{i \mid F_i \text{ missing}\}|}{n}$$ (fraction of missing bins)
  • $$\mathrm{MRE} = \frac{1}{n} \sum_{i=1}^{n} \left| \frac{F_i - A_i}{A_i} \right|$$ (mean relative error)
  • $$d_{\cos} = 1 - \frac{\sum_i F_i A_i}{\sqrt{\sum_i F_i^2}\sqrt{\sum_i A_i^2}}$$ (cosine distance, shape fidelity)
  • Margin and Confidence: The average reported margin of error (from engines reporting confidence intervals) and the out-of-margin frequency
  • Bias: $\frac{\sum_i F_i}{\sum_i A_i}$

Metrics support analysis of the speed-accuracy trade-off, such as plotting MRE versus response latency, and reveal the ability of engines to deliver progressively more accurate results as more computation or data is consumed (Eichmann et al., 2018).

4. Benchmark Architecture and Methodology

The architecture of I-Bench consists of:

1. Data Generator: Produces scalable synthetic datasets matching real-world statistics.
2. Workload Generator: Encodes exploration workflows as sequences of user actions, parameterized by think-time and workflow structure.
3. Benchmark Driver: Orchestrates user workflows, manages interleaving of queries and user events, enforces TR bounds, and records response times.
4. System Adapters: Translate abstraction-layer "visualization specifications" into SQL or system-specific APIs; capture both final and intermediate results.

Default experimental setups utilize servers with high memory and multiple CPU cores to isolate engine performance; the benchmark supports plug-and-play evaluation of commercial, research, and reference systems.

5. Experimental Findings and Comparative Analysis

Empirical evaluation demonstrates distinct system-level trade-offs:

System	TR-Violation @0.5s	@1s	@3s	Median MRE @0.5s	@1s	@3s
MonetDB	~100%	~90%	~50%	— (exact only)	—	—
approXimateDB	~66% (flat)	~66%	~66%	>100%	—	—
System X	>50%	~5%	0%	~120% → ~20%	—	—
IDEA	~1%	0%	0%	~5%	—	—

• MonetDB (blocking exact): Poor interactive latency; nearly all queries violate TR at 0.5 s.
• approXimateDB (wandering joins): High TR-violation rate except for simple aggregates; high error for unsupported types.
• System X (offline AQP): Requires fixed sample size; poor quality at low latency, improves with relaxed TR.
• IDEA (progressive stratified sampling): Satisfies sub-second TR in nearly all cases, achieving low MRE and tight margins.

Schema normalization marginally benefits I/O and latency. Exploiting think-time and speculative computation in engines like IDEA significantly reduces MissingBins and improves user-experience fidelity.

6. Implications and Context within Data Systems Research

IDEBench establishes new standards for benchmarking in the context of modern IDE tools, where rapid iteration, result evolution, interactivity, and varying accuracy requirements are essential. By incorporating both synthetic and real data distributions, supporting fine-grained workload realism, and quantifying the multi-faceted trade-offs of latency, accuracy, and completeness, I-Bench enables systematic cross-comparison across fundamentally different database systems and engine paradigms—blocking, AQP, and progressive.

A plausible implication is that, as IDE becomes central to scientific and data-centric workflows, benchmarks like IDEBench will become prerequisite for evaluating suitability-for-purpose for both emerging and legacy data engines. The framework highlights that blocking engines unable to meet interactive latencies may be fundamentally unsuited for real-time exploration, even if they guarantee exactness.

7. Extensions and Prospective Directions

Future extensions of I-Bench may include broader schema and workflow types, richer support for diverse visualization frontends, more complex multi-user and collaborative scenarios, and benchmarking of distributed or cloud-native progressive engines. As engines develop to further blend interactive sampling, indexing, incremental view maintenance, and speculative execution, benchmarks in the I-Bench tradition will be critical for rigorous, comparable evaluation in interactive analytics (Eichmann et al., 2018).