RubikBench: Industrial NL2SQL Benchmark

• RubikBench is a benchmark for industrial NL2SQL that simulates real-world enterprise challenges such as implicit intent resolution, private domain knowledge, and wide-table complexity.
• It employs a four-stage pipeline—synthetic seed generation, real-world query transfer, augmentation, and manual verification—to ensure robust annotation and realistic query complexity.
• The benchmark integrates user profiles, a maintained knowledge base, and advanced evaluation metrics to support lifelong learning and context-aware SQL synthesis in enterprise environments.

Searching arXiv for RubikBench and RubikSQL to ground the article in the relevant papers. arxiv_search(query="RubikBench OR RubikSQL", max_results=10) RubikBench is a benchmark for industrial natural-language-to-SQL (NL2SQL) introduced in "RubikSQL: Lifelong Learning Agentic Knowledge Base as an Industrial NL2SQL System" (Chen et al., 25 Aug 2025). It is designed to capture enterprise NL2SQL properties that are underrepresented in widely used academic benchmarks, notably implicit intent resolution, dependence on private domain knowledge and synonyms, wide-table schema complexity, and context sensitivity tied to user profile and time. The benchmark is deliberately centered on lifelong learning over a single, realistic enterprise database rather than cross-database generalization, and it is coupled to a maintained, agentic knowledge base (KB) that stores domain lexicon, predicates, indicators and metrics, synonyms, and taxonomies or dependencies (Chen et al., 25 Aug 2025).

1. Design objective and problem formulation

RubikBench was created to model industrial NL2SQL conditions that arise in enterprise finance workflows and that are weakly represented in conventional research datasets (Chen et al., 25 Aug 2025). The benchmark encodes four design pressures.

First, it targets implicit intent resolution. Enterprise users may submit short, informal questions in which key intent is omitted, such as whether a request for revenue should default to YoY rather than raw revenue, whether outliers should be omitted, or which currency should be assumed. RubikBench therefore includes ambiguous, high-level queries from executives alongside more explicit analyst queries, requiring systems to infer defaults and unstated preferences from context rather than from fully specified prompts (Chen et al., 25 Aug 2025).

Second, it targets private domain knowledge and synonyms. The benchmark includes domain-specific terminology, abbreviations, aliases, internal customer groups, self-defined metrics and formulas, temporal and business differences such as fiscal versus calendar year or currency conventions, and table-specific defaults. The paper emphasizes that rules underlying indicators such as “YoY” and “BCR” vary across tables and periods and must be learned from KB context rather than from short oracle hints (Chen et al., 25 Aug 2025).

Third, it targets wide-table schema complexity. In industrial databases, enum values may be transposed into columns for performance, and similar semantic requests may require heterogeneous SQL patterns across tables. RubikBench uses wide tables to stress column-level schema linking and predicate correctness under these conditions (Chen et al., 25 Aug 2025).

Fourth, it targets context sensitivity and lifelong learning. Identical natural-language questions may require different SQL depending on when the query is issued, which user is asking, and what organizational preferences are in force. RubikBench therefore attaches user profiles and temporal context to questions and is designed for in-database continual adaptation over time (Chen et al., 25 Aug 2025).

A central implication of this formulation is that RubikBench evaluates not only SQL synthesis but also the maintenance and use of a living enterprise KB. This suggests a benchmark regime in which retrieval, profiling, schema interpretation, and rule induction are first-class components of the NL2SQL problem rather than auxiliary preprocessing.

2. Dataset composition and enterprise schema

RubikBench is derived from real enterprise finance workloads using an anonymized schema for a hypothetical automobile manufacturer (Chen et al., 25 Aug 2025). It is organized around four table categories: “Income,” “Sales Ledger,” “Profit and Loss,” and “Budget and Forecast.” The organizational structure contains 8 departments; each department has its own “Income” and “Sales Ledger,” while all departments share “Profit and Loss” and “Budget and Forecast.” Each table has approximately 40,000 records and approximately 70 columns (Chen et al., 25 Aug 2025).

The currently described release is RubikBench0.9, which contains over 5,000 annotated queries and supports PostgreSQL and SQLite (Chen et al., 25 Aug 2025). The schema is finance-centric and wide-table oriented. Its columns reflect currency, period encoded as strings in YYYYMM format, budget versus actual amounts, prior-year YTD, and business-defined “caliber” settings such as A/B (Chen et al., 25 Aug 2025).

RubikBench also packages domain resources that are integral to interpretation. These include user profiles specifying default currency, caliber, region, and department; jargon glossaries; and internal rule documents describing how to compute metrics such as YoY and BCR, how to treat nulls, when to use DISTINCT, and how to define “important customers” (Chen et al., 25 Aug 2025). Queries include user profile and query time fields, which makes context sensitivity part of the benchmark input rather than an external assumption (Chen et al., 25 Aug 2025).

The benchmark’s focus on a single stable enterprise database distinguishes its intended use. Rather than distributing attention across many small schemas with sparse queries, RubikBench concentrates query density, domain regularity, and accumulated context within one production-like setting. This design is meant to support KB maintenance and continual adaptation, not only one-shot semantic parsing.

3. Query semantics and workload characteristics

RubikBench couples implicit-intent and context-aware queries to user roles and preferences, and it emphasizes industrial query traits such as multi-table and wide-table usage, complex aggregations, window functions, temporal filters over YYYYMM periods, units and implicit conversions, and long-tail corner cases (Chen et al., 25 Aug 2025). The paper does not report distribution percentages across SELECT, JOIN, GROUP BY, HAVING, ORDER BY, LIMIT, subqueries, CTEs, or window functions. It does, however, indicate heavy use of aggregations such as SUM, COUNT, AVG, CASE, and COALESCE; window functions such as RANK() OVER ...; temporal filters using month ranges and YTD or PY_YTD logic; domain metrics such as YoY growth and budget completion rate; and currency filters and conversions (Chen et al., 25 Aug 2025).

The benchmark also emphasizes paraphrases and implicit-intent questions, with role-specific ambiguity between executive phrasing and analyst phrasing (Chen et al., 25 Aug 2025). This means that lexical diversity is not treated as mere paraphrastic noise; it interacts with business defaults, domain rulebooks, and user-specific priors.

One example given in the paper asks, “Where does Nova rank for Inventory Impairment Loss BCR in Sweden and Spain this month?” The associated user profile specifies default currency EUR, default caliber B, a default region hierarchy, a default department vehicle_brand_group=Nova Dynamics, and query time 2023-12-21. The ground-truth PostgreSQL query computes a BCR-like ratio over periods 202311 to 202312, applies country, brand, report, and caliber filters, uses RANK() OVER (ORDER BY ...), and then restricts the output to EUR (Chen et al., 25 Aug 2025). The example is significant because it compresses several of the benchmark’s defining traits into a single instance: implicit temporal resolution of “this month,” role-conditioned defaults, domain-specific metric semantics, temporal filtering on YYYYMM strings, and a window function over an aggregate.

The benchmark’s workload therefore occupies a different operating point from traditional NL2SQL corpora. Query interpretation depends not only on schema and database contents but also on time, organizational memory, and enterprise-specific vocabulary. A plausible implication is that systems optimized only for cross-database compositional generalization may underperform when asked to recover implicit enterprise semantics from persistent KB context.

4. Construction pipeline, annotation, and lifelong-learning regime

RubikBench is built through a four-stage pipeline intended to balance realism and coverage (Chen et al., 25 Aug 2025). The first stage is synthetic seed query generation, in which the initial KB $K^0$ is populated by generating seed queries from the schema. The second stage is real-world query transfer, where queries are transferred from a private production NL2SQL workload spanning multinational enterprise finance departments, including analysts, administrative staff, and C-suite users; schema and ontology alignment are used to produce executable SQL in the RubikBench database. The third stage is query augmentation, which adds queries through rephrasing, composition, and decomposition, answered by powerful LLMs aided by the evolving KB and reflecting agentic context mining and CoT-enhanced SQL profiling. The fourth stage is manual verification, in which financial experts and experienced SQL analysts manually examine all questions and SQLs for correctness and diversity (Chen et al., 25 Aug 2025).

The pipeline leverages database profiling, structured information extraction (SIE), agentic context mining, and CoT-enhanced SQL profiling both to strengthen annotation robustness and to provide AI-friendly SQL commentary and rationale (Chen et al., 25 Aug 2025). Sensitive enterprise attributes are desensitized while retaining structural fidelity (Chen et al., 25 Aug 2025). Manual verification is explicit, but inter-annotator agreement statistics are not reported (Chen et al., 25 Aug 2025).

RubikBench’s evaluation setting is intentionally lifelong-learning oriented. The paper does not specify fixed train, development, and test splits for RubikBench0.9 (Chen et al., 25 Aug 2025). Instead, the benchmark is designed to support cold-start scenarios with minimal KB, as well as warmed KB scenarios in which context has been accumulated over prior queries. It also supports session-based or incremental evaluation consistent with a transductive lifelong-learning setting, where later queries may leverage context mined from prior unlabeled queries (Chen et al., 25 Aug 2025).

Access assumptions during inference are correspondingly broad. RubikBench provides schema, database content, user profiles, and domain documents describing internal rules and terminology. Systems are expected to use KB-based retrieval, including synonyms, predicates, indicators or metrics, and taxonomies or dependencies, together with toolkits such as schema or enum profilers and fuzzy search, in a manner consistent with the RubikSQL workflow (Chen et al., 25 Aug 2025). Explicit restrictions on inference-time resources are not formalized, and the paper advises researchers to document their access policies, including whether documentation or KB resources are allowed and whether updates during evaluation are permitted (Chen et al., 25 Aug 2025).

5. Metrics and evaluation methodology

RubikBench adopts standard NL2SQL reference metrics but adds a benchmark-specific result-comparison metric designed for enterprise BI usage (Chen et al., 25 Aug 2025). The reference metrics are execution accuracy and exact match:

$$EX = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[\text{exec}(\text{SQL}_i) = \text{ans}_i],$$

$$EM = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[\text{normalize}(\text{SQL}_i) = \text{normalize}(\text{GT}_i)].$$

The paper notes that exact match is included for completeness and is not its main focus (Chen et al., 25 Aug 2025).

The distinguishing evaluation proposal is the Bipartite $F_\beta$-score, denoted $\text{BF}_\beta$, which emphasizes recall when $\beta>1$ and is designed to handle practical discrepancies in result sets, including ordered versus unordered outputs, column-order differences, extra columns, and extra rows (Chen et al., 25 Aug 2025). Predicted and ground-truth results are represented as ordered lists of column value sets with column names removed,

$$P = \langle p_1, p_2, \dots, p_n \rangle,\qquad G = \langle g_1, g_2, \dots, g_m \rangle.$$

Row-wise weights are computed from precision and recall over set overlap, producing a fully connected weighted bipartite graph $\boldsymbol{w}^\beta(P,G)$ over rows (Chen et al., 25 Aug 2025).

For unordered ground truth, the benchmark uses maximum weighted bipartite matching via the Hungarian algorithm, with worst-case complexity $O(N^3)$ and typical near-linear behavior due to sparsity (Chen et al., 25 Aug 2025). For ordered ground truth containing ORDER BY, it uses a non-intersecting maximum weighted matching $WBM_{NI}$ computed by dynamic programming with complexity $$EX = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[\text{exec}(\text{SQL}_i) = \text{ans}_i],$$0, described as roughly linear with greedy pruning (Chen et al., 25 Aug 2025). The final metric over a set of queries $$EX = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[\text{exec}(\text{SQL}_i) = \text{ans}_i],$$1 is

$$EX = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[\text{exec}(\text{SQL}_i) = \text{ans}_i],$$2

The paper presents two motivations for this metric. It is intended to be robust to formatting differences that do not materially affect information content, and its tunable recall emphasis is aligned with BI users’ preference to retrieve all relevant rows (Chen et al., 25 Aug 2025). In effect, RubikBench treats practical answer fidelity as more important than strict SQL-string equivalence.

6. Position within NL2SQL benchmarking, current status, and limitations

The RubikSQL paper positions RubikBench against several existing NL2SQL benchmarks (Chen et al., 25 Aug 2025). Spider, BIRD, and WikiSQL are characterized as focusing on broad generalization across many heterogeneous databases, with emphasis on table discovery, multi-hop reasoning, and generalization power, but with sparse queries per database and limited enterprise context, implicit-intent coverage, and user-profile or KB grounding. BIRD Mini-Dev and LiveSQLBench move toward softer execution metrics and interactive settings, but are still described as lacking deep enterprise KB and role-aware preferences. KaggleDBQA provides denser queries over fewer databases, but is presented as lacking the private domain KB, user profiles, and implicit-intent emphasis central to enterprise lifelong learning. BIS and BEAVER address business data with wide tables, yet are described as having relatively few questions, insufficient to drive or evaluate KB accumulation over time (Chen et al., 25 Aug 2025).

Against that background, RubikBench is presented as complementary rather than merely larger. Its defining additions are a single realistic financial schema with wide tables and domain constraints, a lifelong-learning regime over one stable database, context-aware queries tied to roles and default settings, and a recall-oriented evaluation metric that tolerates practically irrelevant output variations while still penalizing substantive errors (Chen et al., 25 Aug 2025).

At the same time, RubikBench0.9 is explicitly an initial release. The paper does not report baseline or SOTA performance on RubikBench itself, and no public leaderboard is included yet (Chen et al., 25 Aug 2025). RubikSQL’s experimental claims instead concern KaggleDBQA and BIRD Mini-Dev, where the system achieves SOTA with majority voting and SQL-execution verification at $$EX = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[\text{exec}(\text{SQL}_i) = \text{ans}_i],$$3 (Chen et al., 25 Aug 2025). Availability is listed as “Dataset: URL_TO_BE_RELEASED_SOON,” the license is not specified, and the paper describes the evaluation methodology without providing ready-to-run scripts or a public leaderboard (Chen et al., 25 Aug 2025).

Several limitations are also explicit. Query-type and difficulty distributions are not yet reported. Potential domain bias toward automotive finance and management reporting remains. Broader coverage across domains and more detailed splits and protocols are planned (Chen et al., 25 Aug 2025). Annotation noise and other benchmark-internal issues are acknowledged as a class of problem; the paper notes that approximately 14% of errors in its BIRD analysis stem from benchmark issues, and a plausible implication is that RubikBench will likewise require iterative curation (Chen et al., 25 Aug 2025). Future work is described in terms of iterative refinement, more ablation studies for query synthesis and transfer components, and deeper integration of structured information extraction and knowledge distillation to support research on lifelong-learning NL2SQL and KB-centric methods (Chen et al., 25 Aug 2025).

In this form, RubikBench operationalizes industrial NL2SQL as a KB-dependent, context-sensitive, in-database learning problem rather than as a pure semantic parsing task. Its significance lies less in raw dataset size than in the decision to encode enterprise defaults, user roles, temporal business logic, and schema irregularities directly into the benchmark substrate.