OpenMLDB: SQL-based Real-Time Feature Computation

Updated 12 July 2026

OpenMLDB is an industrial real-time relational feature computation system that unifies SQL-based feature extraction for both offline training and online serving.
It leverages a unified execution engine with specialized optimizations like long-window pre-aggregation and self-adjusted window union to meet millisecond-level latency requirements.
The system addresses consistency challenges and performance inefficiencies in traditional ML pipelines while reducing memory usage and deployment overhead.

Searching arXiv for papers on OpenMLDB and closely related systems. arXiv search query: OpenMLDB real-time relational feature computation online ML OpenMLDB is an industrial real-time relational data feature computation system for online machine learning that computes features directly from relational tables for both offline training and online serving using the same SQL-based interface and the same core execution logic (Zhou et al., 15 Jan 2025). It is positioned as a feature computation system rather than merely a storage-oriented feature store: its central objective is to maintain correctness and consistency between training and serving while meeting millisecond-level latency requirements for live inference and supporting large-scale batch feature generation for model training (Zhou et al., 15 Jan 2025). In the FeatInsight system on 4Paradigm’s Sage Studio platform, OpenMLDB serves as the execution engine underneath a broader feature lifecycle workflow covering design, storage, visualization, computation, verification, and lineage management (Tong et al., 1 Apr 2025). A separate optimization study characterizes it as an open-source database system designed specifically for SQL+ML workloads, emphasizing real-time feature computation, unified SQL feature definitions, and avoidance of training-serving skew (Sidiq et al., 19 Sep 2025).

1. System scope and problem setting

OpenMLDB addresses a recurring systems problem in online ML pipelines: feature computation is typically split into an offline stage for model training and an online stage for model serving, and these stages often rely on execution engines with different interface languages and function implementations, causing significant inconsistencies (Zhou et al., 15 Jan 2025). The problem is not confined to storage or data access. The data block emphasizes that many online ML scenarios involve hundreds of features, multiple data sources, and complex computation patterns such as long-window computations and multi-window joins, while feature management itself can take “up to 70% of the overall latency in sales prediction service” (Tong et al., 1 Apr 2025). OpenMLDB is therefore designed around feature computation as a first-class workload.

The system is motivated by the observation that many online ML features are not well served by generic streaming or analytical engines. Existing data processing systems such as Spark, Flink, and DuckDB often incur multi-second latencies for the relevant computations, which makes them unsuitable for real-time online ML applications demanding timely feature updates (Zhou et al., 15 Jan 2025). The paper also argues that general-purpose online storage engines such as MySQL, PostgreSQL, Cassandra, HBase, DynamoDB, and MongoDB are not optimized for high-frequency ingestion, time ordering, low-latency temporal access, or feature-oriented window queries (Zhou et al., 15 Jan 2025). This places OpenMLDB in a distinct design space: a relational feature computation layer specialized for time-sensitive ML workloads rather than a conventional DBMS or a storage-centric feature repository.

The workload assumptions are explicitly relational and temporal. Online ML feature computation is described as depending on long window aggregations, multi-table window unions, top-N, conditional, and time-series features, as well as repeated reuse of the same feature logic in both training and inference (Zhou et al., 15 Jan 2025, Sidiq et al., 19 Sep 2025). This suggests that OpenMLDB’s significance lies in operational unification: one SQL feature script is written once and then deployed in multiple modes without a separate hand-engineered online rewrite.

2. Architectural organization and execution model

OpenMLDB has four major components: a unified query plan generator, an online real-time execution engine, an offline batch execution engine, and compact time-series data management (Zhou et al., 15 Jan 2025). It supports multiple execution modes using the same SQL syntax, specifically offline execution mode, online preview mode, and online request mode (Zhou et al., 15 Jan 2025). The architectural claim is that offline and online behavior are unified not by a high-level compatibility layer alone but through shared compilation and runtime mechanisms.

The unified query plan generator is the principal mechanism for offline-online consistency. It accepts standard SQL plus OpenMLDB extensions, compiles queries to efficient C++ machine code using LLVM/JIT, uses the same shared C++ library functions for both offline and online engines, and applies cycle binding and compilation caching (Zhou et al., 15 Jan 2025). In the FeatInsight integration, visual SQL is transformed into executable SQL statements and run through OpenMLDB’s SQL execution engines; once verified, the SQL can be compiled into efficient C++ machine code with parsing optimization, cyclic binding, and compilation caching (Tong et al., 1 Apr 2025). The same paper describes OpenMLDB as the runtime that evaluates feature definitions, while FeatInsight provides the orchestration, management, validation, versioning, and deployment layer (Tong et al., 1 Apr 2025).

A later optimization analysis describes the system in terms of a dual-engine architecture. In that description, the online mode is built for real-time inference, serves as a low-latency feature store, is optimized for time-series data, and uses a custom SQL execution engine with LLVM-based JIT compilation, in-memory processing, pre-aggregation, and caching or materialization; the offline mode is integrated with Apache Spark for large-scale historical feature computation while executing the same SQL feature definitions used in online mode (Sidiq et al., 19 Sep 2025). The data block thus presents two complementary views: one stresses shared plan generation and unified runtime semantics, and the other foregrounds the operational split between the low-latency online engine and Spark-based offline execution. Both accounts agree that the defining principle is identical SQL feature definitions across training and serving.

FeatInsight’s use of OpenMLDB also clarifies the broader system role. OpenMLDB sits in the “unified feature computation” layer and underpins both offline export for training and online feature services for inference (Tong et al., 1 Apr 2025). Verification is performed by computing test data through both offline and online execution engines and comparing results for consistency (Tong et al., 1 Apr 2025). A plausible implication is that OpenMLDB is intended not merely to accelerate feature extraction but to make feature semantics auditable across the full deployment lifecycle.

3. SQL model, extensions, and feature-oriented semantics

OpenMLDB exposes a SQL-based interface tailored for ML feature extraction rather than generic relational querying alone (Zhou et al., 15 Jan 2025). The paper highlights several feature-oriented functions, including topN_frequency, avg_cate_where, drawdown, ew_avg, split_by_key, and multiclass_label (Zhou et al., 15 Jan 2025). It also defines specialized clauses such as WINDOW UNION, LAST JOIN, and ROWS_RANGE ... MAXSIZE (Zhou et al., 15 Jan 2025). These constructs are presented as reducing query complexity relative to standard SQL rewrites.

WINDOW UNION is especially important because it supports combining data from multiple tables directly inside the window definition, which is critical for ML features dependent on multiple streams or tables (Zhou et al., 15 Jan 2025). LAST JOIN is described as fetching the latest matching tuple efficiently, and ROWS_RANGE ... MAXSIZE bounds the number of tuples in a window (Zhou et al., 15 Jan 2025). In the broader feature-management setting described by FeatInsight, feature views are defined by SQL, and the system executes these SQL-defined transformations in OpenMLDB for both offline and online pipelines before comparing outputs on test data for consistency verification (Tong et al., 1 Apr 2025).

The SQL model is explicitly oriented toward feature reuse and unified definition. The optimization study states that OpenMLDB allows users to define features once in SQL and execute them consistently in both environments, thereby addressing training-serving skew (Sidiq et al., 19 Sep 2025). This makes OpenMLDB conceptually adjacent to other database-centric ML efforts, but not identical to them. For example, sql4ml is a declarative system in which feature engineering and model expressions are written in SQL and then translated to TensorFlow for training (Makrynioti et al., 2019). The key contrast is that sql4ml uses SQL primarily as a specification language whose numerical optimization is delegated to TensorFlow, whereas OpenMLDB-style systems are described as executing feature pipelines directly in the database or serving engine and focusing on integrated feature computation and serving (Makrynioti et al., 2019). This distinction matters because OpenMLDB’s novelty lies in native runtime support for feature-oriented SQL under online constraints rather than in SQL-defined model training.

The name similarity to the OpenML ecosystem can be misleading. “OpenML Benchmarking Suites” concerns the OpenML platform, benchmark suites, OpenML tasks, and the OpenML-CC18 classification suite, and explicitly does not discuss OpenMLDB, database systems, SQL engines, or serving or storage infrastructure (Bischl et al., 2017). This disambiguation is necessary because OpenMLDB is a database and execution system for SQL+ML feature computation, whereas OpenML is a platform for benchmark management and reproducible ML experimentation.

4. Online execution optimizations

The online execution engine is built from scratch to achieve millisecond-level latency and focuses on two major optimizations: long window pre-aggregation and self-adjusted window union (Zhou et al., 15 Jan 2025). These target the computational structures that generic stream processors handle poorly in the online ML setting.

Long-window features are expensive because they may require scanning large volumes of historical data, including windows spanning months or years or windows associated with hotspot keys (Zhou et al., 15 Jan 2025). OpenMLDB’s solution is long window pre-aggregation. The mechanism has three steps: aggregator initialization, query refinement, and aggregator update (Zhou et al., 15 Jan 2025). During initialization, a hierarchy of aggregators at different time granularities, such as daily and monthly, is built based on query frequency, window sizes, data distribution, and maintenance overhead; the paper mentions using segment trees to manage historical aggregation results (Zhou et al., 15 Jan 2025). At query time, raw scans are replaced by lookups and merges from the aggregator hierarchy, with only residual portions computed from raw data; updates are asynchronous and coordinated through the binlog or replicator path (Zhou et al., 15 Jan 2025). For one long-window case, the configuration deploy test1 OPTIONS(long_windows="w1:1d") reduced response time from 300 ms to 6 ms, a 45× improvement (Zhou et al., 15 Jan 2025).

Self-adjusted window union addresses the problem that a static key-based distribution scheme can become badly load-imbalanced when key cardinality is low or access patterns are skewed (Zhou et al., 15 Jan 2025). Its core ideas are on-the-fly load balancing and incremental computation (Zhou et al., 15 Jan 2025). Keys are dynamically remapped to worker threads based on observed runtime load, and multiple workers can cooperate on the same key subset if needed; instead of recomputing a whole window, the engine subtracts expired tuples and incrementally incorporates new ones (Zhou et al., 15 Jan 2025). In FeatInsight’s description of online processing, OpenMLDB similarly uses pre-aggregation for long intervals and hotspot data plus dynamic data adjustment for out-of-order streams over multiple tables, explicitly citing scalable online interval join in OpenMLDB (Tong et al., 1 Apr 2025).

The optimization study provides a broader decomposition of online performance factors. It evaluates better query plans, cached execution plans, parallel processing, and resource management, reporting approximately 35% of performance gains from query optimization, 25% from caching or materialization, 20% from parallel processing, and 10% from resource management, while noting some inconsistency across sections in the exact percentages (Sidiq et al., 19 Sep 2025). The same study reports that OpenMLDB can support approximately 12,500 QPS with less than 1 ms latency in optimized settings, and elsewhere reports approximately 17,000 QPS with approximately 4 ms latency (Sidiq et al., 19 Sep 2025). Because the data block itself notes these inconsistencies, the most rigorous interpretation is that the paper presents multiple benchmark summaries under different reporting contexts rather than a single canonical figure.

5. Offline execution and consistency mechanisms

The offline execution engine targets high-throughput batch feature computation for training data preparation and introduces two principal optimizations: multi-window parallel optimization and time-aware data skew resolving (Zhou et al., 15 Jan 2025). These address the fact that large historical feature jobs frequently contain multiple windows over the same table and are highly sensitive to skew.

Multi-window parallel optimization exploits the independence of multiple window functions over the same table. Traditional systems often compute such windows serially, whereas OpenMLDB parallelizes them and then stitches the outputs together (Zhou et al., 15 Jan 2025). The implementation introduces two special plan nodes, Simple Project and Concat Join, marking the beginning and end of a parallel segment; an index column is injected into input rows so that each window emits features plus this index, allowing results to be joined back correctly despite different partitioning or ordering strategies (Zhou et al., 15 Jan 2025). The paper states that the extra merge overhead is reduced by using the specialized Last Join path (Zhou et al., 15 Jan 2025).

Time-aware data skew resolving is designed for order-sensitive window computations where simple salting would be unsafe (Zhou et al., 15 Jan 2025). Instead of random repartitioning, OpenMLDB repartitions based on data distribution and time structure (Zhou et al., 15 Jan 2025). The steps are to determine partition boundaries using statistics such as quantiles and percentiles with HyperLogLog-based approximation, assign repartitioning identifiers including PART_ID and an EXPANDED_ROW flag, augment window data by copying needed boundary tuples from neighboring partitions, redistribute data by original key plus PART_ID, and then compute on repartitioned data while emitting outputs only for original rows (Zhou et al., 15 Jan 2025). This preserves correctness while increasing parallelism.

The consistency problem is central to the offline path as well. The OpenMLDB paper argues that using the same SQL-based interface and the same shared C++ library functions for both engines significantly reduces feature deployment overhead and avoids semantic divergence (Zhou et al., 15 Jan 2025). FeatInsight operationalizes this with explicit verification, computing test data through both offline and online execution engines and comparing the results for consistency (Tong et al., 1 Apr 2025). This suggests that OpenMLDB’s offline engine should be understood as part of a joint semantics-preservation mechanism, not simply as a batch accelerator.

The later optimization study again reinforces the point by describing the offline engine as Spark-based but executing the same SQL feature definitions used in online mode (Sidiq et al., 19 Sep 2025). Its description of a modular optimization framework in which batch and stream processing are continuously integrated but operationally separated enough to avoid interference aligns with the original system objective of unified feature definitions with differentiated runtime paths (Sidiq et al., 19 Sep 2025).

6. Data layout, indexing, and resource management

A major part of OpenMLDB’s performance derives from its compact data format and stream-focused indexing (Zhou et al., 15 Jan 2025). The in-memory row encoding has four parts: a 6-byte header storing field version, schema version, and row size; a BitMap indicating NULL columns; basic fixed-size type data; and variable-length fields stored by offsets rather than full length metadata (Zhou et al., 15 Jan 2025). In a concrete comparison, a Spark row is reported as 556 bytes and an OpenMLDB row as 255 bytes, corresponding to over 54% memory saving in that example (Zhou et al., 15 Jan 2025). The system also uses JNI to read Java’s UnsafeRow directly from C++ to avoid unnecessary encode or decode overhead (Zhou et al., 15 Jan 2025).

For online access, OpenMLDB uses a refined skiplist whose first layer is ordered by key and second layer points each key to a list or skiplist of tuples ordered by timestamp (Zhou et al., 15 Jan 2025). This structure supports concurrent lock-free reads and writes via CAS, efficient retrieval of the latest tuple for a key, and quick eviction of expired tuples by timestamp order (Zhou et al., 15 Jan 2025). The FeatInsight paper gives a closely related account of a compact in-memory data structure with pre-sorting by key and timestamp, combining fixed-size and variable-size data structures, lock-free reads and writes with atomic compare-and-swap, and removal of outdated data through timestamp ordering and batch deletion; in the fraud detection discussion it mentions a refined skiplist sorted by both key and timestamp and a lightweight row-encoding scheme that supports lock-free updates (Tong et al., 1 Apr 2025).

Persisted online data is stored using RocksDB with multiple column families as indexes, a shared memtable based on the refined skiplist, and composite keys formed from key plus timestamp (Zhou et al., 15 Jan 2025). This keeps on-disk and in-memory layouts aligned with time-series access patterns. The system also offers an empirical memory estimation model, max_memory_mb-based memory isolation, and threshold-based alerting (Zhou et al., 15 Jan 2025). If memory is exceeded, writes fail while reads continue and service stays online (Zhou et al., 15 Jan 2025). The paper advises in-memory storage when low latency is required and memory is sufficient, and disk-based storage when memory is tight and 20–30 ms latency is acceptable (Zhou et al., 15 Jan 2025).

The optimization analysis frames resource management more abstractly, formalizing an objective $\max Q(C, M)$ subject to resource constraints and claiming that resource management contributes about 10% of performance gain, with 40–50% less memory and 30–40% less CPU compared with baseline implementations (Sidiq et al., 19 Sep 2025). Since these claims come from a different study with synthetic experiments, they are best read as complementary evidence for the importance of resource control rather than as replacements for the memory-layout evidence in the system paper.

The OpenMLDB paper evaluates the system on MicroBench and real workloads including TalkingData, Akulaku RTP, and Akulaku GLQ (Zhou et al., 15 Jan 2025). On MicroBench, OpenMLDB outperforms MySQL (in-memory) by over 68.4% lower latency, DuckDB by 87.7% lower latency, and Trino + Redis by over 96% lower latency, while achieving over 17× higher throughput than these baselines (Zhou et al., 15 Jan 2025). On the Akulaku RTP workload, it achieves approximately 0.98 ms for Top1 and about 5 ms for Top8, while Flink remains in the sub-100 ms range (Zhou et al., 15 Jan 2025). For offline workloads, it reports 2.6× speedup on single-window queries, 6.3× speedup on multi-window workloads, and 7.2× latency reduction on skewed data, with 180 s versus 1302 s (Zhou et al., 15 Jan 2025). On the 10-billion-tuple GLQ workload, response time is around 30 ms, with speedups ranging from 5× to over 22× over Spark as workload complexity increases (Zhou et al., 15 Jan 2025).

Memory comparisons against Trino + Redis show substantial savings across scales, with reductions from 74.77% at 10,000 tuples to 45.66% at 184,903,890 tuples (Zhou et al., 15 Jan 2025). Ablation findings further report that long-window pre-aggregation can reduce one request from 300 ms to 6 ms, that multi-table window union optimization maintains around 1,000,000 tuples/sec while static approaches can degrade to around 1,000 tuples/sec on large windows, and that multi-window parallel optimization yields 4.8×, 5.3×, and 4.6× speedups for small, medium, and large windows respectively (Zhou et al., 15 Jan 2025).

FeatInsight supplies additional production-facing evidence because OpenMLDB is the execution engine in a full feature lifecycle platform deployed in over 100 real-world scenarios on 4Paradigm’s Sage Studio platform, handling up to a trillion-dimensional feature space and enabling millisecond-level feature updates (Tong et al., 1 Apr 2025). In a Vipshop product recommendation scenario with 720 million order records updated daily, the platform can deploy features within an hour, reducing feature development-to-deployment to five person-days and improving efficiency by over 60% (Tong et al., 1 Apr 2025). In fraud detection at a major Fortune Global 500 bank, the system handles about 100 million customer records, 500 million card records, and 1.7 billion transactions, with roughly 4.5 million transactions per day; a naive Spark system gets around 200 ms response time, a customized in-house Spark version gets around 50 ms, and the FeatInsight-based solution achieves sub-20 ms response with QPS exceeding 1000 while maintaining higher recall (Tong et al., 1 Apr 2025). Because OpenMLDB is identified as the central execution engine in this architecture, these results connect the database’s feature-computation model to end-to-end operational performance.

The OpenMLDB paper also emphasizes practical adoption, stating that the system is deployed in 4Paradigm’s SageOne platform and over 100 real scenarios, and that the open community has over 150 contributors and 1.6k stars on GitHub (Zhou et al., 15 Jan 2025). The same source notes that it has been recognized as a recommended feature product by Gartner (Zhou et al., 15 Jan 2025). This suggests that OpenMLDB should be understood not as a narrow benchmark artifact but as a production-oriented system whose research contribution is inseparable from deployment constraints.

In relation to nearby research areas, OpenMLDB occupies the intersection of feature stores, relational stream processing, and database-centric ML systems. The system paper explicitly argues that products such as Feast, Tecton, Hopsworks, and Feathr focus mostly on feature serving or retrieval rather than on high-performance feature computation (Zhou et al., 15 Jan 2025). Sql4ml represents a different line of work in which SQL unifies specification but TensorFlow remains the optimization backend (Makrynioti et al., 2019). OpenML benchmarking suites, despite nominal similarity, belong to experiment management and reproducibility infrastructure rather than to SQL execution or online feature serving (Bischl et al., 2017). Within this landscape, OpenMLDB’s defining characteristic is the unification of offline and online feature computation around specialized SQL semantics, feature-oriented operators, and runtime mechanisms tuned for real-time relational data feature computation (Zhou et al., 15 Jan 2025).