IOLM-DB: OLAP Query-Specific LLM Integration

• IOLM-DB is an OLAP query-processing architecture that generates tiny, query-specific LLMs for efficient per-row inference on large datasets.
• It leverages a calibration set from the data to drive post-training compression via quantization, sparsification, and structural pruning without gradient fine-tuning.
• The system achieves up to 76% model footprint reduction and 3.31× throughput gains by integrating OLAP-aware caching, batching, and hardware parallelism.

Searching arXiv for the target paper and named related methods. arXiv search: target paper (Mohammadi et al., 7 Jul 2025) arXiv search: GPTQ SparseGPT LLM-Pruner IOLM-DB is an OLAP query-processing architecture for embedding LLM functionality inside analytical queries by generating, on the fly, a tiny, query-specific version of a large pretrained model rather than invoking a general-purpose LLM for every row. It is designed to make “one-row-at-a-time” LLM calls practical at scale, especially for workloads that process millions to billions of rows. The system is presented as an extension of an existing OLAP engine and is motivated by the observation that LLM-enhanced analytics—such as data summarization, cleaning, and semantic transformation—remain prohibitively expensive in computation and memory when deployed naïvely. The reported prototype reduces model footprints by up to 76% and increases throughput by up to $3.31\times$ while maintaining accuracy through aggressive compression, higher parallelism, and OLAP-aware caching and batching (Mohammadi et al., 7 Jul 2025).

1. System role within OLAP execution

IOLM-DB sits as an extension of an existing OLAP engine; in the prototype, the host engine is based on pandas DataFrames. When the SQL planner encounters a special prompt(...) operator in a query, that sub-expression is dispatched to IOLM-DB’s LLM-invocation layer rather than to normal relational operators. At query compile time, the system extracts the text of the prompt, identifies the target column or columns, samples a small calibration set of rows to represent the prompt’s data distribution, and triggers a model-generation pipeline that produces a tiny, optimized LLM tailored for that prompt and those data patterns. At runtime, a custom “LLM-operator” invokes the compressed model like any other operator in the pipeline and returns a new column of predictions (Mohammadi et al., 7 Jul 2025).

This architecture reframes LLM execution in OLAP from a generic inference service into a compile-time and runtime operator specialization problem. A plausible implication is that IOLM-DB treats the prompt and the input-column distribution as part of physical query planning, rather than as an external application-layer concern. That design choice is central to its claim that per-row LLM invocation can become feasible inside analytical systems.

2. Query-specific model generation

The model-generation procedure is driven by a representative sample $S$ drawn from the full table. IOLM-DB takes a random, or stratified, subset of the table—often $0.1\%$ to $1\%$ of rows, capped at a few hundred examples—as calibration data. The same $S$ is used both to estimate quantization scales and to score the importance of weights and heads. The system uses only post-training adaptation and explicitly avoids gradient-based fine-tuning in order to keep overhead in the single-digit minutes (Mohammadi et al., 7 Jul 2025).

Calibration data drives three categories of decisions. First, it determines quantization scaling factors by minimizing $\lVert W_{fp16} - W_q \cdot scale \rVert_2$ over $S$. Second, it determines unstructured or semi-structured pruning thresholds, for example by removing weights satisfying $|w| < \tau$ where $\tau$ is chosen so that $p\%$ of weights become zero. Third, it determines structural pruning choices, such as dropping attention heads whose mean output magnitude on $S$0 falls below a threshold.

A common misunderstanding is that query specialization in this setting implies task-specific fine-tuning. IOLM-DB instead specializes via post-training compression and calibration. This suggests a systems-oriented interpretation of specialization: the model is adapted to the query instance and its observed data distribution, but not retrained through gradient updates.

3. Compression pipeline and model transformations

The compression recipe is expressed in terms of an original model $S$1 and a compressed model $S$2. In the reported configuration, $S$3 is the original 8B-parameter model of size $S$4 GB on GPU, instantiated as Llama-3.1-8B-Instruct with measured footprint $S$5 GB, while $S$6 has size $S$7 GB, specifically $S$8 GB. The paper defines the compression ratio as

$S$9

and reports this as a $0.1\%$0 reduction in footprint (Mohammadi et al., 7 Jul 2025).

The recipe combines quantization, sparsification, and structural pruning in a one-shot pipeline. For weight quantization, $0.1\%$1 is mapped to $0.1\%$2 or even $0.1\%$3 under a symmetric per-channel scheme; for each output channel $0.1\%$4, the scale $0.1\%$5 is chosen to minimize

$0.1\%$6

Activation quantization is optional and uses dynamic 8-bit per-token scaling. Sparsification includes semi-structured $0.1\%$7 sparsity, where in every block of four weights exactly two are zero, and unstructured pruning toward a target sparsity such as $0.1\%$8 zeros. Structural pruning removes entire attention heads or feed-forward sublayers whose $0.1\%$9 contribution on $1\%$0 is negligible; with LLM-Pruner, heads $1\%$1 are dropped when

$1\%$2

In practice, the pipeline is given as GPTQ $1\%$3 SparseGPT $1\%$4 LLM-Pruner so that quantization and sparsity interact harmoniously (Frantar et al., 2022, Frantar et al., 2023, Ma et al., 2023). Within IOLM-DB, these are not presented as independent optimizations but as a coordinated transformation sequence for constructing a query-specific runtime model.

4. Execution model: parallelism, caching, and batching

The systems consequence of compression is not limited to smaller storage. Smaller weight matrices and lower bit-width arithmetic reduce GPU memory pressure, allowing more model “batch-copies” to reside concurrently on-chip, and increase arithmetic intensity with INT8 or INT4 kernels, unlocking hardware DPUs/TPUs. With the compressed $1\%$5, a batch of $1\%$6 rows can be subdivided into $1\%$7 micro-batches—for example, $1\%$8—processed in parallel across separate CUDA streams or CPU threads. This is contrasted with $1\%$9, whose 16-bit weights would only fit, for example, $S$0 micro-batches simultaneously (Mohammadi et al., 7 Jul 2025).

IOLM-DB further adds OLAP-aware caching and batching layers. For caching, when the same prompt plus input text recurs, the system hashes the input and consults an LRU cache, thereby avoiding re-inference. For batching, the LLM-operator collects several rows—for example, $S$1–$S$2—before invoking the transformer stack so that tokenization and kernel launch overhead are amortized. Effective throughput in rows per second is expressed as

$S$3

where $S$4 is near-zero for $S$5 because the model is already in memory and $S$6 scales sublinearly in $S$7.

The latency model is given approximately by

$S$8

Using IOLM-DB’s $S$9, $\lVert W_{fp16} - W_q \cdot scale \rVert_2$0 drops by approximately $\lVert W_{fp16} - W_q \cdot scale \rVert_2$1 thanks to INT8 kernels and pruning. This suggests that IOLM-DB’s gains arise from a composition of model compression and operator scheduling rather than from compression alone.

5. Empirical behavior

The reported experiments quantify both footprint reduction and end-to-end throughput gains for several OLAP-style LLM tasks. The baseline model is Llama-3.1-8B-Instruct at $\lVert W_{fp16} - W_q \cdot scale \rVert_2$2 GB, and the compressed variants—described as IOLM-DB-Perf or IOLM-DB-Acc—occupy $\lVert W_{fp16} - W_q \cdot scale \rVert_2$3 GB. Throughput is reported in rows per second for summarization, data correction, and fuzzy join workloads (Mohammadi et al., 7 Jul 2025).

Workload	Throughput change	Accuracy note
Summarization	$\lVert W_{fp16} - W_q \cdot scale \rVert_2$4 ($\lVert W_{fp16} - W_q \cdot scale \rVert_2$5)	$\lVert W_{fp16} - W_q \cdot scale \rVert_2$6 accuracy
Summarization	$\lVert W_{fp16} - W_q \cdot scale \rVert_2$7 ($\lVert W_{fp16} - W_q \cdot scale \rVert_2$8)	full accuracy
Data correction	$\lVert W_{fp16} - W_q \cdot scale \rVert_2$9 ($S$0)	full accuracy
Fuzzy join	$S$1 ($S$2)	full accuracy

These results are presented as evidence that “one-row-at-a-time” LLM calls can be integrated into analytical pipelines without prohibitive throughput collapse. At the same time, the summarization result at $S$3 rows/s with $S$4 accuracy indicates that aggressive compression can impose a measurable quality trade-off. The existence of both IOLM-DB-Perf and IOLM-DB-Acc variants reflects an explicit performance-versus-accuracy operating point rather than a claim of universally lossless compression.

6. Operational considerations, limitations, and future directions

The system’s principal limitation is its upfront compilation cost. Building $S$5 takes on the order of single-digit minutes. For long-running OLAP scans over $S$6–$S$7 rows, that cost is described as quickly amortized, but the amortization assumption depends on the duration and scale of the workload (Mohammadi et al., 7 Jul 2025).

Model management introduces a second constraint. In a multi-tenant environment, $S$8 variants for different prompts must be cached on disk or in GPU memory, and intelligent eviction policies such as LRU or cost-based policies are needed. Accuracy trade-offs constitute a third limitation: the paper explicitly notes that aggressive compression can slightly degrade quality, with summarization moving from $S$9 to $|w| < \tau$0. To expose this trade-off, IOLM-DB provides a “Perf vs. Acc” knob so that DBAs can choose the desired operating point.

The future directions named for the system are cascading inference, knowledge distillation, and tighter integration into a production OLAP engine’s C++ kernel. Cascading inference would use coarse and fine models to avoid querying a “heavy” compressed model on every row. Knowledge distillation would transfer behavior from $|w| < \tau$1 to an even smaller student tailored to the prompt. Replacing the pandas-based prototype path with a production C++ kernel is identified as a route to eliminating Python overhead and potentially yielding another $|w| < \tau$2–$|w| < \tau$3 speedup.

More broadly, IOLM-DB positions LLM functionality as a first-class OLAP operator whose implementation can be specialized per query. This suggests a shift in database-system design: rather than treating foundation models as fixed external services, future analytical engines may synthesize compact per-query inference artifacts from representative data samples and execute them under conventional database concerns such as batching, caching, and hardware-aware parallelism.