Learned Local Tables

• Learned local tables are data structures that combine memorized corrections with statistical learning to dynamically adjust to local data trends.
• They underpin diverse applications including tabular prediction (LoCalPFN), database indexing (Shift-Table), video in-loop filtering (LUT-ILF++), and Bayesian network structure learning.
• By fusing global model predictions with localized adjustments, these tables enhance accuracy, speed, and memory efficiency in practical deployments.

A learned local table is a class of data structure that leverages local regularities or model predictions in order to fuse the advantages of “memorized” table-based methods with those of statistical learning, providing either faster queries (databases), more efficient context adaptation (deep learning), or more compact representations (probabilistic models). The form and function of learned local tables spans supervised deep tabular modeling, database range indexing, in-loop filtering for video codecs, and local structure discovery in probabilistic graphical models. This article systematically reviews key methodologies, architectures, and empirical results from recent state-of-the-art approaches including LoCalPFN for tabular prediction, Shift-Table for learned database indexing, LUT-driven video ILF, and local structure learning in Bayesian networks.

1. Local Table Constructions in Modern Machine Learning

Learned local tables arise in diverse domains but share a core principle: empirical data is partitioned or represented locally, and small, efficiently retrievable tables encode the corrections, context, or parameters needed for high-accuracy prediction or rapid lookup.

LoCalPFN: Local Context Calibration for Tabular Transformers

In tabular deep learning, LoCalPFN leverages $k$-nearest-neighbour retrieval in a canonical embedding space and adapts a large transformer (TabPFN) to each query’s locality. Each test row $q$ retrieves its $k$ nearest neighbours $S = \text{NN}_k(q)$, and these, plus $q$, are packed into the model’s input context, which is then processed by the unchanged TabPFN backbone. This “local table” of context points is dynamically constructed per inference, enabling the model to adjust to local data density and feature correlations (Thomas et al., 2024).

Shift-Table: Micro-local Correction for Learned Indexes

In database range indexing, Shift-Table applies a table-correction layer to learned CDF models (e.g., RMI, interpolation models). For a predicted position $f(x)$, a local table indexed by $k=f(x)$ stores a precomputed drift $\Delta_k$ and bucket count $C_k$, which corrects the raw model output to near-exact lookup via at most one table read and subsequent local scan or binary search. The shift table thereby “memorizes” local deviations not modeled by a compact global predictor (Hadian et al., 2021).

LUT-ILF++: Lookup Table-Based Video In-Loop Filtering

In video coding, neural in-loop filtering models ($F_\theta$) are distilled into multi-dimensional LUTs by enumerating model outputs over coarse quantized input grids. These LUTs, possibly cascaded and pruned, provide rapid low-latency filtering by direct table retrieval and interpolation for local pixel neighborhoods, pairing the accuracy of DNNs with the efficiency of lookup operations (Li et al., 11 Sep 2025).

Local Structure in Bayesian Networks

In graphical models, learned local tables take the form of default tables or context trees within conditional probability tables (CPTs). Instead of a full, exponentially-sized CPT, a local structure partitions parent-variable configurations so that each class shares a parameter vector, dramatically reducing complexity and improving sample efficiency (Friedman et al., 2013).

2. Mathematical Formulations and Construction Algorithms

Learned local tables are constructed using domain-specific algorithms tailored to their use-case.

LoCalPFN Retrieval and Packing

Let the dataset be $q$0, $q$1. Each query $q$2 retrieves $q$3 neighbours under standardized embedding space via Euclidean distance:

$q$4

A context table is formed as a sequence of $q$5 for each $q$6, followed by $q$7 for prediction (Thomas et al., 2024).

Shift-Table Layer Construction

Given learned model $q$8 mapping key $q$9 to CDF prediction, bucket $k$0, store per-bucket:

• $k$1
• $k$2

At query time, correct prediction via $k$3 and limit scan to $k$4 records (Hadian et al., 2021).

Algorithm Table: Key Construction Steps

Method	Table Construction	Query/Inference Mechanism
LoCalPFN	kNN retrieval, context packing	Transformer over context+query
Shift-Table	Per-bucket $k$5, $k$6 precompute	Model+1 table fetch; corrected scan
LUT-ILF++	Enumerate DNN on quant grid for LUT	LUT lookup, interpolation
Local CPTs	Greedy add/prune default rows, trees	Parent assignment → default/cell → param retrieval

3. Training, Optimization, and Local Adaptivity

LoCalPFN Fine-Tuning

Training optimizes cross-entropy plus $k$7 regularization over local context tables constructed for each query. Efficient batching groups queries with overlapping contexts to maximize hardware utilization. The entirety of TabPFN’s parameters is fine-tuned end-to-end, but only in the context of locally retrieved data (Thomas et al., 2024).

Shift-Table Model Correction

Shift-Table is a post-processing layer that requires only a single pass over keys to build the table and offers $k$8 correction at lookup. It can use buckets $k$9 to trade accuracy for space. Correction is deterministic and requires no further training (Hadian et al., 2021).

LUT Training and Distillation

In LUT-ILF++, a compact DNN is trained under restricted input ranges, after which all quantized input values are swept to fill out the LUT. Storage is pruned via quantization, index partitioning (diagonal/non-diagonal), and further binning schemes, balancing accuracy with memory footprint. Interpolation enables continuous-valued input support at inference (Li et al., 11 Sep 2025).

Local Structure Scoring

For Bayesian networks, the minimum description length (MDL) or Bayesian scoring is used, jointly searching over both the global graph and local representation (default table or tree per node). Parameters are estimated for each local context group, with local table size regularized in the score (Friedman et al., 2013).

4. Empirical Performance and Scalability

Tabular Deep Learning

On OpenML’s 95-task benchmark, LoCalPFN outperforms both the base TabPFN and top-tuned tree models. In interquartile mean AUC, LoCalPFN achieves 0.958 (all datasets) and 0.968 (medium/large), versus XGBoost’s 0.943/0.965 and RandomForest’s 0.936 (Thomas et al., 2024). Fine-tuning on random context or using kNN context alone each yield $S = \text{NN}_k(q)$02% AUC gains, but the combined approach achieves $S = \text{NN}_k(q)$1.

Indexing and Search

In the SOSD benchmark, Shift-Table yields 1.7–1.8x speedup in lookup latency over tuned RMI and Radix-Spline on real datasets, reducing maximum per-query residuals to $S = \text{NN}_k(q)$2 and lowering mean absolute errors correspondingly (Hadian et al., 2021).

Video Coding

LUT-ILF++ achieves bitrate reductions of $S = \text{NN}_k(q)$3 (AI) and $S = \text{NN}_k(q)$4 (RA) on Y/U/V channels respectively, with storage cost $S = \text{NN}_k(q)$5 KB (full DNN: $S = \text{NN}_k(q)$6–$S = \text{NN}_k(q)$7 KB) and negligible runtime overhead, compared to pure DNN-based in-loop filtering (Li et al., 11 Sep 2025).

Probabilistic Models

Experiments in learned local structure for Bayesian networks show that default tables and context trees reduce parameter count by 50–80%, “unlock” denser learned graphs, and converge faster (by 20–40% in sample requirements) to true cross-entropy than naive full table learners (Friedman et al., 2013).

5. Representational Variants and Design Trade-offs

Table Dimensionality and Granularity

• LoCalPFN: Table is variable-sized per-query, determined by $S = \text{NN}_k(q)$8.
• Shift-Table: Bucket count and per-bucket granularity can be tuned ($S = \text{NN}_k(q)$9).
• LUT-ILF++: Pruning/quantization (4 MSBs; diagonal/non-diagonal splitting) determines table scale, interpolation handles in-between values.
• Bayesian local structures: Default tables minimize parameter count when many contexts share a distribution, trees capture context-specific regularities.

Storage and Computational Cost

• All methods rely on O(N) or sublinear memory for their local tables, with hardware-aware layout (cache-friendly, vectorized).
• For LoCalPFN, inference/fine-tuning compute is dominated by transformer forward/back (Thomas et al., 2024).
• LUT-based methods exploit integer arithmetic, caching, and fast indexing to maximize speed (Li et al., 11 Sep 2025, Hadian et al., 2021).

Adaptivity and Generalization

• LoCalPFN and Shift-Table adjust dynamically to local data properties.
• LUT-pruned ILF maintains generalization via training on augmented/rotated input neighborhoods.
• Local structure methods grow richer graphs by freeing model selection from prohibitive exponential parameterization penalties (Friedman et al., 2013).

6. Applications, Practical Guidelines, and Limitations

Practical Recommendations

• LoCalPFN: Use FAISS for kNN with $q$0, batch size $q$1, $q$2 for fine-tuning, early stop on validation AUC, standardized features (Thomas et al., 2024).
• Shift-Table: Memory layout as two arrays, choose $q$3 to fit cache/memory, backward fill for empty buckets, local search threshold (e.g., $q$4) (Hadian et al., 2021).
• LUT-ILF++: Train compact DNN, enumerate to LUT, prune via diagonal/non-diagonal split, use multi-dimensional interpolation, batch LUT accesses in cache-line-aligned order (Li et al., 11 Sep 2025).
• Bayesian networks: Search both global graph and local structure, default tables for many-context merging, trees for finer context specificity; add/prune rows or splits greedily (Friedman et al., 2013).

Limitations

• LoCalPFN inherits TabPFN’s architectural constraints (e.g., input dimension cap via one-hot).
• Shift-Table is less effective on synthetic smooth CDFs; cost of correction may not be justified in uniform regimes.
• LUT-based filtering is constrained by quantization and table pruning; very large input spaces may push storage limits.
• Local structure learning in probabilistic models increases search complexity and may become expensive for high-arity parents.

Cross-Domain Generality

A plausible implication is that learned local tables, though domain-specific in construction, reveal a generic pattern: fusing global modeling (deep network, index, probabilistic factorization) with localized, memorized correction or adaptation yields models with higher accuracy, faster inference, more robust parameters, and substantial space/computation savings in practical deployments.

• Retrieval & Fine-Tuning for In-Context Tabular Models (Thomas et al., 2024)
• Shift-Table: A Low-latency Learned Index for Range Queries using Model Correction (Hadian et al., 2021)
• In-Loop Filtering Using Learned Look-Up Tables for Video Coding (Li et al., 11 Sep 2025)
• Learning Bayesian Networks with Local Structure (Friedman et al., 2013)