jXBW: Extended BW for JSONL Search

• jXBW is a method that uses the JSONL eXtended Burrows–Wheeler Transform to rapidly index and search substructures within large-scale JSON Lines datasets.
• It merges individual JSON trees into a succinct merged-tree and employs an XBWT-based index to efficiently decompose queries into path hits and candidate subtree matches.
• Empirical evaluations show jXBW achieves speedups up to 4,700× over traditional tree-based approaches, making it ideal for prompt engineering and retrieval-augmented generation.

jXBW, expanded as JSONL eXtended Burrows–Wheeler Transform, is a method for fast substructure search over large-scale JSON Lines (JSONL) datasets. It targets a setting in which each JSONL line is treated as a structured object and queries are specified as subtree patterns rather than flat text predicates. The method combines three components: a merged-tree representation of many JSON objects, a succinct index derived from the eXtended Burrows–Wheeler Transform (XBWT), and a three-step search procedure based on path decomposition, ancestor computation, and adaptive tree-identifier collection. Reported experiments on real-world datasets show substantial speedups over tree-based and XML-based baselines while maintaining competitive memory usage (Tabei, 18 Aug 2025).

1. Problem setting and scope

Substructure search in JSONL datasets is described as essential for modern applications such as prompt engineering in foundation models, yet existing methods are characterized as computationally expensive because they rely on exhaustive tree traversal and subtree matching (Tabei, 18 Aug 2025). In this setting, each JSON object must be parsed into a tree, and a query asks whether some subtree pattern occurs within one or more of those trees.

The operational target of jXBW is not generic document retrieval but substructure retrieval over large JSONL corpora. The paper situates this need in workflows where JSONL acts as the de facto format for Foundation Model inputs, including GPT, Claude, and LLaMA. Fast retrieval of structured fragments is therefore connected to on-the-fly selection of context for database-style prompts and retrieval-augmented generation. A plausible implication is that jXBW addresses a bottleneck that emerges when structured corpora are queried repeatedly in latency-sensitive inference pipelines (Tabei, 18 Aug 2025).

A common misconception in this area is that JSONL substructure search can be reduced straightforwardly to XML tooling. The reported comparison against a JSON→XML + XQuery baseline indicates that such a reduction can incur severe performance penalties, especially at larger scales. Another misconception is that speedups in this domain must come from approximate or weakened matching; jXBW is instead framed as a correctness-preserving method that avoids exhaustive traversal through indexing and search decomposition (Tabei, 18 Aug 2025).

2. Merged-tree data model

Each JSONL line $O_i$ is parsed into a rooted, labeled tree $T_i$. Objects become unordered children under an “object” node, arrays become ordered children, and every leaf of $T_i$ is tagged with its source ID $i$ (Tabei, 18 Aug 2025). This object/array distinction is central because it determines where ordering constraints must later be enforced during matching.

The individual trees $\{T_i\}_{i=1}^N$ are merged into a single tree $MT$ by identifying common root-to-node label sequences, described as prefix paths. If two trees share the same label at a given position, those nodes are fused; otherwise, the divergent subtree is attached under the last common prefix. This merged-tree construction preserves per-object identity at the leaves: each leaf $\ell$ of $MT$ carries a set

$ids(\ell)\subseteq\{1,\dots,N\}$

containing all original trees contributing to that path (Tabei, 18 Aug 2025).

The resulting structure is explicitly bounded by

$|MT|\le \sum_i |T_i|,$

with the additional observation that in practice $T_i$0 when structure is widely shared. This means that repeated schemas or repeated nested patterns across a corpus are collapsed into shared prefixes rather than stored redundantly. The method therefore uses structural commonality as an indexing primitive rather than merely as a compression side effect.

Construction cost is analyzed in two variants. A naïve sequential merge would require

$T_i$1

By instead performing pairwise merges in a balanced divide-and-conquer schedule over $T_i$2 rounds, the total merge time becomes

$T_i$3

This suggests that jXBW’s scalability depends not only on succinct indexing but also on avoiding pathological merge orderings during preprocessing (Tabei, 18 Aug 2025).

3. XBWT-based succinct index

The merged tree $T_i$4 is converted into a compact index, also called jXBW, composed of four synchronized arrays of length $T_i$5. These are $T_i$6, which stores a symbol for each node via a bijection from string labels to an alphabet $T_i$7 with $T_i$8; $T_i$9, which marks whether a node is the rightmost child of its parent; $T_i$0, which marks leaves; and $T_i$1, a compacted list of leaf-ID sets indexed by $T_i$2 (Tabei, 18 Aug 2025).

Construction proceeds in four stated steps. First, a DFS is run on $T_i$3 with children sorted lexicographically. Second, the arrays $T_i$4, $T_i$5, $T_i$6, and $T_i$7 are recorded. Third, the sequence is stably sorted by the ancestor-string key $T_i$8, which is used only during build. Fourth, a wavelet matrix is built on $T_i$9 for rank/select over $i$0, and rank/select dictionaries are built on the bitvectors (Tabei, 18 Aug 2025).

The stated space complexity is

$i$1

plus low-order overhead. The emphasis on wavelet matrices and rank/select dictionaries shows that the index is designed for navigation and search rather than passive storage. It is therefore not merely a succinct encoding of the merged tree; it is a searchable representation whose primitives are directly exposed.

The key operations include Children(i), RankedChild(i,k), CharRankedChild(i,c,k), Parent(i), TreeIDs(i), and SubPathSearch(P). Except where noted, these operations run in $i$2 time, while TreeIDs(i) for leaves is $i$3 via rank on $i$4 (Tabei, 18 Aug 2025). SubPathSearch(P) is especially significant: for a path

$i$5

it returns a contiguous range $i$6 of all positions reachable by $i$7 from any root. The method uses a small “first-occur” array $i$8 to locate the region where nodes whose parent label is $i$9 begin, then refines the interval by rank/select steps over $\{T_i\}_{i=1}^N$0. A plausible implication is that the contiguous-range property is what allows the later search algorithm to aggregate path matches without scanning the entire structure.

4. Three-step search procedure

For a query tree $\{T_i\}_{i=1}^N$1 with $\{T_i\}_{i=1}^N$2 root-to-leaf paths $\{T_i\}_{i=1}^N$3, jXBW decomposes substructure search into three stages: path decomposition and matching, ancestor computation, and adaptive tree-ID collection (Tabei, 18 Aug 2025).

In the first stage, all root-to-leaf label sequences of $\{T_i\}_{i=1}^N$4 are extracted: $\{T_i\}_{i=1}^N$5 For each $\{T_i\}_{i=1}^N$6, the algorithm calls

$\{T_i\}_{i=1}^N$7

which costs $\{T_i\}_{i=1}^N$8, and gathers the resulting ranges $\{T_i\}_{i=1}^N$9. This converts a subtree query into a set of indexed path queries over the merged tree (Tabei, 18 Aug 2025).

In the second stage, each matching leaf position $MT$0 is treated as an occurrence of path $MT$1, and the algorithm ascends exactly $MT$2 times using Parent. The computed set

$MT$3

produces ancestor positions $MT$4 for each path. Their intersection

$MT$5

defines candidate subtree-root positions containing all query paths. This stage is the mechanism by which independent path hits are reassembled into candidate subtree matches (Tabei, 18 Aug 2025).

The third stage adapts to JSON structure. For each root candidate $MT$6, if $MT$7 contains JSON arrays, the method invokes StructMatch(r,Q), which enforces ordering via repeated CharRankedChild calls and yields sets of leaf IDs. Otherwise, it invokes CollectPathMatchingIDs(r,\{P_j\}): from each $MT$8, the algorithm walks down from $MT$9 using CharRankedChild, collects leaf positions, unions their TreeIDs, and then intersects the $\ell$0 ID sets for that root. The final answer is obtained by taking the union over all candidate roots (Tabei, 18 Aug 2025).

This design clarifies the role of arrays. Ordering is not globally imposed on all matching, because objects are represented as unordered children while arrays are represented as ordered children. The adaptive branch therefore prevents unnecessary structural checking when the query does not require ordered semantics, while still preserving correctness for array-containing patterns.

5. Complexity profile

The preprocessing costs are separated into merged-tree construction and jXBW construction. The merged-tree build takes $\ell$1 time and $\ell$2 space. jXBW construction then takes $\ell$3 time and space, reflecting the cost of the wavelet matrix and bitvectors (Tabei, 18 Aug 2025).

For substructure search, the analysis uses the following parameters: $\ell$4, the number of query paths; $\ell$5, the average path length; $\ell$6, the total number of leaf hits; $\ell$7, the candidate root count; and $\ell$8, the average branching factor for structural match. The stated time bound is

$\ell$9

equivalently

$MT$0

The contrast baseline is a naive tree scan on $MT$1, which costs

$MT$2

This establishes that jXBW’s asymptotic advantage comes from restricting work to path-hit intervals, ancestor backtracking, and candidate-root refinement rather than scanning all nodes against all query structure (Tabei, 18 Aug 2025).

The complexity expression also makes explicit that search cost is shaped by query decomposition and data-dependent hit counts. This suggests that performance will depend not only on corpus size but on how selective the decomposed paths are and on how many candidate roots survive the ancestor intersection step.

6. Empirical evaluation and reported use cases

The evaluation uses seven real-world JSONL datasets ranging from 36 K to 6.6 M objects, with depths 2–6 and with some datasets containing 0–100% arrays. The reported baselines are Ptree, a pointer-based merged tree; SucTree, a LOUDS-based succinct tree; and Saxon, a JSON→XML + XQuery pipeline (Tabei, 18 Aug 2025).

Two datasets are summarized numerically in the provided results. On movies with 36 K objects, average query time is reported as 0.016 ms/query for jXBW, compared with 0.265 for Ptree, 2.05 for SucTree, and 3.96 for Saxon. On pubchem with 1 M objects, average query time is 0.043 for jXBW, compared with 26.45 for Ptree and 205.19 for SucTree; Saxon did not finish in 24 h, corresponding to a reported gap of more than $MT$3 (Tabei, 18 Aug 2025). These figures instantiate the broader summary claim that jXBW achieves speedups of $MT$4 for smaller datasets and up to $MT$5 for larger datasets over tree-based approaches, while exceeding XML-based processing by more than $MT$6.

Dataset	Average query time (ms/query)	Memory usage (MB)
movies (36 K objs)	jXBW 0.016; Ptree 0.265; SucTree 2.05; Saxon 3.96	jXBW 56; Ptree 66; SucTree 50; Saxon 329
pubchem (1 M objs)	jXBW 0.043; Ptree 26.45; SucTree 205.19; Saxon $MT$7 h	jXBW 3 880; Ptree 4 702; SucTree 3 453; Saxon 64 840

The reported memory figures indicate competitive usage rather than universal minimization. On movies, jXBW uses 56 MB, compared with 66 for Ptree, 50 for SucTree, and 329 for Saxon. On pubchem, jXBW uses 3 880 MB, compared with 4 702 for Ptree, 3 453 for SucTree, and 64 840 for Saxon (Tabei, 18 Aug 2025). This suggests that the method’s primary contribution is latency reduction with memory remaining in the same broad regime as succinct tree alternatives, rather than uncompromising minimization of footprint.

Index construction time is described as dominated by tree merging, consistent with the $MT$8 bound. The paper further states that jXBW’s build is comparable to Ptree and SucTree and within a constant factor of Saxon’s XML pipeline (Tabei, 18 Aug 2025). This matters because the reported speedups are not attributed to an unrealistic preprocessing budget.

The application discussion connects jXBW to prompt engineering and structured RAG, where JSON-native retrieval can return only the JSON documents containing a given subtree pattern in sub-millisecond time. A concrete scientific case study is given for PubChem: querying for $MT$9-containing substructures took 21 ms with jXBW, versus 145 ms for Ptree and 335 ms for SucTree, and the resulting compounds were immediately fed into GPT-4 for downstream analysis, described as a symbolic–neural workflow (Tabei, 18 Aug 2025). Proposed extensions include approximate matching by edit distance, support for other semi-structured formats such as Avro and Protobuf, and distributed jXBW for corpora larger than 100 GB. These extensions are not reported as completed features; they are forward-looking directions that indicate where the method could be generalized.