LPDS: Language Processing Data Structure

Updated 26 November 2025

LPDS is a formal data structure that organizes linguistic information through standardized folder hierarchies, relational schemas, grammar-compressed indexes, and symbolic graphs.
It ensures reproducibility and extensibility by enforcing strict naming conventions, metadata provenance, and normalization across diverse language processing workflows.
LPDS supports efficient querying and data compression, enabling scalable annotation and rapid pattern search through optimized, grammar-based indexing methods.

The Language Processing Data Structure (LPDS) is a formal systematization of linguistic data organization and annotation, designed to ensure transparency, extensibility, and methodological rigor in language technology workflows. LPDS exists in several distinct research traditions, spanning standards for folder hierarchies and file names, extensible relational annotation schemas, grammar-compressed index structures, and symbolic representations for natural language understanding and inference. Each instantiation addresses critical bottlenecks in linguistic data management, large-scale annotation, compressive representation, and the interface between information encoding and processing.

1. Formal Definitions Across Paradigms

LPDS is not a monolithic standard but a term that recurs in several influential approaches:

Standardized Data Hierarchies: Modern LPDS is defined as a BIDS-inspired, directory- and entity-driven structure for linguistic data projects. It mandates a fixed folder hierarchy (project → participants → [sessions] → tasks), metadata files at standard locations, and controlled file naming conventions using ordered key–value pairs (entities) and defined suffixes (Pauli et al., 19 Nov 2025).
Relational Annotation Models: In computational annotation, LPDS refers to a set-theoretic and normalized relational schema. Documents ( $D$ ), sentences ( $S$ ), tokens ( $T$ ), and annotation layers (dependencies, entities, coreference, word embeddings) are formalized as sets and functions, yielding normalized tables with composite primary and foreign keys (Arnold, 2017).
Grammar-compressed Index Structures: In string processing, LPDS designates the data structure induced by Lyndon straight-line programs (SLPs), offering a grammar-compressed self-index with $O(g)$ space and near-linear pattern search based on unique factorization properties of Lyndon words (Tsuruta et al., 2020).
Symbolic Processing and "Chunk"-based Models: A third formalization grounds LPDS in graph/forest memory models where lexical "chunks"—classified as data, structure, pointer, and task—encode both linguistic information and the procedures for its processing, linking concepts by hierarchical and associative relations (Zhang, 2020).

2. Directory Schema, Entity Naming, and Configuration (Standardization Perspective)

The LPDS project structure is defined as follows (Pauli et al., 19 Nov 2025):

Hierarchy:

<root>/
  ├── dataset_description.json   # required metadata
  ├── participants/             # required (one per participant)
  │   └── part-<id>/
  │       └── [ses-<id>/]      # optional sessions
  │           └── <task>/
  │               └── data files
  └── config.yml                # reproducibility config

File Naming: Files are named using regular expressions enforcing key-value entity order, e.g.,
1 2
part-01_ses-01_task-interview_proc-cleaned_text.txt part-02_task-fluency_metric-embeddings_model-bert.csv
Valid entities are: part, ses, task, cat, acq, run, proc, metric, model, group, param. Suffixes (such as recording, transcript, embeddings) dictate file content and extension (.wav, .txt, .csv, etc.).
Configuration: A single YAML or JSON config file governs workflow, specifying pipeline name, LPDS version, data sources, preprocessing and feature extraction parameters, and logging (see full schema in (Pauli et al., 19 Nov 2025)).
Provenance and Metadata: Each folder includes JSON sidecars (e.g., dataset_description.json) for reproducibility and compliance. Additional files, such as participants.tsv and README.md, provide project- or subject-level details.

3. Relational Annotation and Integration Model

Within corpus annotation, LPDS is realized as a normalized, relational schema (Arnold, 2017):

Core Tables:
- document: id, time, language, etc.
- sentence: (id, sid), sentiment, fk ← document(id)
- token: (id, sid, tid), word, lemma, upos, pos, char offset, fk ← sentence, document
- dependency: (id, sid, tid, tid_target), relation, fk ← token
- entity: (id, sid, tid, tid_end), entity_type, entity, fk ← token
- coreference: (id, rid, mid), mention, type, fk ← token
- vector: (id, sid, tid), k-dimensional float vector, fk ← token
Annotation Layer Mapping: All annotation tables are linked by composite keys, supporting efficient and lossless joins. Token-level queries do not require loading higher-level structures, optimizing scalability and selective data access.
Normalization Principles: The schema adheres to Codd’s 3NF with justified denormalizations. This ensures atomicity, reduces update anomalies, and facilitates extensibility—new annotations (e.g., semantic role labeling) are added as new tables keyed by existing composite identifiers.
Usage Example:

# Load dependencies table and filter for direct objects
deps %>%
  filter(relation == "dobj") %>%
  select(doc_id = id, sent_id = sid, governor = word, dependent = lemma_target)

4. Grammar-compressed LPDS for String Indexing

LPDS has a specialized interpretation as a grammar-compressed self-index based on Lyndon SLPs (Tsuruta et al., 2020):

Lyndon SLP: For a string $T \in \Sigma^+$ made into a Lyndon word, the Lyndon SLP is a Chomsky-normal-form grammar where each production’s right-hand side is the standard factorization of a Lyndon word, and all production strings are unique.
Self-Index Architecture:
- The core structure stores the grammar ( $g$ rules), random-access and fingerprint data structures, two z-fast tries for prefix/suffix range-queries, and a labeled binary-relation structure for rapid tuple retrieval.
- Total space is $O(g)$ words.
Construction and Query:
- Lyndon tree computation: $O(n)$ .
- SLP and index structure: $O(n \log n)$ expected time.
- Pattern search (Locate( $P$ )): $O(m + \log m \log n + occ \log g)$ .
- Uniquely, the partitioning of $P$ during search is limited to $O(\log m)$ positions, due to the logarithmic bound on significant suffixes—a gain over earlier SLP-index schemes.
Comparison Table:

| Index | Space | Construction | Locate-time | |:------------------------------------------|:----------:|:----------------:|:-----------------------------------------:| | Claude & Navarro ’12 (SLP-index) | $O(g)$ | $O(n + g \log g)$ | $O(m^2 \log\log_g n + (m+occ)\log g)$ | | Christiansen et al. ’18 (attractor-based) | $O(\gamma \log (n/\gamma))$ | $O(n \log n)$ | $O(m+occ)$ (optimal) | | This paper (Lyndon SLP) | $O(g)$ | $O(n \log n)$ | $O(m + \log m \log n + occ \log g)$ |

Here $g$ = Lyndon SLP size, $\gamma$ = minimal attractor size.

5. Symbolic and Graph-based LPDS for Natural Language Understanding

In symbolic approaches, LPDS is an architecture built on chunk classifications and graph structures (Zhang, 2020):

Primitive Chunk Types:
- Data Chunks ( $D$ ): Attributes ( $A$ ), attribute-space, entity, measuring, and verbal chunks.
- Structure Chunks ( $\mathcal{R}$ ): Encode relations (e.g., "Be", "Have", "’s", "Of").
- Pointer Chunks ( $\mathcal{P}$ ): Instantiation, scope, and positioning pointers (e.g., "the", "who", "at").
- Task Chunks ( $\mathcal{T}$ ): Sentences specifying processing actions (description, verification, search).
Information Architecture:
- Memory-tree: Entities are organized hierarchically as a forest (partial order $\subseteq$ ), with nodes as entity chunks and edges as subclass/superclass relations.
- Memory-graph: Augments the hierarchy with virtual (associative) links reflecting co-occurrence or shared properties; supports both set-theoretic and associative retrieval.
- Encoding Modes: Direct encoding (atomic elements), cluster/measuring (subsets or distributions), and change-feature (verb as attribute-sequence abstraction).
Processing Procedures:
- Encoded both in structure and pointer chunks (direct reading/processing rules) and at the sentence level, where chunk combinations compile to a logical form specifying database operations and updates.
- Example: Parsing and executing dialogue acts (e.g., "Do we have any apple?" → logical form: Verify(Have(we, apple) ∧ Count(apple) > 0)).

6. Extensibility, Reproducibility, and Current Limitations

Extensibility: LPDS folder schema admits extensions for multi-modal data via new entities and suffixes; relational models can add annotation tables keyed on composite IDs (Pauli et al., 19 Nov 2025, Arnold, 2017).
Reproducibility: Workflows specify all preprocessing, feature extraction, and aggregation steps in a machine-actionable config. Validation of file naming, directory structure, and version control is rigorously enforced (Pauli et al., 19 Nov 2025).
Interoperability: All layers (folder, file, relational, and symbolic) yield datasets that can be queried programmatically or read by third-party tools without human intervention. Derivative files retain full provenance by encoding all processing parameters and versioning in sidecar JSONs (Pauli et al., 19 Nov 2025).
Limitations:
- Legacy datasets may require manual restructuring to fit LPDS.
- Not all NLP subfields are covered by the reference toolchains (e.g., highly exotic or multi-modal forms).
- Model-dependent outputs may reflect inherent biases or version drift.
- Scalability on large datasets can be limited by hardware constraints and requires careful parameter validation (Pauli et al., 19 Nov 2025).

7. Comparative Summary Table

Paradigm	LPDS Focus	Key Formal Properties
BIDS-inspired standard	Folder, file, config for project and workflow	Directory depth, key-value entities, regex
Relational annotation	Normalized tables per annotation layer	Composite PK/FK, set-theoretic mapping
Grammar-compressed SLP	Efficient, compressed self-index for string data	$O(g)$ space, $O(n \log n)$ build, queries
Symbolic/graph-based	Chunk graph for information/procedure separation	Typed chunk sets, tree/graph, logical form

This diversity in LPDS formalisms reflects the heterogeneity of problems in language processing, from storage/Ontology to annotation, compression, and symbolic inference. The unifying feature is a commitment to principled organization, extensibility, and reproducibility in linguistic data analysis (Pauli et al., 19 Nov 2025, Arnold, 2017, Tsuruta et al., 2020, Zhang, 2020).