Shared Data Layer (SDL) Overview

• Shared Data Layer (SDL) is a unified data substrate that centralizes storage, metadata, and access coordination across multiple engines without replicating data.
• It supports multi-domain applications such as enterprise analytics, self-driving laboratories, and O-RAN by reducing data duplication and enabling vendor-neutral interoperability.
• Trade-offs include increases in query latency and concentrated security challenges, necessitating enhanced access controls and anomaly detection measures.

A Shared Data Layer (SDL) is a logically single, physically shared data substrate that multiple services or engines can use without each one owning and copying its own database (Miyamoto et al., 3 Dec 2025). Across the systems described in recent arXiv work, the term denotes a common architectural pattern rather than a single implementation style: a shared lakehouse storage layer for enterprise analytics, a central FAIR repository for collaborative self-driving laboratories, a semi-standardised shared store inside the O-RAN Near-RT RIC, and a software-distributed shared memory for heterogeneous machines (Miyamoto et al., 3 Dec 2025, Deucher et al., 24 Jun 2025, Dayaratne et al., 11 Aug 2025, Cudennec, 2020). The unifying idea is that storage, metadata, and access coordination are centralized logically, while compute, query, or control functions remain decoupled.

1. Definitional scope and common architectural properties

The canonical goals associated with an SDL are a single logical data layer shared by multiple engines, avoidance of data duplication and silos, decoupling of compute or query engines from storage, and vendor-neutral or multi-engine interoperability (Miyamoto et al., 3 Dec 2025). In enterprise analytics, these goals are expressed through the “Write-Once, Read-Anywhere” principle and the supporting principles of Single Source of Truth, Federated In-Place Access, and Layer Independence (Miyamoto et al., 3 Dec 2025). In self-driving laboratories, the same pattern appears as a central shared database into which geographically dispersed contributors write standardized records and from which machine-learning services retrain and serve new experimental suggestions (Deucher et al., 24 Jun 2025). In O-RAN, the SDL is simultaneously a data abstraction layer and a shared message bus / database over which xApps coordinate and may interfere with each other if access control is weak (Dayaratne et al., 11 Aug 2025). In distributed heterogeneous systems, the SDL perspective appears as a middleware layer that aggregates independent local memories into a global logical address space while hiding data placement, replication, and movement (Cudennec, 2020).

Context	Shared substrate	Distinctive property
Enterprise analytics	Iceberg tables on object storage	“Write-Once, Read-Anywhere”
Collaborative self-driving labs	ResultsDB	FAIR data and cloud-hosted Sim2L workflows
O-RAN Near-RT RIC	Semi-standardised SDL	Shared message bus / database for xApps
Heterogeneous machines	SAT S-DSM	Global logical address space with chunk-based coherence

These realizations differ in physical organization and workload assumptions. EDSP is specialized for analytical and data-science workloads with seconds-to-minutes latency expectations rather than ultra-low-latency OLTP (Miyamoto et al., 3 Dec 2025). The collaborative digital twin uses thin clients and central cloud services, so storage, processing, and machine learning are performed centrally rather than locally (Deucher et al., 24 Jun 2025). O-RAN uses the SDL as a shared operational substrate for control-plane signaling inside the Near-RT RIC (Dayaratne et al., 11 Aug 2025). SAT, by contrast, exposes shared-memory style access over distributed nodes using explicit consistency scopes (Cudennec, 2020). This suggests that SDL is best understood as an architectural abstraction whose concrete realization depends on access pattern, latency budget, and trust model.

2. Enterprise analytics and the lakehouse-based SDL

The most explicit realization of SDL in the provided corpus is the Enterprise Data Science Platform (EDSP), which is described as a concrete realization of a Shared Data Layer for analytics and data-science workloads, built on top of an open lakehouse storage layer (Miyamoto et al., 3 Dec 2025). EDSP organizes the system into four layers: Data Preparation, Data Store, Access Interface, and Query Engines. Data flows in one direction from sources to preparation to store to engines, while access flows from engines into the shared store (Miyamoto et al., 3 Dec 2025).

The Data Preparation Layer ingests raw data from upstream systems, standardizes schemas, transforms raw inputs into analysis-ready tabular structures, and writes data into Iceberg tables in object storage using a fixed table path per dataset. The Data Store Layer is the physical and logical core of the SDL: canonical datasets are stored in cloud object storage, Apache Iceberg provides ACID transactions, time travel, snapshot-based reads, and schema evolution, and metadata files are stored alongside data in object storage (Miyamoto et al., 3 Dec 2025). EDSP adds a fixed-name pointer file, latest.metadata.json, which always references the current Iceberg metadata file, allowing BigQuery and Snowflake external tables to bind to a stable path while the underlying Iceberg metadata advances commit by commit (Miyamoto et al., 3 Dec 2025).

The Access Interface Layer presents a unified, engine-agnostic view of datasets and their access methods through a web-based documentation/catalog platform. For each dataset it contains schema, description, update cadence, Snowflake external table DDL referencing the Iceberg pointer, BigQuery BigLake external-table configuration, and Python/PyIceberg code examples (Miyamoto et al., 3 Dec 2025). The Query Engine Layer then attaches Snowflake, BigQuery, and Python environments to the same canonical data. BigQuery uses BigLake external tables to read Iceberg, Snowflake uses Iceberg external table features, and Python environments use PyIceberg to read Iceberg tables directly into DataFrames with predicate pushdown (Miyamoto et al., 3 Dec 2025).

The duplication argument is central. For $n$ datasets and $m$ query engines, traditional warehouse-centric patterns can produce up to $n \times m$ durable replicas, whereas EDSP aims to bound durable replicas by $n$:

$$\text{traditional replicas} \le n \times m,\quad \text{EDSP replicas} \le n.$$

Operationally, the paper reports a 33–44% reduction in operational steps for data sharing across different analytical environments compared with conventional copy-based approaches requiring data migration. The example counts are 9 steps versus 5 steps for Source = BigQuery, Engine = Snowflake, and 9 steps versus 6 steps for Source = Snowflake, Engine = BigQuery (Miyamoto et al., 3 Dec 2025). The trade-off is a query-latency increase of up to a factor of 2.6 compared with native tables, although all p95 latencies remain below 10 seconds and end-to-end completion times remain on the order of seconds (Miyamoto et al., 3 Dec 2025). A common misconception is that an SDL must be a single federated query frontend; EDSP explicitly contrasts its design with federated query engines by allowing multiple engines to attach directly to a common shared store as peers (Miyamoto et al., 3 Dec 2025).

3. FAIR repositories and collaborative self-driving laboratories

In the nanoHUB-based collaborative digital twin, the shared data layer is ResultsDB, a central, cloud-hosted, multi-tenant FAIR data repository that records every Sim2L execution as a typed, queryable record (Deucher et al., 24 Jun 2025). Local, geographically distributed SDLs are intentionally thin: each lab is a human-plus-simple-hardware setup using food dyes, cups, a printable template, and a camera, with no local servers or ML infrastructure required (Deucher et al., 24 Jun 2025). Data submission occurs through the HueLogic “Contribute Data” GUI, which triggers a nanoHUB Sim2L workflow that encapsulates image-processing and validation logic, defines input/output schema, and automatically writes all inputs and outputs into ResultsDB (Deucher et al., 24 Jun 2025).

The ingestion path is standardized. A user performs dye mixing locally, captures an image using a standard template with embedded ArUco markers, uploads the image, recipe, and metadata through the GUI, and invokes the HueLogic Sim2L. The workflow detects four ArUco markers, computes a geometric transform, identifies a central Region of Interest, extracts it, computes average RGB using OpenCV, and stores both inputs and outputs in ResultsDB together with Sim2L ID and version, execution timestamp, and user identity (Deucher et al., 24 Jun 2025). This produces a minimal provenance model in which each experiment is a function evaluation of the form

$$\text{Sim2L}_\text{HueLogic}:\ \left( x, \text{image}, \text{metadata} \right) \mapsto y,$$

where $x$ is the recipe vector and $y$ is the derived RGB color (Deucher et al., 24 Jun 2025).

Machine learning consumes the SDL directly. The HueLogic “Evaluate Model” GUI queries ResultsDB for all relevant records, retrains three Gaussian Process Regressors on demand, and returns both an “optimal recipe” and an “exploration recipe” using active learning (Deucher et al., 24 Jun 2025). Because training uses all available data in the shared repository, pooled data can benefit contributors with different optimization objectives. The paper reports that sharing data either matches or outperforms individual efforts for all target colors in the four-scientist simulation, with Cavaliers Red showing significant improvement from sharing due to overlap with Giants Orange in the explored recipe space (Deucher et al., 24 Jun 2025).

The SDL function here is not merely persistence. It makes workflows findable through unique dataset identity, accessible through web interfaces and standard APIs, interoperable because Sim2L declares machine-readable inputs and outputs, and reusable because every input and output is logged and the underlying code for both GUIs, the Sim2L workflow, and the dataset are available online (Deucher et al., 24 Jun 2025). This establishes an SDL that is simultaneously a data repository, a workflow provenance layer, and a retraining substrate for collaborative active learning.

4. Security, trust boundaries, and SDL manipulation in O-RAN

The O-RAN paper treats the SDL as a semi-standardised data store within the O-RAN Near-RT RIC that is used by xApps to exchange and persist information, especially control-plane protocol messages (Dayaratne et al., 11 Aug 2025). Its under-standardised status is identified explicitly as an attack surface. Multiple xApps read from and write to the same SDL; anomaly detection xApps retrieve RRC/NAS messages together with associated TMSI and RNTI identifiers from it; and a malicious xApp with read/write access can manipulate those records before other xApps consume them (Dayaratne et al., 11 Aug 2025).

The attack described is Unicode hypoglyphing. Visually similar characters with different Unicode code points are inserted into message names, so a string such as RRCSetupRequest is modified into a visually identical but programmatically distinct variant (Dayaratne et al., 11 Aug 2025). In the experiments, 5 messages were manipulated out of 1,016 total messages, including 2 Blind DoS attack instances and 3 normal messages (Dayaratne et al., 11 Aug 2025). Traditional AutoEncoder-based anomaly detection xApps from the 6G-XSec setting, which rely on fixed vocabularies or feature encodings learned from well-formed messages, crash when the first hypoglyphed message appears. The paper reports that all 10 traditional ML-based AutoEncoder models experienced an immediate failure/crash upon the encounter of the first hypoglyphed message in the test set (Dayaratne et al., 11 Aug 2025).

The proposed countermeasure is an LLM-based anomaly detection xApp built with Llama-3.1-8B-Instruct, operating in zero-shot mode with temperature set to 0 and deployed via vLLM on an NVIDIA A100 80GB GPU (Dayaratne et al., 11 Aug 2025). Its pipeline includes an SDL Access Logic module, a Detection Prompt Constructor module, and an LLM-based classifier. It processes sliding windows of recent SDL messages per UE, constructs prompts from message sequences and identifiers, and outputs “Normal” or “Anomalous” labels (Dayaratne et al., 11 Aug 2025). Unlike the AutoEncoder-based detectors, the LLM-based xApp successfully processed every single message in the dataset, including all Unicode-wise manipulated instances, without any system crashes or early terminations (Dayaratne et al., 11 Aug 2025). The trade-off is lower detection quality in the initial configuration: the AutoEncoder baseline achieves F1 scores above 75% for most window sizes on clean data, whereas the LLM-based xApp yields F1 scores in the range 0.087–0.319 under hypoglyph manipulation, though with average detection latency under 0.07 seconds and therefore within the Near-RT RIC’s 1-second constraint (Dayaratne et al., 11 Aug 2025).

The SDL is therefore not merely a data-sharing convenience; it is a critical security boundary. The paper’s design implications are stronger access control for SDL writes, Unicode normalization and script-mixing checks at SDL boundaries, SDL-level monitoring xApps, and hybrid defensive arrangements in which robust detectors complement higher-accuracy models that assume validated inputs (Dayaratne et al., 11 Aug 2025). A common misconception is that consolidation reduces only storage or ETL complexity. In shared control planes, consolidation also concentrates attack surface.

5. Consistency, coherence, and the distributed shared-memory interpretation

A more systems-level interpretation appears in SAT (Share Among Things), a user-level software-distributed shared memory designed for distributed heterogeneous microservers and implemented atop MPI (Cudennec, 2020). From an SDL perspective, SAT aggregates independent local memories into a global logical address space, hides data placement and replication, and supports chunk-based coherence with pluggable consistency protocols (Cudennec, 2020).

SAT’s core abstraction is the chunk. Large shared objects are automatically decomposed into chunks, which become the atomic units for allocation, metadata, coherence, and placement (Cudennec, 2020). Each chunk has a logical identifier in a global unsigned long space, a size, a home node, and protocol state. The default mapping of a chunk to its home server is

$home\_server(chunkid) = chunkid \bmod |servers|.$

Access is governed by scope consistency. Applications acquire shared data with [READ](https://www.emergentmind.com/topics/reconstruction-and-alignment-of-text-descriptions-read), WRITE, or READWRITE and terminate the scope with RELEASE; outside the scope, consistency is not guaranteed (Cudennec, 2020). The report characterizes this as weaker than strict sequential consistency but more efficient, with writes becoming visible at scope boundaries (Cudennec, 2020).

The runtime uses a semi-structured super-peer topology. Servers form a peer-to-peer overlay and manage coherence automata and metadata, while clients run user tasks and expose the S-DSM API (Cudennec, 2020). Coherence is multi-protocol: different chunks may use different protocols, with home-based MESI as the default (Cudennec, 2020). SAT also integrates an event-based model directly into the shared layer through publish–subscribe. A client may SUBSCRIBE to a chunk, and every modification of that chunk triggers a subscriber handler when coherence events propagate (Cudennec, 2020). This hybrid model combines shared-memory style access with event-driven coordination and is used in the report’s video-processing example to implement channels and implicit scheduling (Cudennec, 2020).

The design extends SDL beyond databases and object stores. In SAT, the shared layer is an address space plus coherence machinery rather than a repository of tables or records. It also explicitly optimizes the runtime substrate itself: the micro-sleep mechanism replaces aggressive polling with adaptive clock_nanosleep intervals to reduce energy consumption while controlling latency (Cudennec, 2020). This broadens the SDL concept from “shared data store” to “shared data abstraction with consistency and event semantics.”

6. Terminological ambiguity and related concepts

The acronym SDL is not semantically unique in the research literature. In "Representing Extended Finite State Machines for SDL by A Novel Control Model of Discrete Event Systems," SDL means Specification and Description Language, a formal language used in telecommunications and reactive/distributed systems to specify communicating processes (Wang et al., 2016). There, each SDL process is described by an EFSM, predicates induce pairs of conflicting transitions, and the central contribution is a transformation from EFSM$_{\mathrm{SDL}}$ to a plant-plus-supervisor representation in discrete-event supervisory control (Wang et al., 2016). This usage is distinct from Shared Data Layer.

The distinction matters because O-RAN, telecom systems, and self-driving laboratories all use the acronym SDL, but not always with the same referent. A reader encountering “SDL flow graphs” or “EFSM for SDL” is reading about Specification and Description Language, not about a shared storage substrate (Wang et al., 2016). Conversely, in EDSP, O-RAN, ResultsDB, and SAT, SDL refers to shared storage, shared metadata, or shared-memory abstractions (Miyamoto et al., 3 Dec 2025, Dayaratne et al., 11 Aug 2025, Deucher et al., 24 Jun 2025, Cudennec, 2020).

Several adjacent concepts should also be distinguished. A traditional warehouse emphasizes native-table performance but can produce $m$0 copies when $m$1 datasets must serve $m$2 environments, whereas the EDSP-style SDL uses one canonical table per dataset with external-table or direct file access (Miyamoto et al., 3 Dec 2025). A lakehouse deployment may still anchor on a single engine and therefore not solve cross-engine symmetry; EDSP explicitly designs for BigQuery, Snowflake, and Python as peers (Miyamoto et al., 3 Dec 2025). A federated query engine centralizes queries from one frontend into many sources, whereas EDSP inverts that pattern by allowing multiple engines to attach directly to a common shared store (Miyamoto et al., 3 Dec 2025). In self-driving laboratories, the SDL is coupled tightly to workflow provenance and retraining rather than only query interoperability (Deucher et al., 24 Jun 2025). In O-RAN, it is inseparable from authorization, validation, and resilience against adversarial input manipulation (Dayaratne et al., 11 Aug 2025).

Taken together, these works show that Shared Data Layer is a cross-domain architectural pattern defined by shared logical authority over data, stable access interfaces, and decoupling between data substrate and consuming engines. The practical consequences depend on the domain: reduction of data duplication and vendor lock-in in enterprise analytics, FAIR collaboration and pooled learning in self-driving laboratories, concentrated attack surface and robustness requirements in O-RAN, and explicit consistency/coherence management in distributed heterogeneous machines (Miyamoto et al., 3 Dec 2025, Deucher et al., 24 Jun 2025, Dayaratne et al., 11 Aug 2025, Cudennec, 2020).