Digital Lab Book (DLB): Provenance & Research Record

• Digital Lab Book (DLB) is a software system that unifies experimental documentation, data capture, and provenance into a tamper-evident digital record.
• DLB implementations use diverse architectures—from web-based to serverless—to automate metadata capture, ensure audit trails, and support regulatory compliance.
• Practical impacts include enhanced reproducibility, improved data traceability, and significant efficiency gains in managing complex research workflows.

Digital Lab Book (DLB), also known as an Electronic Laboratory Notebook (ELN), denotes a software system that replaces or augments the traditional bound laboratory notebook and captures experiments, observations, data files, analyses, and metadata in a secure, time-stamped, and tamper-evident manner (Ng et al., 2023). In the current literature, the term encompasses regulated ELNs, provenance-aware scientific data management systems, interactive notebooks extended for robotics, browser-only notebooks for high-security environments, and virtual laboratories for computational experiments; across these variants, the central objective is to treat documentation, data, and analytic context as a single research record rather than as disconnected artifacts (Garcia, 2024).

1. Historical transition and conceptual scope

The modern DLB emerged from a tension between established laboratory practice and the scale of digital research. Physical, permanent notebooks remain the “gold standard” of record-keeping for auditing, yet advances in technology produced large quantities of electronic records, making it “virtually impossible to maintain a full set of records in physical notebooks” (Ng et al., 2023). In parallel, computerized research systems made acquired data more elaborate, increasing the need for electronic notebooks with data storage, computational features, and reliable electronic documentation under the rubric of good laboratory practice (Ney et al., 2011).

The scope of the DLB broadened further as disciplines encountered domain-specific pressures. In chemistry teaching and research data management, handwritten notebooks and PDF reports were described as inaccessible, non-machine-readable, and limiting reuse and machine learning applications; the Chemotion-based implementation at RWTH Aachen therefore framed the DLB as a web-based system for planning, documenting, and evaluating experiments while capturing rich metadata and linking records to a repository (Hoffmann et al., 23 Sep 2025). In robotics, researchers were found to rely on diverse and often incompatible tools for managing experimental data, with resulting challenges in reproducibility and data traceability; the proposed response was not a generic notebook, but a specialized interactive notebook incorporating properties of a traditional lab notebook, including comprehensive data entry, continuous context capture, and agile data staging (Garcia, 2024).

A recurrent misconception is that a DLB is simply a digitized page or a file store. The literature does not support that narrow view. The surveyed systems combine structured metadata, provenance, logging, signatures, storage back ends, export mechanisms, and, in several cases, executable or replayable context. This suggests that the distinctive feature of a DLB is not merely electronic persistence, but the operational linkage between experimental actions, recorded data, contextual state, and later verification.

2. Data models, metadata schemas, and provenance formalisms

One explicit formalization defines each experiment as a record

$E = (C, D, M),$

where $C$ is context, $D$ is data, and $M$ is metadata (Garcia, 2024). In that formulation, context includes machine and environment state such as ROS nodes, hardware configuration, camera calibration, lighting, and floor-plan; data includes raw and processed streams such as joint angles, trajectories, images, videos, audio, and sensor logs; metadata includes human-entered fields such as date, experimenter, objective, and notes, as well as automatic fields such as timestamps, git hash, and parameter settings. The corresponding pseudocode schema extends this with versioned datasets and provenance lineage:

$M$1

The same robotics-oriented model proposes a metadata completeness check,

$$\mathrm{completeness}(M) = \frac{|fields_{filled}|}{|fields_{required}|},$$

and defines a run as “validated” when $\mathrm{completeness}(M) \ge \tau$ (Garcia, 2024). This is significant because it converts documentation quality from an informal expectation into an automatically testable condition.

Other systems organize the DLB around typed collections rather than notebook cells. In the DataFinder-based GLP framework, the repository model is customized to the five canonical phases of a non-clinical study—Preparation, Execution, Evaluation, Interpretation, and Archiving—with mandatory metadata fields enforced within each sub-collection (Ney et al., 2011). An Execution collection, for example, requires instrumentID, operator, dateTime, and protocolRef, while artifacts carry fields such as fileType, creationDate, and checksum. The use of typed collections instead of free-form folders is intended to guarantee a structured vocabulary for later GLP audits and automated report generation.

Provenance is modeled explicitly in multiple DLB lines of work. The DataFinder system adopts the Open Provenance Model, defining a graph

$G_{prov} = (V, E),$

with processes $P$, artifacts $A$, and agents $Ag$, and edges of the forms used, wasGeneratedBy, wasControlledBy, and wasDerivedFrom; the transitive ancestry of an artifact is then expressed as

$C$0

This allows lineage queries over a Neo4j-backed provenance store (Ney et al., 2011). Bernstetter et al. use a related but Git-centered abstraction for immersive virtual fieldwork, modeling the provenance repository as a directed acyclic graph

$C$1

whose nodes are commits $C$2, with each commit represented as

$C$3

where $C$4 is the Git hash, $C$5 the timestamp, $C$6 the message, $C$7 the screenshot image, and $C$8 the parent commits (Bernstetter et al., 24 Jul 2025). In computational experimentation, SCHEMA lab formalizes tasks as tuples $C$9 and experiments as named groupings of completed tasks or workflows, $D$0 (Adamidi et al., 1 Apr 2025). Across these formulations, provenance is not an auxiliary annotation; it is a first-class element of the DLB data model.

3. Architectural patterns and implementation frameworks

The DLB literature does not converge on a single software architecture. Instead, it exhibits several stable implementation patterns, ranging from pure client-side applications to MVC web systems, Git-backed provenance tools, and Kubernetes-orchestrated virtual laboratories.

System	Implementation pattern	Storage / provenance
CyNote (Ng et al., 2023)	Web2py MVC	Three SQLite databases; EventLog; Signature
DataFinder DLB (Ney et al., 2011)	Python client–server with pluggable persistence	WebDAV, Subversion, FTP, GridFTP, S3, local disk, MataNui; prOOst/Neo4j
“Lazy approach” ELN (Schubotz et al., 2022)	Django 3.x with five apps	Relational DB; file links; Jupyter integration
Solar Lab Notebook (Tsalaportas et al., 2014)	Pure client-side single-page application	XML files conforming to sln.xsd
Virtual fieldwork DLB (Bernstetter et al., 24 Jul 2025)	Web front end with local backend service	Native Git repositories of states and screenshots
SCHEMA lab (Adamidi et al., 1 Apr 2025)	Single-page web app plus REST API and TESK/Kubernetes	Relational metadata store and S3 buckets
Robotics interactive notebook (Garcia, 2024)	Extended Jupyter-style notebook	.ipynb plus sidecar SQLite or Git-LFS, TSDB, object store

CyNote exemplifies a classical ELN architecture based on the Web2py model–view–controller paradigm, with models defining the schema via the DAL, controllers mapped to URLs for business logic, and HTML views for rendering responses (Ng et al., 2023). The Django-based “lazy approach” instead decomposes functionality into five apps—Exp_Main, Exp_Sub, Analysis, Lab_Dash, and Lab_Misc—backed by a relational database and a web front end using django-tables2, django-filter, django-plotly-dash, Plotly/Dash, Bootstrap, and Jupyter integration (Schubotz et al., 2022). These frameworks support conventional web interactions, authentication, table views, dashboards, and export.

At the other end of the spectrum, Solar Lab Notebook (SLN) is intentionally serverless: all logic runs inside the browser, there is no server-side code, and XML load/save, image processing, and validation are performed with HTML5, CSS3, JavaScript, FileReader, Blob, Local Storage, and Canvas APIs (Tsalaportas et al., 2014). This architecture is specifically motivated by portability and by use in high-security or air-gapped laboratories.

Federated and distributed storage appears in the DataFinder line of work. Its persistence layer is fully pluggable, allowing the same logical repository to span WebDAV, Subversion, FTP, GridFTP, Amazon S3, local disk, and MataNui, while missing metadata for storage systems such as plain GridFTP can be pushed into a centrally indexed sidecar (Ney et al., 2011). The result is a DLB that treats heterogeneity of storage as a design assumption rather than an exception.

The newer computational and interactive systems bring orchestration and runtime capture to the foreground. In SCHEMA lab, the front end interacts with a RESTful API that authenticates requests, enforces quotas, persists metadata, and delegates execution to TESK, which schedules Docker containers on Kubernetes; inputs and outputs are stored in per-user S3 buckets (Adamidi et al., 1 Apr 2025). In the robotics notebook proposal, back ends may include extended Jupyter notebooks with embedded metadata, sidecar SQLite or Git-LFS for large binaries, time-series databases such as InfluxDB for high-frequency sensor streams, and object stores such as S3 or MinIO for media assets and calibration files, with integration points to ROS/ROS2, Docker/Kubernetes, and HDF5 or ROS-bag conversion utilities (Garcia, 2024).

4. Workflow construction and user interaction

DLB workflows vary according to whether the system prioritizes explicit authoring, automated ingestion, or automatic provenance capture. In the robotics notebook design, interaction begins with an Experiment object, continues with form-based entry for contextual fields such as lighting or floor maps, and then uses capture contexts such as capture_ros to bind ROS topics, sensor logs, parameters, plots, and resulting data to specific notebook code regions (Garcia, 2024). The same design introduces a “stage” area in which data are tentatively recorded and a “commit” step that moves them into a permanent, versioned store; parameter-sweep cells can generate multiple runs, each with its own metadata record. The stated traceability benefit is that each subsection of code is bound to its data, parameters, and plot, and scrolling upward reveals lineage.

CyNote follows a more classical notebook-entry workflow. A researcher logs in, creates a notebook, creates entries, and triggers controller logic that persists the Entry record and appends a corresponding EventLog entry. Finalization occurs through explicit notarization, after which the record may be exported as PDF or XML with timestamps, authors, and hashes suitable for regulatory submission or long-term archiving (Ng et al., 2023). This workflow foregrounds auditability and formal closure of records rather than continuous runtime capture.

The Django-based “lazy approach,” described by Schubotz et al., shifts effort from manual entry to file-system convention. Users place measurement files into a prescribed folder hierarchy encoding experiment stream, experiment code, date, sample ID, and time stamp; a single “Generate entries” action walks the tree, applies extension and file-age filters, extracts metadata, resolves foreign-key links to experiment and sample models, and inserts or updates database rows (Schubotz et al., 2022). The generated web book then exposes table views, detail views, interactive plots, and linked sub-experiments. No manual data-entry is needed beyond placing the file, although users may add or override metadata later.

Bernstetter et al. describe a still more automated interaction style for immersive virtual fieldwork. Every placement or removal of a measurement inside the Virtual Fieldwork Tool triggers the DLB backend to capture a screenshot, stage metadata such as timestamp and action type, and create a Git commit in the active provenance repository; users can then branch, annotate, restore, or re-run past states through the DLB interface (Bernstetter et al., 24 Jul 2025). The system also provides a mind-map view that supports free-form grouping and annotation of commits.

In chemistry education, Hoffmann et al. use Chemotion ELN as a DLB in which students select or build reaction templates, enter reagents and equivalents through structured web forms, upload NMR, IR, and UV–Vis exports, and use built-in viewers whose results are stored as machine-readable JSON; with one click, experiments are pushed to the Chemotion Repository, which mints DOIs for complete data sets (Hoffmann et al., 23 Sep 2025). The resulting workflow integrates planning, protocol execution, metadata capture, analysis, and repository deposition within one interface.

5. Integrity, compliance, and reproducibility criteria

For regulated environments, DLB requirements are formal rather than aspirational. The CyNote framework states that compliance with 21 CFR Part 11 requires audit trails, electronic signatures, data integrity controls, authentication and authorization, and record export and retention (Ng et al., 2023). In CyNote, every significant action is appended to EventLog, signatures are stored in a Signature table, passwords are protected with BCrypt-based hashing, forced password changes can be enforced after 90 days, and role-based permissions are available through the authentication layer. The notarization mechanism computes

$D$1

binding content to time and enabling later verification.

Good Laboratory Practice is treated in the DataFinder work as a provenance problem. By recording used, wasGeneratedBy, and wasControlledBy relations, and by attaching digital signatures to artifacts at hand-off, the system aims to provide an unbroken trace from archived result back to specimens, protocols, software versions, instrument IDs, and operators (Ney et al., 2011). In distributed settings it combines this with uniform cross-site Grid authentication based on X.509 proxy certificates and with archiving that bundles artifacts and provenance for long-term preservation.

Robotics-oriented DLBs make reproducibility constraints explicit at the level of experimental outcome. The informal requirement is that, given experiment record $D$2 and code $D$3, a third party should be able to reconstruct $D$4 such that $D$5, with

$D$6

where $D$7 is a distance metric on trajectories or sensory streams and $D$8 is a tolerance (Garcia, 2024). The same work recommends automatic capture of git hashes, container images, ROS parameters, and environment snapshots, and regular reproducibility audits that attempt to re-run older experiments and index missing context. SCHEMA lab addresses related concerns for computational experiments through containerization, capture of experiment configurations and performance metrics, explicit task and workflow groupings, and the ability to re-run tasks or cancel ongoing executions (Adamidi et al., 1 Apr 2025). This suggests that digitization alone is insufficient for reproducibility; the decisive issue is whether the DLB binds records to executable context, provenance, and verifiable metadata.

Security-oriented DLB designs can take an opposite architectural route. SLN avoids network dependencies entirely: there is no AJAX, no WebSockets, no server calls, no background processes, and all data remain on the local machine unless explicitly exported (Tsalaportas et al., 2014). In such systems, integrity is enforced not by server-side transaction logs but by XML schema validation, self-contained execution, and controlled file exchange. The contrast with server-centric ELNs is substantive: both seek trustworthy records, but they do so through different threat models.

6. Domains, empirical findings, limitations, and future directions

The DLB literature is notable for its disciplinary spread. Implementations appear in robotics (Garcia, 2024), regulated biomedical or pharmaceutical-style record-keeping (Ng et al., 2023), Grid-based scientific data management (Ney et al., 2011), small-group data-driven research (Schubotz et al., 2022), chemistry teaching and FAIR data stewardship (Hoffmann et al., 23 Sep 2025), immersive geoscientific fieldwork (Bernstetter et al., 24 Jul 2025), heliophysics and high-security laboratories (Tsalaportas et al., 2014), and computational experiment management (Adamidi et al., 1 Apr 2025). This breadth indicates that “Digital Lab Book” is not a domain-specific product category but a family of record-keeping and provenance systems adapted to distinct epistemic and regulatory conditions.

Several studies report concrete operational outcomes. In the Django-based “lazy approach,” manual steps were reduced from $D$9 per-file form filling to a single “Generate” click, and for labs producing approximately 3,000 files per year, initial tests showed more than 90% time savings on metadata entry; the same study reports “very fast onboarding,” while also noting requests for a more granular permission system (Schubotz et al., 2022). In the RWTH Aachen deployment, over three years the Chemotion tutorial series accumulated 1,600+ Zenodo downloads and 5,000+ YouTube views, and a survey reported that 90% of students rated the videos “very helpful” and reported fewer data-entry errors after training; tutors observed faster experiment write-ups and more complete metadata capture than in previous cohorts (Hoffmann et al., 23 Sep 2025). In Bernstetter et al.’s evaluation, pooled median Technology Acceptance Model scores were 69.44/100 for perceived usefulness and 72.22/100 for perceived ease of use; no significant difference was found for perceived usefulness, and although Mann–Whitney suggested a small perceived-ease-of-use advantage for the immersive setting ($M$0), usage-pattern metrics showed no statistically significant differences between immersive and non-immersive groups (Bernstetter et al., 24 Jul 2025). For SLN, Chrome-based benchmarks reported loading and rendering of SLN files larger than 400 MB in 3–4 seconds on a desktop with at least 200 MB free RAM, with UI animations tuned to remain below 16 ms per frame; the system has been operational since May 2013 at NASA SDAC and has also been used at ESA research centers (Tsalaportas et al., 2014).

Limitations are equally prominent. High-frequency logging such as 100 Hz IMU capture can bloat notebooks, interactive data-entry forms add UI complexity, and “commit-on-every-cell” improves reproducibility at the cost of write latency (Garcia, 2024). The “lazy approach” currently uses global user rights, lacks full audit logging of changes, and remains Windows-specific for automatic opening of local file links, while deeper decentralized trusted timestamping integration was explicitly left out of scope (Schubotz et al., 2022). Hoffmann et al. identify varying digital literacy, metadata curation overhead, and evolving laboratory protocols as practical obstacles to large-scale ELN-centered teaching (Hoffmann et al., 23 Sep 2025). Bernstetter et al. note a small, homogeneous sample, short sessions, shallow use of advanced features, and the absence of direct comparison with traditional notebooks or other provenance systems (Bernstetter et al., 24 Jul 2025). SCHEMA lab identifies future RO-Crate export, tighter workflow-language integration, and stronger version linkage as relevant extensions, while the virtual fieldwork DLB proposes PROV-compliant export, richer lineage queries, remote Git integration, auto-branching heuristics, and even head-mounted provenance interfaces (Adamidi et al., 1 Apr 2025).

Taken together, these systems show that the DLB has evolved from a digital substitute for the paper notebook into a broader infrastructure for provenance, metadata stewardship, compliance, and reproducible experimentation. The exact balance among structured forms, automated capture, signatures, storage federation, and executable context differs by domain, but the underlying trajectory is consistent: laboratory documentation is being redefined as a machine-actionable, queryable, and shareable component of the research workflow rather than a retrospective narrative appended to it.