PhysDB: Physics-Oriented Database Solutions

• PhysDB is a database system designed for physics research that integrates structured and unstructured data management with scalable, high-availability features.
• It employs a hybrid design using CouchDB for flexible metadata and PostgreSQL for high-throughput sensor data, ensuring efficient data retrieval and revision control.
• PhysDB supports advanced simulation and interactive 3D asset annotation, facilitating reproducible experiments and generative model training in physics and AI.

PhysDB refers to a class of physics-oriented databases architected to capture, organize, annotate, and serve complex, multidimensional data in physics research. These databases are integral to both experimental and simulation-driven projects, supporting a range of data types—from experimental metadata to simulation results and richly attributed 3D assets. State-of-the-art implementations of PhysDB exemplify rigorous data modeling, scalable computational integration, hierarchical annotation, and tight coupling with analysis frameworks.

1. Architectural Approaches and Technology Stacks

PhysDB systems are engineered according to the data modality, scale, and operational demands of the host experiment or application. In large-scale experimental physics (e.g., DEAP-3600), PhysDB solutions use a hybrid architecture that partitions data between structured and unstructured storage:

• Unstructured, Evolving Metadata: CouchDB is selected for storing non-event metadata, favoring a schema-free document model (arbitrary JSON) that accommodates high-volume (O(10^5) documents), per-doc attachments (e.g., PNG, ROOT files), and schema evolution without downtime. This design supports efficient metadata curation across the project lifecycle.
• Structured, Write-Heavy Telemetry: PostgreSQL is employed for time-series (so-called "slow control") data, such as temperature, pressure, or other sensor streams, leveraging its strong support for append-only, indexed storage and relational consistency.
• Redundancy and Availability: High availability is achieved via multi-server CouchDB replication (mirrors on DAQ, on-site, and off-site analysis servers) and PostgreSQL data is exposed through HTTP façades for distributed analytics (Pollmann et al., 2019).
• Integrated Analysis Interfaces: Analytical workflows leverage runtime APIs (e.g., RAT-DEAP for DEAP-3600) that abstract storage backends and maintain unified user-facing operations in C++ and Python.

For physics-grounded simulation asset repositories (e.g., PhysDB in PhysForge), the architecture is tailored toward hierarchical object annotation—linking geometric, physical, and functional metadata in standard file formats with machine- and human-generated labels (Yang et al., 6 May 2026).

2. Data Models and Annotation Schemes

PhysDB encodes diverse data types through carefully designed logical models:

Experimental Metadata and Controls

• Relational (PostgreSQL) Schema: Sensor readings are stored with a composite primary key (sensor_tag, timestamp), referencing a table of sensor metadata. Indexing by sensor and descending time supports efficient retrieval of recent telemetry.

CREATE TABLE sensor_readings ( sensor_tag TEXT NOT NULL REFERENCES sensors(tag), ts TIMESTAMP WITH TIME ZONE NOT NULL, value DOUBLE PRECISION NOT NULL, PRIMARY KEY (sensor_tag, ts) );

• Unstructured (CouchDB) Schema: Each document ("dbdoc") is a JSON object with enforced top-level keys (name, index, run_range, author, createdOn), optional keys for notes, versioning, attachments, and arbitrary data fields. Document approaches favor fine granularity (e.g., per-PMT constants per doc) for flexible run validity and efficient revision control.

{ "name": "PMTSPE", "index": "100", "run_range": [17600, 17800], "author": "T.R.Pollmann", "createdOn": 1469112443, "SPE": [10.0], "_attachments": { "charge_hist.root": { ... } } }

• Indexing and Views: CouchDB "design documents" define JavaScript map functions for major views (e.g., selection by [name, index, startRun]), with additional logic for revision-controlled retrieval.

Physics-Grounded Digital Assets

PhysDB for interactive 3D content introduces a four-tier annotation schema mapped onto a part-wise tree $T(a) = (V, E)$ (Yang et al., 6 May 2026):

• Tier 1: Object-level properties (e.g., real-world scale, semantic category, usage scenes)
• Tier 2: Static part-wise parameters (e.g., semantic label, material type, mass)
• Tier 3: Functional properties (e.g., intrinsic function, state machine for allowed interactions)
• Tier 4: Kinematic/interactive affordances (e.g., joint type/axis/limits, affordance enumeration)

This enables per-part atomic affordances, rich assembly trees, and explicit state/kinematics definitions, supporting physics-based simulation and manipulation.

3. Database Operations, APIs, and Access Patterns

API Design

• RESTful Interfaces: Both CouchDB and PostgreSQL are exposed via REST endpoints for programmatic access. Typical CouchDB endpoints enable filtered selection and bulk fetch/update of JSON documents; PostgreSQL endpoints provide array responses of time-series sensor data or arbitrary SQL queries, both returned as JSON.

Example: Retrieve sensor time-series for a pressure reading

GET /slowcontrol?sensor=cryostat_pressure&from=2017-01-01&to=2017-02-01

Example: Query latest valid calibration metadata

GET /deapdb/_design/rat_interface/_view/select?startkey=[name,index,runID]&endkey=[name,index,endrun]&descending=true&limit=1&include_docs=true

• Index/View Evolution: CouchDB views are kept generic to enable cheap index rebuilds and stable client interfaces, with versioning managed through document field nesting (e.g., 'old_n' fields) and corresponding views for historical queries.

Example Query Workflows

Analysis clients request metadata (CouchDB) and slow control (PostgreSQL) data through a unified API, supporting transparent fail-over to local/offline stores if server connectivity fails. Operational tools (CouchApp or web front ends) provide role-specific interfaces (e.g., shift assignment statistics, sensor trend plots), often built around filtered, indexed view queries.

4. Performance, Scalability, and Data Integrity

• Throughput and Load: In DEAP-3600, CouchDB after ~4 years managed ~165,000 documents (dominated by PMT metadata and run configurations), sustaining 30 HTTP GET/s average (475 docs/s) with peaks up to 1,648 GET/s. View responses averaged 48 ms; grid-job failure due to DB connectivity remained below 0.1%.
• Growth and Partitioning: For high-velocity time-series, partitioning and time-based indexing keep query latency low even as rows approach several million per year. For document stores, continuous multi-site mirroring ensures high availability, and eventual consistency is managed via automated merge utilities.
• Maintenance: Access roles and field-level validation are encoded in update-validation functions; revision control in CouchDB is achieved by field nesting, and database replicas allow both disaster recovery and "offline mode" operation.

System	Unstructured Store	Structured Store	Replication
DEAP-3600	CouchDB	PostgreSQL	Multisite (3-mirror)
PhysForge	N/A*	Hierarchical annotation (file-based)	N/A

*PhysForge assets have document-centric annotation, not distributed via a separate DB node.

5. Formal Definitions and Metadata Hierarchies

Practices in PhysDB emphasize extensibility via formal annotation mappings. For a set of assets $D = \{ a_1, \ldots, a_N \}$:

• Each asset $a$ has a part-tree $T(a) = (V, E)$.
• An annotation function $\Phi : V \times \{1,2,3,4\} \to \text{Attrs}$ assigns properties at each schema tier.
• Kinematic constraints are encoded as $$K: V \to \{\text{fixed}, \text{revolute}, ...\}\times\mathbb{R}^3\times\mathbb{R}^3\times[\mathbb{R}^2]$$ mapping each part to its joint definition and motion bounds (Yang et al., 6 May 2026).

This schema underpins both simulation (enabling import into robotics and game engines) and algorithmic content generation.

6. Applications and Impact in Physics and AI

• Experimental Metadata Management: PhysDB enables robust organization, access, and reproducibility in large-scale detector environments, supporting high-throughput multi-user curation and runtime analysis for dark matter, neutrino, and similar experiments.
• Simulation and Virtual Worlds: Hierarchically annotated databases such as PhysDB in PhysForge serve as direct engines for embodied AI training, robotics simulation, and interactive 3D content generation. Physical properties (mass, kinematics, affordances) are made explicit for programmatic reasoning and manipulation.
• Vision–Language and Multimodal Planning: VLMs (e.g., Qwen2.5-VL) utilize PhysDB annotations to ground instruction planning, affording multi-step physical reasoning for agents that interact with physical models.
• Generative Model Training: Rich, attribute-wise labeled assets enable new classes of generative pipelines that synthesize physically plausible 3D interactables, as demonstrated in PhysForge's VLM-to-diffusion workflow (Yang et al., 6 May 2026).

7. Lessons Learned and Best Practices

• Flexibility in Evolving Projects: Schema-free document stores (e.g., CouchDB) support agile iteration in early project phases, facilitating rapid integration of new metadata types and workflows.
• Granularity and Partitioning: Storing rapidly-changing or highly heterogeneous data (per-sensor, per-PMT) in individual documents/tables minimizes transaction collisions and costly dataset re-indexing.
• Stable Interfaces: Maintaining a minimal set of generic indexed views decouples client logic from backend data representations, supporting smooth migration and long-term maintainability.
• Revision and Provenance: Embedding user-managed revision histories in document structures (with explicit views) ensures the reproducibility of analyses over a long project horizon.
• High Availability: Built-in, continuous replication (CouchDB) and clear roles for backups/replicas (PostgreSQL, file-based) are mandatory for collaborative, distributed research environments (Pollmann et al., 2019).

PhysDB thus encapsulates a principled approach to metadata and asset management for both experimental and simulation-based physical sciences, tightly integrating hierarchical annotation, scalable architecture, and operational flexibility for enduring, reproducible research.