HIDDENdb: Scalable Graph Extraction & Gene Interactions

• HIDDENdb is a framework for extracting, analyzing, and publishing latent interrelations from non-graph native and top-k constrained data using a declarative DSL.
• It employs condensed graph representations and deduplication strategies, achieving significant memory savings while supporting scalable network inference.
• The platform integrates gene and protein co-dependencies via multi-omic data fusion and structural cross-validation, enabling the discovery of authentic biological interactions.

HIDDENdb denotes a class of systems and methodologies for extracting, analyzing, and publishing latent interrelations—ranging from graph-theoretic structures to biological co-dependencies—from non-graph native or top−k-constrained data sources. The two dominant lineages are (1) the HIDDENdb system for graph extraction and in-memory analytics from relational data, and (2) the HIDDENdb database integrating gene and protein co-dependencies from heterogeneous bioinformatic datasets. Both share an emphasis on scalable representation, network inference, and interactive querying over data that are typically “hidden” or nontrivially accessible (Xirogiannopoulos et al., 2017, Silva et al., 6 Mar 2026, Thirumuruganathan et al., 2012, Sheng et al., 2012).

1. Declarative Graph Extraction from Relational Data

The HIDDENdb system for relational data enables users to declaratively specify, with a domain-specific language (DSL), rules for extracting graphs from an arbitrary database schema (Xirogiannopoulos et al., 2017). The DSL is grounded in a non-recursive Datalog fragment including:

• Nodes rules: $\text{Nodes}(ID, …) :- R_1(...), R_2(...), …$
• Edges rules: $$\text{Edges}(ID_1, ID_2, …) :- S_1(...), S_2(...), ...$$

Each rule defines a vertex or edge mapping over one or more base relations. The Edges rules are necessarily acyclic, aggregation-free conjunctive queries, allowing the system to construct a condensed in-memory representation with space complexity at most linear in the underlying tables.

A typical application is the extraction of a co-authorship network from publication and author tables, defined by joining author-publication relationships without materializing the full join output.

2. Condensed Representation and Duplication Mitigation

A foundational innovation in HIDDENdb’s graph extraction is its condensed, acyclic graph representation ($G_C$), which decouples “real” nodes (entities) from “virtual” nodes (high-cardinality join keys) (Xirogiannopoulos et al., 2017). The construction algorithm processes Edges rules by identifying “large” joins, splitting relation-chains accordingly, and assigning virtual nodes to join hubs. The condensed graph thus contains paths that encode multi-way relationships without fully expanding to $|V| + |E|$ scale.

However, this condensation causes a duplication problem: different join-paths between the same endpoints manifest as logically duplicate edges. This affects the correctness of duplicate-sensitive analytics (e.g., PageRank), necessitating deduplication strategies:

• C-DUP (on-the-fly deduplication): Employs runtime hashing to filter repeated visits, incurring per-traversal overhead.
• BITMAP representations: Precompute bitmaps at virtual nodes to mark traversable edges, trading off memory for speed.
• DEDUP-1/DEDUP-2: Greedy, structural preprocessing to eliminate duplicate paths, at the expense of temporary memory increase during construction.

Empirical benchmarks demonstrate that HIDDENdb’s representations (particularly BITMAP and DEDUP-1) achieve 3–10× memory savings on large real datasets without major performance degradation on core analytics relative to full expansion (Xirogiannopoulos et al., 2017).

3. Approaches to Hidden Database Crawling and Top‑k Barriers

A complementary issue addressed under the HIDDENdb umbrella is the challenge of crawling web-accessible hidden databases with proprietary interfaces and the top−k output constraint (each query only yields k answers, possibly omitting lower-ranked tuples) (Thirumuruganathan et al., 2012, Sheng et al., 2012). Two classes of algorithms have emerged:

• Optimal crawling: Algorithms such as rank-shrink (for numeric attributes) and slice-cover (for categorical attributes) provably minimize query cost, matching lower bounds up to constants. These methods partition the data space (via bisection or trie-based traversal) so that each query yields close to the maximal number of tuples (as allowed by the interface limit k). Partial results are obtained in proportion to the number of queries issued, enabling progressive crawling (Sheng et al., 2012).
• GetNext meta-algorithm: Allows iterative retrieval of tuples ranked below the public top−k set, even without knowledge of the site’s proprietary ranking function. This is achieved through candidate generation (issuing conjunctive queries to isolate possible next-ranked tuples) and candidate testing (identifying whether a candidate is truly the next tuple by issuing “beyond‑h minimal” queries mapping to minimal infrequent itemsets) (Thirumuruganathan et al., 2012). The approach is query-efficient, with empirical evidence demonstrating applicability to real-world web databases.

These techniques are foundational for both extracting unseen content and for supporting third-party analytics over hidden or restricted-access sources.

4. Gene and Protein Co-dependency Integration (HIDDENdb, 2026)

A separate major application under the HIDDENdb designation is an end-to-end system and database for integrating gene and protein co-dependency relationships across loss-of-function screens, curated interaction datasets, and GWAS—enabling multi-omic, cross-context co-dependency mapping (Silva et al., 6 Mar 2026).

Data Integration Pipeline:

HIDDENdb integrates:

• CRISPR-Cas9 loss-of-function (Achilles, Sanger)
• Genome-wide shRNA (Behan)
• Curated hits (BioGRID-ORCS)
• GWAS Catalog (NHGRI-EBI)
• Feature annotation databases (MitoCarta3.0)

Data are preprocessed to Z-scores using dataset-normalized transformations:

$$Z_{g,d} = \frac{\text{LFC}_{g,d} - \mu_{d}}{\sigma_{d}}$$

Meta-analysis aggregates per-pair evidence using Fisher’s method:

$X^2_{i,j} = -2 \sum_{d=1}^{D} \ln p^{(d)}_{i,j}$

resulting in meta p-values and co-dependency scores $s_{i,j}$ (as inverse Gaussian quantiles).

Co-dependency Scoring:

Pairwise Pearson correlations across cell lines produce $r_{i,j}$, converted to p-values with Student’s t-distribution, then corrected for FDR using the Benjamini-Hochberg procedure. Gene pairs are retained for $p_{i,j}^{(adj)} \le 0.01$ and $|r_{i,j}| \ge 0.3$; the main metric is the combined Fisher Z-score.

5. Network Assembly, Module Detection, and Validation

The weighted adjacency graph is defined as:

$$\text{Edges}(ID_1, ID_2, …) :- S_1(...), S_2(...), ...$$0

with community detection via the Louvain modularity-maximization algorithm:

$$\text{Edges}(ID_1, ID_2, …) :- S_1(...), S_2(...), ...$$1

Validation leverages AlphaFold3-predicted protein-protein interfaces (ipTM scores) for the highest-scoring gene pairs, with quartile-based enrichment calculation using hypergeometric tests and odds-ratios. FDR correction is applied for multiple testing, and significant structural co-enrichment is observed for top co-dependency tiers. This suggests that an appreciable fraction of inferred gene pairings reflect true physical interactions (Silva et al., 6 Mar 2026).

6. System Architecture, Query Interfaces, and Use Cases

HIDDENdb systems expose both graphical and programmatic APIs:

• Relational HIDDENdb: Core tables include nodes, edges, real/virtual node mappings, and metadata for deduplication strategy. Interfaces accept acyclic, aggregation-free DSL programs for graph extraction, and allow memory/accuracy trade-off configuration.
• Genome-focused HIDDENdb: Schemas comprise genes, dependencies, datasets, meta_scores, AlphaFold interfaces, and GWAS catalog. The web UI and REST API support gene-centric queries, dataset filters, Z-score cutoffs, chromosome views, and network downloads in multiple formats.

Example programmatic access (genome HIDDENdb):

$$\text{Edges}(ID_1, ID_2, …) :- S_1(...), S_2(...), ...$$2

For relational HIDDENdb, extraction and analysis pipelines seamlessly integrate with graph engines and support fast switching between condensed, deduplicated, and fully expanded representations to address different workload profiles.

7. Implications and Best Practice Recommendations

The HIDDENdb paradigm enables scalable network analytics on massive, relational or federated data without incurring prohibitive expansion costs (Xirogiannopoulos et al., 2017), and facilitates discovery in top−k-limited or web-facing data with provably minimal query overheads (Thirumuruganathan et al., 2012, Sheng et al., 2012). For biological co-dependency analysis, the integration of orthogonal screening and trait association sources, together with structural cross-validation, provides a stringent framework for identifying functionally coherent modules (Silva et al., 6 Mar 2026). The architecture supports incremental extensibility (algorithmic plug-in design), trade-off between memory and preprocessing effort, and offers user-facing controls for data filtering, network exploration, and export.

A plausible implication is that these design patterns—declarative extraction, representation efficiency, cross-dataset fusion, and flexible deduplication—can be generalized across domains where network inference must be performed at scale and under interface or memory constraints.

---

• (Xirogiannopoulos et al., 2017): "Extracting and Analyzing Hidden Graphs from Relational Databases"
• (Thirumuruganathan et al., 2012): "Digging Deeper into Deep Web Databases by Breaking Through the Top-k Barrier"
• (Sheng et al., 2012): "Optimal Algorithms for Crawling a Hidden Database in the Web"
• (Silva et al., 6 Mar 2026): "HIDDENdb: Co-dependency database reveals a plethora of genetic and protein interactions"