FuDoBa: Document & Knowledge Graph Fusion

• FuDoBa is an early-fusion framework that integrates LLM and knowledge graph embeddings to produce low-dimensional, task-relevant document representations.
• It employs truncated SVD and Bayesian optimisation to jointly select subspace dimensions and fusion weights, ensuring efficient and interpretable classification pipelines.
• Empirical evaluations across diverse datasets demonstrate FuDoBa achieves comparable or superior performance to high-dimensional baselines with drastically reduced feature dimensionality.

FuDoBa (Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation) is an early-fusion document representation learning framework that combines LLM embeddings with structured knowledge graph (KG) embeddings. By integrating both globally sourced (e.g., WikiData5m) and locally constructed knowledge graphs, FuDoBa generates low-dimensional, task-relevant representations optimized for downstream classification performance. The method employs Bayesian optimisation to jointly determine per-modality subspace dimensions and fusion weights, achieving classification performance comparable to high-dimensional LLM baselines while significantly reducing feature dimensionality and allowing interpretable attribution of modality importance (Koloski et al., 9 Jul 2025).

1. Motivation and Conceptual Foundations

The proliferation of high-performing LLM-based document encoders has resulted in embeddings that are often high-dimensional (e.g., 1,536 dimensions for OpenAI’s text-embedding-2-small), generic, and computationally expensive for domain-specific applications and AutoML pipelines. These embeddings may inadequately capture domain-specific or fine-grained relational structures present in textual data. Complementary to LLMs, knowledge graphs offer structured representations of entities and relations, with global KGs encoding broad world knowledge and local KGs (constructed via relation extraction) providing task- or dataset-specific relational context. FuDoBa is motivated by the hypothesis that fusing these complementary modalities within an optimized low-dimensional subspace preserves or enhances classification accuracy, reduces computational and storage overhead, and yields interpretable representations suitable for rapid AutoML model development (Koloski et al., 9 Jul 2025).

2. Architecture and Pipeline

A. Embedding Acquisition and Preprocessing

FuDoBa’s pipeline comprises three principal embedding modalities:

• LLM Embeddings: 1,536-dimensional fixed embeddings from OpenAI’s text-embedding-2-small API.
• Global KG Embeddings: Entities linked via Babelfy to WikiData5m, embedded using 512-dimensional RotatE vectors.
• Local KG Embeddings: Entity–relation–entity triplets per document, extracted by relation extraction models (e.g., GPT-4o-mini, REBEL, ReLiK), embedded using 512-dimensional RotatE.

Each modality is normalized using an Elastic-Net style norm: $$N(\mathbf{x}) = \frac{\mathbf{x}}{w_1 \|\mathbf{x}\|_1 + w_2 \|\mathbf{x}\|_2}$$ with $w_1 = w_2 = 0.5$.

Dimensionality reduction is achieved via truncated singular value decomposition (SVD) for each modality $m$, projecting $E_m \in \mathbb{R}^{N \times d}$ to $P_m \in \mathbb{R}^{N \times l_m}$.

B. Fusion Mechanism

Per-modality projections $P_m$ are scaled by global weights $\alpha_m$, yielding $S_m = \alpha_m P_m$. The concatenated vector

$$\mathbf{S}_{\mathrm{concat}} = [S_{\mathrm{LLM}} \;\|\; S_{\mathrm{kg}} \;\|\; S_{\mathrm{loc}}]$$

is then normalized to produce the final representation: $$\mathbf{X}(\theta) = N(\mathbf{S}_{\mathrm{concat}})$$ where the parameter vector $w_1 = w_2 = 0.5$0 collects the SVD dimensions and fusion weights for each modality.

The scaling weights $w_1 = w_2 = 0.5$1 directly influence the contribution of each modality, and setting $w_1 = w_2 = 0.5$2 effectively drops that modality. The learned $w_1 = w_2 = 0.5$3 values offer direct interpretability regarding the importance of each source of information in downstream tasks.

3. Bayesian Optimisation for Representation Fusion

FuDoBa employs Gaussian-Process-based Bayesian optimisation (BO) to search the joint space of modality-specific subspace dimensions $w_1 = w_2 = 0.5$4 and fusion weights $w_1 = w_2 = 0.5$5. The objective is to maximize downstream k-fold cross-validated macro F1 when training AutoGluon classifiers on the fused representations: $w_1 = w_2 = 0.5$6 BO is conducted over a search space of approximately $w_1 = w_2 = 0.5$7 combinations, with a 50-trial budget per dataset. The BO surrogate model is a Gaussian Process (GP) with a Matérn kernel: $w_1 = w_2 = 0.5$8 The acquisition step uses Expected Improvement (EI), iteratively selecting candidate parameters to maximize anticipated F1 gains.

4. Interpretability, Dimensionality, and Efficiency

The scalar modality weights $w_1 = w_2 = 0.5$9 encode the relative importance "trusted" for each modality after optimisation. Averaged across six datasets, post-optimisation mean weights were $m$0, $m$1, and $m$2, with no significant pairwise differences ($m$3). This indicates the model flexibly adapts to dataset-specific utility among modalities. Importantly, FuDoBa achieves drastic dimensionality reduction—producing representations between 80–144 dimensions (approx. $m$4–$m$5), as compared to the 1,536 dimensions of LLM-base or up to 2,560 with simple concatenation.

FuDoBa Table: Feature Dimensions and Modality Importance

Modality	SVD Dimensionality	Mean $m$6 Weight
LLM	16 – 64	0.78
Global KG	16 – 64	0.43
Local KG	16 – 64	0.50

This mechanism supports AutoML training within strict runtime budgets (e.g., 300 s per run), on commodity CPU hardware, while incurring orders-of-magnitude reductions in embedding storage and compute requirements.

5. Empirical Evaluation on Diverse Tasks

Experiments were performed on six English datasets spanning two domains:

• Sentiment (Binary): Amazon Reviews (Books, DVD, Music), HateSpeech (social media).
• News Genre (Multi-class): MLDoc (4 genres), XGENRE (9 genres).

All employed standard train/test splits. The following table summarizes downstream classification F1 (%) on test splits for FuDoBa and high-dim LLM-only baselines:

Method	Books	DVD	Music	HateSpeech	MLDoc	XGENRE
LLM (1536-d)	92.80	92.70	92.90	75.68	96.74	44.66
FuDoBa (80–144-d)	93.40	93.90	92.70	74.61	96.19	50.25

FuDoBa matches or surpasses the LLM-only baseline in 5 of 6 cases, with salient gains (e.g., +5.6 F1 on XGENRE) at less than 10% the dimensionality. Nonparametric tests (Friedman/Nemenyi) found no significant difference between FuDoBa and high-dimensional sum-concat baselines, establishing practical equivalence in accuracy at a greatly reduced computational cost.

Ablation studies substantiate the value of FuDoBa’s architecture:

• Per-Modality Projection + Weighting outperforms concat-then-project strategies in 5 of 6 datasets.
• Local KG extraction improves entity coverage in short or noisy documents (HateSpeech: “no-entity” reduction from 22%→17%), yielding higher mean entity counts.
• Relation extraction method (GPT-4o-mini, REBEL, ReLiK) made no significant difference in downstream F1 ($m$7, average F1 ≈ $m$8%).

6. Strengths, Limitations, and Future Directions

Strengths

• Achieves low-dimensional, cost-efficient representations using only frozen, off-the-shelf encoders and SVD.
• Produces interpretable modality weights ($m$9), facilitating insight into which information sources drive performance on each task.
• Highly extensible: the early-fusion paradigm can incorporate new modalities, such as vision or additional KGs.
• Suited for resource-constrained deployments, running on commodity hardware with fast AutoML pipelines.

Limitations

• Performance can degrade on highly noisy or unstructured texts (e.g., HateSpeech), where large high-dimensional models may retain some edge.
• Heavily reliant on high-quality relation extraction; upstream errors propagate through to KG embeddings.
• Bayesian optimisation search is sequential and may be resource-limited at small trial budgets (e.g., 50).
• Evaluations to date are limited to English and two task types.

Future Directions

• Employ parallel or population-based search (e.g., genetic algorithms) to address BO’s sequential constraints at scale.
• Extend to multilingual/cross-lingual corpora.
• Incorporate additional modalities (images, structured tables).
• Investigate meta-learning to warm-start hyperparameter searches based on previous tasks’ optima.
• Explore LLM-driven soft-labeling to enrich KG construction or provide auxiliary supervision to AutoML.

7. Significance and Impact

FuDoBa demonstrates that early fusion of LLM and KG modalities, combined with Gaussian-Process-based Bayesian optimisation, can yield compact, interpretable, and performant embeddings for document classification tasks. By drastically reducing dimensionality and enabling transparent modality weighting, FuDoBa facilitates deployment of robust, domain-adaptive models in settings where high-dimensional LLM embeddings are otherwise prohibitive. The framework’s flexibility and empirical efficacy across multiple domains and tasks highlight its practical utility and establish a basis for future extension to broader representation learning contexts (Koloski et al., 9 Jul 2025).