Bi-Directional Text Fusion (BiT)

• Bi-Directional Text Fusion (BiT) is a dual-modality framework that integrates graph structure and text using iterative cross-modal attention for enriched node representations.
• It employs stacked Graph-Text Fusion Units with soft structural prompts and cross-attention to continuously refine both text and graph features.
• Empirical evaluations show BiT outperforms GNN-only baselines on node classification tasks, yielding higher accuracy and improved clustering metrics.

Bi-Directional Text Fusion (BiT) is a mechanism for text-attributed graph representation learning in which text representations influence structural graph representations, and structural representations influence how text is interpreted inside a pretrained LLM, both repeatedly and within the same trainable block. In the formulation introduced with BiGTex, BiT is designed for text-attributed graphs (TAGs), where each node has both graph context and node-associated text, and it is realized through stacked Graph-Text Fusion Units with bi-directional attention. The central claim is that continuous mutual refinement is preferable to one-shot or one-way coupling between graph neural networks (GNNs) and pretrained LLMs (PLMs), because GNN-only systems cannot natively read raw texts and LLM-only systems are typically unaware of graph structure (Beiranvand et al., 16 Apr 2025).

1. Problem setting and conceptual scope

A text-attributed graph is defined as

$G = (V, E, \{S_n\}_{n\in V}, Y),$

where $V$ denotes nodes, $E \subseteq V \times V$ denotes edges, $S_n$ is a variable-length token sequence associated with node $n$, and $Y$ is the label space for a subset of nodes. Each node therefore has structural context and a textual description, such as a paper title and abstract in a citation graph or profile text in a social graph. Typical tasks include node classification $f: V \to Y$, link prediction, and node clustering (Beiranvand et al., 16 Apr 2025).

The motivation for BiT follows from three stated difficulties. First, TAGs involve heterogeneous modalities: graph structure and free-form text. Second, the inductive biases differ: GNNs are designed for local message passing over $E$, whereas PLMs and LLMs are strong text encoders but remain ignorant of $E$ unless graph structure is explicitly injected. Third, prior joint models often permit only limited interaction. In PLM$\rightarrow$GNN pipelines, text is encoded once and then treated as static node features. In GNN$V$0PLM pipelines, the PLM receives a structural vector, but that vector itself was learned without text. Alignment or contrastive approaches typically rely on separate encoders and auxiliary losses, with limited direct fusion. The stated deficiency is the absence of continuous mutual refinement: text does not affect how structure is processed within the GNN in a fine-grained way, and structure does not affect token-level self-attention inside the LLM (Beiranvand et al., 16 Apr 2025).

A common misunderstanding is to equate bi-directionality with mere feature concatenation or a one-time prompt construction. In this literature, BiT denotes repeated coupling inside the model’s internal computation. The intended effect is iterative semantic and structural refinement rather than a sequential handoff from one encoder to another.

2. Core architecture: Graph-Text Fusion Units

BiGTex realizes BiT through stacked Graph-Text Fusion Units. Each unit contains a GNN component and an LLM or PLM component, with BERT-Base used in the reported experiments. The inputs to Fusion Unit $V$1 are the graph structure, tokenized node text, and a node representation $V$2; for the first unit, $V$3 is an initial feature $V$4 projected to the PLM dimension. The unit implements three steps: Structure $V$5 Text through a soft structural token, Text $V$6 Structure through cross-attention, and graph message passing over the resulting text-refined structural features (Beiranvand et al., 16 Apr 2025).

Direction	Mechanism	Immediate role
Structure $V$7 Text	Soft structural token	Injects graph signal into PLM self-attention
Text $V$8 Structure	Cross-attention	Produces a text-conditioned structural feature
Structure aggregation	GNN layer	Propagates text-aware node features over $V$9

For Structure $E \subseteq V \times V$0 Text, the previous structural representation $E \subseteq V \times V$1 is prepended to the node text as a soft prompt: $E \subseteq V \times V$2 Because this token participates in PLM self-attention, each text token can attend to structural context and the structural token can attend back to text. The attention primitive is the standard scaled dot-product form

$E \subseteq V \times V$3

For Text $E \subseteq V \times V$4 Structure, PLM outputs over $E \subseteq V \times V$5 serve as keys and values, while the previous GNN output $E \subseteq V \times V$6 serves as query: $E \subseteq V \times V$7 The cross-attention output is a text-conditioned structural feature.

That feature is then passed to a GNN layer. The generic message-passing form is

$E \subseteq V \times V$8

In BiGTex, these node features already encode text through the cross-attention stage, so neighborhood aggregation becomes text-aware. Stacking units yields an iterative loop,

$E \subseteq V \times V$9

in which graph-refined prompts are repeatedly returned to the PLM and text-refined structural features are repeatedly returned to the GNN. This repeated loop is the defining architectural property of BiT (Beiranvand et al., 16 Apr 2025).

3. Representation space, parameter adaptation, and supervision

The reported textual backbone is BERT-Base. The node representation extracted from the PLM is typically the output of the special $S_n$0 token in encoder-based PLMs; for decoder-only models, the proposed alternative is the average of hidden states across all tokens. On the graph side, BiGTex instantiates the structural backbone with standard GNN layers—GCN, GAT, and GraphSAGE—in separate experiments. Initial node features are dataset-specific shallow features projected to the PLM hidden dimension when necessary: PubMed uses TF-IDF (dim=500), ogbn-Arxiv uses skip-gram (dim=128), the ogbn-Products subset uses BoW (dim=100), and Arxiv2023 uses word2vec (dim=300) (Beiranvand et al., 16 Apr 2025).

Alignment between text and structure is enforced through projection into a common latent space. If raw features are not already in $S_n$1, the first Fusion Unit applies a linear projection,

$S_n$2

Cross-attention also assumes shared dimensionality, with internal linear projections mapping $S_n$3, $S_n$4, and $S_n$5 to a common dimension when needed. This common latent space is what makes bi-directional attention well-defined.

PLM adaptation is parameter-efficient rather than full-model fine-tuning. The base PLM weights are frozen, and LoRA is used with low-rank updates

$S_n$6

so that

$S_n$7

In the experiments, the LoRA rank is $S_n$8. The soft prompt is itself a learned continuous vector rather than a discrete vocabulary token, and it changes dynamically as GNN representations change over training. The paper states that LoRA allows the PLM to learn how to use those structural prompts without full fine-tuning (Beiranvand et al., 16 Apr 2025).

Node classification uses a final MLP classifier head on the last Fusion Unit output $S_n$9, with cross-entropy loss

$n$0

No explicit auxiliary or contrastive loss is used; node labels are the sole supervision. Training uses learning rate $n$1, weight decay $n$2, mini-batch size 6, and PyG’s NeighborLoader to sample subgraphs. The reported setup uses only two Fusion Units and 10 epochs.

Ablation results isolate the role of the main BiT components. Without LoRA, performance drops, especially on ogbn-Arxiv from $n$3 to $n$4 and on Products from $n$5 to $n$6. Without the soft prompt, accuracy decreases on all datasets. Without both, degradation is severe, for example on Products from $n$7 to $n$8. These figures are presented as evidence that parameter-efficient PLM adaptation and soft structural prompting are key to effective Bi-Directional Text Fusion (Beiranvand et al., 16 Apr 2025).

4. Benchmarks, transfer behavior, and representation analysis

The experimental evaluation is conducted on four TAG datasets used for node classification: PubMed, ogbn-Arxiv, an ogbn-Products subset, and Arxiv2023. Each combines graph connectivity with natural-language node descriptions (Beiranvand et al., 16 Apr 2025).

Dataset	Graph statistics	Task and initial features
PubMed	19,717 nodes, 44,338 edges, avg degree 4.5	3-class node classification, TF-IDF (500-dim)
ogbn-Arxiv	169,434 nodes, 1,166,243 edges, avg degree 13.7	40 classes, skip-gram (128-dim)
ogbn-Products subset	54,025 nodes, 74,420 edges, avg degree 50.5	47 classes, BoW (100-dim)
Arxiv2023	46,198 nodes, 78,548 edges, avg degree 3.4	40-class node classification, word2vec (300-dim)

On node classification, BiGTex is reported to outperform GNN-only baselines and strong TAG baselines such as TAPE, SimTeG, GLEM, and ENGINE. On ogbn-Arxiv, the best baseline is described as being around $n$9–$Y$0, whereas BiGTex with a SAGE backbone reaches $Y$1, a $Y$2 absolute gain over TAPE’s $Y$3. On the Products subset, TAPE reaches $Y$4 and BiGTex reaches $Y$5, a $Y$6 gain over the best baseline. On PubMed, TAPE reaches $Y$7 and BiGTex reaches $Y$8, a $Y$9 gain. BiGTex also leads on Arxiv2023. On the official ogbn-Arxiv leaderboard, BiGTex is reported as the highest among methods listed at test accuracy $f: V \to Y$0, using approximately $f: V \to Y$1M trainable parameters and a single RTX 4090 (24GB), whereas some baselines use more than $f: V \to Y$2B parameters and multiple A100 GPUs (Beiranvand et al., 16 Apr 2025).

Although trained only for classification, the learned embeddings are also evaluated for link prediction with ROC-AUC. On ogbn-Arxiv, X-SimTeG reaches $f: V \to Y$3, X-TAPE $f: V \to Y$4, and X-BiGTex $f: V \to Y$5, so BiGTex is better than TAPE but slightly below SimTeG. On Arxiv2023, X-SimTeG reaches $f: V \to Y$6, X-TAPE $f: V \to Y$7, and X-BiGTex $f: V \to Y$8, which is the best among all compared methods.

The representation analysis is similarly favorable. In t-SNE projections on ogbn-Arxiv and Arxiv2023, BiGTex clusters are described as more compact per class and more clearly separated across classes. In unsupervised clustering with K-Means, ARI and NMI improve substantially. On ogbn-Arxiv, X-orig has ARI $f: V \to Y$9 and NMI $E$0, X-TAPE has ARI $E$1 and NMI $E$2, and X-BiGTex reaches ARI $E$3 and NMI $E$4. On Arxiv2023, X-orig has ARI $E$5 and NMI $E$6, X-TAPE has ARI $E$7 and NMI $E$8, and X-BiGTex reaches ARI $E$9 and NMI $E$0. The paper interprets these results as evidence that BiT captures intrinsic semantics plus graph structure (Beiranvand et al., 16 Apr 2025).

5. Relation to adjacent bidirectional fusion paradigms

Within graph-text research, BiT as formalized in BiGTex is preceded by related efforts but differs from them in how directly and repeatedly it couples the two modalities. An earlier precursor is BiTe-GCN, which transforms a text-rich network into an augmented bi-typed heterogeneous network, introduces document, word, and document-word subgraphs, and performs convolutions of both topology and features simultaneously. Its bidirectionality lies in a graph construction where text semantics can shape topology and an attention-based aggregator can learn how much to trust topology-derived versus text-derived signals (Jin et al., 2020).

A later TAG formulation, PromptGNN-sim, also pursues deep, bi-directional structure-semantic fusion, but with a different mechanism. It uses a Graph Attention Network for semantically aware neighborhood selection by combining structural attention with textual similarity; the selected context is then used to generate structure-aware prompts for an LLM, and training adds bi-directional cross-modal contrastive learning and cross-attention. Its stated emphasis is robustness under sparse perturbations, cross-task transfer, and cross-dataset generalisation (Hu et al., 29 Jun 2026).

Beyond TAGs, bi-directional fusion appears in several multimodal literatures. In referring image segmentation, FCNet adopts a bi-directional vision-language guided framework in which a vision-guided fusion stage is followed by a language-guided calibration module, with the explicit claim that single-guided fusion limits fine-grained cross-modal correlation (Yan et al., 2024). In generative image-text modeling, a unified multimodal Transformer has been used to jointly study image-to-text and text-to-image generation within one shared architecture, treating both modalities as token sequences and using the same Transformer for both directions (Huang et al., 2021). CoBIT further unifies contrastive image-text alignment, image-to-text generation, and text-to-image generation in one framework through image and text unicoders plus a cross-modal decoder (You et al., 2023).

The acronym is not unique across machine learning. In neural machine translation, “BiT” denotes “Bidirectional Training,” a pretraining strategy in which the same model is updated on both translation directions by duplicating each parallel pair in reverse before standard one-direction fine-tuning (Ding et al., 2021). This suggests that the stable element across uses is not a single architecture, but a design principle: mutual conditioning across two directions rather than a fixed one-way pipeline.

6. Limitations, misconceptions, and prospective extensions

The reported limitations of BiGTex are threefold. First, computational cost remains higher than for pure GNNs, because each Fusion Unit adds PLM computation and cross-attention on top of graph message passing. Second, the method depends on text quality: it assumes node texts are present and semantically rich, and benefits may diminish when texts are sparse or noisy. Third, the current design targets static, single-relation graphs and does not directly address dynamic graphs, heterogeneous node or edge types, or multi-relational structures (Beiranvand et al., 16 Apr 2025).

These limitations clarify another misconception: BiT does not eliminate the need for structural inductive bias or textual quality; it attempts to coordinate them. When structure is informative, soft prompts and PLM self-attention help text encodings reflect the node’s neighborhood. When text is rich and structure is sparse or noisy, PLM semantics fed back via cross-attention help the GNN represent nodes whose graph neighbors do not fully explain their labels. The empirical ablations are presented precisely to show that both directions are necessary.

The proposed future directions follow the same logic. The paper suggests scalable variants for long texts and extremely large graphs, including more efficient PLMs, pruning or compression in bi-directional fusion, and sparse attention mechanisms; domain-adapted or instruction-tuned LLMs; support for dynamic and heterogeneous graphs through relation-aware GNNs and metadata-aware prompts; and semi-supervised or few-shot setups that leverage PLM priors more heavily when labels are scarce (Beiranvand et al., 16 Apr 2025). A plausible implication is that future BiT systems will continue to move toward richer PLMs, more structured GNNs, and more efficient cross-modal transformations, while preserving the central requirement that structure and text repeatedly refine one another rather than interact only once.