FedLAB: Traceable Semantic Codebooks for Federated Multimodal Graph Foundation Learning

Abstract: Multimodal graph foundation models aim to learn reusable knowledge from graphs enriched with text, images, attributes, and relational topology, thereby supporting diverse graph-centric and modality-centric tasks. In practice, however, such multimodal graphs are often distributed across decentralized clients, where raw contents and local structures cannot be centrally shared due to privacy constraints. This motivates federated multimodal graph foundation learning, which requires not only transferable representation learning but also intrinsic semantic traceability under strict data isolation. Existing methods usually exchange or store knowledge through parameters, prototypes, embeddings, or compact codebooks, which support optimization and transfer but do not explicitly expose how modality evidence, node semantics, and topology context jointly support predictions. To bridge this gap, we propose FedLAB, a traceable semantic codebook framework that organizes multimodal graph knowledge into typed hierarchical codebooks for modality evidence, node semantics, and topology context. FedLAB further refines these trace units through federated semantic barycenter pre-training while keeping raw multimodal contents and graph structures local. Extensive experiments on 10 benchmarks and 6 downstream tasks show that FedLAB improves over state-of-the-art baselines by up to 7.53\%, while preserving a native semantic trace interface.

• The paper introduces FedLAB, a federated multimodal graph foundation framework that uses typed modality, node-semantic, and topology codebooks to create intrinsic, reusable trace paths without sharing raw client data.
• FedLAB outperforms the strongest baselines across all ten benchmark–task settings, achieving an average improvement of 4.38%, a maximum gain of 7.53%, and examples such as 84.14% accuracy on Toys node classification.
• The paper shows that traceability improves prediction faithfulness through code sufficiency, topology-drop, and semantic-concentration measures, but increases resource demands to 6.66×10⁸ communication scalars and 1,583.3 MB of storage.

Problem and motivation

Multimodal graph foundation models (MM-GFMs) learn reusable knowledge from graphs enriched with text, images, attributes, and relational topology. In federated deployments, such graphs are distributed across clients that cannot share raw contents, labels, or structures. The paper identifies a specific deficiency in this setting, which it terms the semantic traceability gap: existing federated graph learning (FGL) and foundation-style methods exchange knowledge through parameters, gradients, prototypes, embeddings, or compact codebooks, which support optimization and transfer but do not expose how modality evidence, node semantics, and topology context jointly support a prediction. Post-hoc explainers such as GNNExplainer and PGExplainer are instance-specific and are not learned as reusable trace interfaces during pre-training.

FedLAB addresses this gap by defining three trace interfaces that a federated multimodal graph foundation model should expose: modality-evidence traceability (which modality signals drive a prediction), node-semantic traceability (which reusable node-level semantic units activate across clients), and topology-context traceability (how local structure modulates node semantics). The framework is evaluated under the MM-OpenFGL protocol: $K$ clients each hold a private multimodal graph with a modality-availability mask, and only model updates plus aggregated posterior code statistics leave the client.

Method

Shared trace vocabulary

FedLAB maintains typed codebooks $$\mathcal{B} = (\{\mathcal{B}^r\}_{r\in\mathcal{M}}, \mathcal{B}^s, \mathcal{B}^t)$$ for modality evidence, node semantics, and topology context. For any type-$a$ representation $h$, a typed readout operator computes an entropy-regularized posterior assignment over codes, a relaxed trace representation, and a hard selected code index, using a straight-through estimator in the style of VQ-VAE to keep optimization differentiable. The server broadcasts the backbone and codebooks each round so all clients encode traces in a shared semantic coordinate system.

Hierarchical trace encoding

Local forward computation is restructured into an explicit path rather than a monolithic fused embedding:

1. Modality evidence: each observed modality's projected feature is quantized into a modality-evidence code.
2. Evidence–semantic routing: masked softmax over modality support scores composes evidence into a routed summary, which is then read from the node semantic codebook.
3. Topology-context routing: neighbor contributions are scored from semantic codes and lightweight structural descriptors; the aggregated neighborhood state is read from the topology context codebook.

Each prediction returns a trace path consisting of activated evidence codes, a node semantic code, a topology context code, and contribution scores $(\rho_{k,v}, \beta_{k,v})$. Because these scores directly participate in modality composition and neighbor routing, the trace is intrinsic to the prediction process rather than appended post hoc.

Semantic barycenter pre-training

The local objective combines task loss, three trace-preservation losses (modality reconstruction from the hierarchical trace state, alignment of evidence with node semantics, and preservation of structural relations by topology traces), and a codebook regularizer combining commitment loss with a KL term against uniform usage to prevent code collapse. After local training, clients upload only posterior usage counts and posterior-weighted local centers per code. The server forms usage-weighted barycenters across clients and updates global code units via spherical projection with momentum. This aligns local traces into cross-client references without exposing raw data.

Empirical results

Experiments cover ten multimodal-attributed benchmarks (e-commerce, video platforms, Flickr30k, SemArt) partitioned into 10 non-IID clients via Louvain community detection, with six downstream tasks spanning graph-centric (node classification, link prediction) and modality-centric (matching, retrieval, G2text, G2image) families.

Method family	Representative	Typical standing
FL	FedAvg	Weakest overall
MM-GNN	Fed-MGNet, Fed-MHGAT	Competitive on some tasks
MM-FL	FedMVP, FedMAC	Mid-range
GFM	Fed-GFT, Fed-GraphCLIP	Strong on matching/retrieval
FGL-GFM	FedGFM+, FedBook	Strongest baselines
FedLAB	—	Best on all ten benchmark–task columns

FedLAB reports an average improvement of 4.38% over the strongest baseline, with a maximum gain of 7.53%. Representative results include 84.14% accuracy on Toys node classification versus 80.19% for FedBook, 74.32% AUC on Bili Music link prediction versus roughly 68% for the best baseline, and 94.49% AUC on QB modality matching. Notably, FedBook—the closest architectural relative—degrades sharply on modality matching (77.44%) and retrieval (65.31%), suggesting that its unified codebook does not preserve fine-grained cross-modal evidence as effectively as FedLAB's typed decomposition.

Ablations confirm complementarity of the three codebooks: removing the node, modality, and topology codebooks reduces average performance by 7.81%, 6.91%, and 6.23%, respectively. The node codebook contributes most to cross-client semantic reuse, consistent with its role as the shared abstraction layer.

Robustness analyses show stability across hyper-parameter ranges ($\lambda$, edge-drop ratio, topology codebook size), mild degradation as client count increases, and smooth convergence. Efficiency analysis concedes real overhead: FedLAB requires $6.66\times10^{8}$ communication scalars, 1583.3 MB of space, and 188.99 s wall-clock time—roughly double FedBook's space cost and substantially above FedAvg—though it remains faster than FedBook (254.39 s). Traceability therefore carries a measurable resource premium, which the authors present as a practical trade-off rather than a free benefit.

Traceability itself is validated with three quantitative probes: trace sufficiency (TS@K), showing retained prediction confidence from top-$K$ evidence codes; topology trace drop (TTD), where removing FedLAB-selected topology traces causes steeper confidence drops than GNNExplainer-, PGExplainer-, GraphLIME-, or random-based removal; and semantic code concentration (SCC@K), where same-label nodes concentrate on compact reusable codes more than under random assignment. These metrics indicate the traces are faithful to the decision process, not merely decorative outputs.

Limitations and open questions

Several constraints qualify the results. First, the efficiency overhead is substantial: memory cost roughly doubles relative to FedBook and grows about 25× over FedAvg, so deployment on resource-constrained clients remains unaddressed. Second, the evaluation uses Louvain-based partitioning into 10 clients on ten benchmarks; behavior under adversarial or highly skewed client distributions, larger client counts, and dynamic client participation is not characterized beyond mild scalability curves. Third, faithfulness is measured through sufficiency/drop/concentration proxies; whether the exposed trace paths withstand stricter causal or counterfactual audits is left open. Fourth, the barycenter refinement assumes posterior statistics are informative summaries of local data; privacy leakage through aggregated code statistics is not analyzed. Finally, the framework presupposes a fixed modality set with availability masks; extension to open-ended or evolving modality vocabularies remains an open question.

Conclusion

FedLAB reframes federated multimodal graph foundation learning around intrinsic semantic traceability, instantiating it with typed hierarchical codebooks refined through federated semantic barycenter pre-training. It delivers consistent empirical gains—an average 4.38% improvement and up to 7.53% over state-of-the-art baselines across ten benchmarks and six tasks—while providing validated, faithful trace paths at a quantified but nontrivial computational and memory cost. The work establishes traceability as a first-class design objective for federated graph backbones and leaves open the questions of statistical-privacy guarantees for shared code statistics and scalability of typed codebooks to very large federations.