SheafAlign: Decentralized Multimodal Alignment

• SheafAlign is a decentralized multimodal alignment framework that leverages sheaf theory to encode pairwise modality relationships, preserving both shared and unique information.
• It employs edge-local contrastive and reconstruction losses alongside a sheaf Laplacian for enforcing global consistency in a distributed sensor network.
• Experimental evaluations show significant gains in zero-shot accuracy, robust performance under sensor dropout, and reduced communication overhead compared to conventional global-space methods.

SheafAlign is a decentralized multimodal alignment framework that leverages sheaf-theoretic models to systematically resolve the challenges posed by real-world distributed sensor and data fusion scenarios. Unlike conventional methods that rely on embedding all modalities into a single alignment space and require extensive redundancy among modalities, SheafAlign operates by encoding pairwise relationships in dedicated comparison spaces, thereby preserving both shared and unique modality information. This enables improved robustness to missing data, decentralized deployment, and efficient communication, while attaining superior generalization and retrieval performance relative to global-space alignment approaches (Ghalkha et al., 23 Oct 2025).

1. Limitations of Global-Space Multimodal Alignment

Traditional alignment techniques such as CLIP, AudioCLIP, and ImageBind embed all modalities into a single global vector space, with a fixed reference modality (usually vision). This monolithic approach assumes high mutual redundancy between all modality pairs and universal co-observation, which is frequently violated in distributed sensor networks, spatially separated platforms, and cross-domain environments. As a result, such methods display “visual bias,” incompletely capture modality-unique information, and are especially brittle to partial sensor dropout or incomplete co-occurrence between modalities. Additionally, these designs require relaying high-dimensional embeddings through a central server, resulting in communication inefficiencies and loss of local data privacy (Ghalkha et al., 23 Oct 2025).

2. Sheaf-Theoretic Foundation

SheafAlign models a decentralized sensor/data network by representing $N$ clients as nodes in a connected undirected graph $\G=(\V,\E)$. Each node $i\in\V$ acquires its own modality $m_i$ and encodes samples to a latent vector $\h_i$. The mathematical core is a cellular sheaf $\F$ over $\G$:

• Each vertex $i$ carries a stalk $\F(i) = \R^{d_i}$, the local embedding space for modality $m_i$.
• Each edge $e=(i,j)$ carries a stalk $\F(e) = \R^{d_{ij}}$, the comparison space for modalities $m_i$ and $m_j$.
• Each incident (vertex, edge) pair is linked by a linear restriction map $\F_{ie}: \F(i)\to\F(e)$, represented as $\P_{ij}\in\R^{d_{ij}\times d_i}$.
• A global (compatible) section is an assignment $\{\h_i\}$ so that $\P_{ij}\h_i = \P_{ji}\h_j$ for each $e=(i,j)$.

These constructs enable modality pairs to agree in dedicated lower-dimensional comparison spaces, without constraining all modalities to inhabit the same set of representational factors. Agreement in these spaces identifies shared (redundant) content, while the null spaces of the projections localize unique information.

The sheaf Laplacian $L_\F$ encapsulates the global consistency of local sections, where the quadratic form $\h_n^T L_\F \h_n$ is a sum of squared deviations $\|\P_{ij}\h_{i,n} - \P_{ji}\h_{j,n}\|^2$ over all observation pairs.

3. Decentralized Contrastive and Reconstruction Objectives

Alignment is optimized with decentralized, edge-local losses:

• Edge-wise contrastive loss: On each $(i,j)\in\E$, local embeddings are projected via $\P_{ij},\P_{ji}$ into $\F(e)$, and paired with co-observed samples. An InfoNCE-style loss (cosine similarity, temperature $\tau$) encourages matched samples to be close in comparison space, while non-matches are pushed apart. For batch index $n$:

$\L_{\mathrm{contrast}}^{(e)} = -\frac{1}{B}\sum_{n=1}^B \log \frac{\exp(\mathrm{sim}(\p_{i,n}^{(e)},\p_{j,n}^{(e)})/\tau)}{\sum_{m=1}^B \exp(\mathrm{sim}(\p_{i,n}^{(e)},\p_{j,m}^{(e)})/\tau)}$

• Reconstruction loss (missing-modality inference): Each edge $e=(i,j)$ is equipped with a dual map $\Q_{ij}:\F(e)\to\F(i)$. Reconstruction MSE penalizes the difference between $\Q_{ij}\P_{ji}\h_{j,n}$ and $\h_{i,n}$, facilitating inference of missing node data from neighbors.
• Sheaf Laplacian loss (global consistency): The quadratic form above is added to encourage a global approximate section.

The aggregate loss for training is:

$\L_{\mathrm{total}} = \lambda \sum_{n=1}^B \h_n^T L_\F \h_n + \beta\sum_{e\in\E}\L_{\mathrm{contrast}}^{(e)} + \gamma\sum_{e\in\E}\L_{\mathrm{recon}}^{(e)}$

where $(\lambda, \beta, \gamma)$ are tunable hyperparameters.

These objectives can be computed and optimized locally at each node through neighbor-to-neighbor message passing with no server coordination. At each training step, nodes exchange projected embeddings, compute pairwise losses, and update their parameters locally via gradients.

4. Algorithmic Structure and Decentralized Training

Each node $i$ manages its own encoder parameters $\theta_i$ and edge-specific projection/reconstruction maps $\{\P_{ij},\Q_{ij}\}_{j\in\N(i)}$. The training process per epoch is as follows:

1. Draw a local minibatch $\{\x_{i,n}\}_{n=1}^B$, compute local embeddings $\h_{i,n}$.
2. For each neighbor $j$, project embeddings into $\F(e)$ and exchange projected representations.
3. Compute local edgewise losses (contrastive, reconstruction, Laplacian).
4. Aggregate these losses into a node-local loss $\L_i$.
5. Update all local parameters via a gradient step.

No central server is needed; updates are determined exclusively by neighbor interactions. The decentralized protocol reduces communication, preserves privacy, and facilitates scalable alignment in distributed sensor networks (Ghalkha et al., 23 Oct 2025).

5. Experimental Evaluation

SheafAlign was empirically validated across three main benchmarks:

• DeepSense Blockage Prediction: Modalities included RGB image, 2D LiDAR, and RF power for binary blockage classification.
• Multi-view MNIST: Three synthetic image views with controlled redundancy: original, edge-filtered, pixel-inverted.
• Semantic Inpainting: 3 camera images and CSI from 9 sensors, totaling 12 modalities.

Baselines included ImageBind (shared-image-space alignment) and fully supervised methods using all available labels.

Results demonstrated:

• Zero-shot/few-shot accuracy gains: On DeepSense and MNIST, SheafAlign outperformed ImageBind by ≥5 percentage points in zero-shot generalization and maintained superiority for up to 10 shots.
• Cross-modal retrieval enhancement: On MNIST, Recall@1 improved by 20% and Recall@10 by 18%; for Semantic Inpainting, average recall improved by 10%.
• Robustness and communication cost reduction: Under sensor dropout ($P_{\mathrm{drop}}=0.1, 0.01$), SheafAlign reduced bytes-transmitted by approximately 50% relative to ImageBind for the same or better test accuracy.

$P_{\mathrm{drop}}$	Algorithm	Accuracy	Transmitted Bytes [KB]
0.1	SheafAlign	0.87	46.20
0.1	ImageBind	0.83	92.48
0.01	SheafAlign	0.90	4.86
0.01	ImageBind	0.84	9.72

SheafAlign thus demonstrates improved robustness to partial observability and a substantial reduction in inter-node communication overhead (Ghalkha et al., 23 Oct 2025).

6. Theoretical Properties, Insights, and Limitations

The sheaf-theoretic formulation enables each comparison space to encode only the pairwise redundancy necessary for each modality intersection, explicitly modeling both shared and unique information. Local consistency is enforced only where data overlap occurs, and the dual reconstruction maps allow for missing modality inference via low-dimensional messages. Minimizing the edgewise contrastive loss maximizes a lower bound on mutual information between paired projections (InfoNCE), while the smallest nonzero eigenvalue of the sheaf Laplacian quantifies the alignment difficulty of the system.

Key limitations include dependency on a fully connected communication graph, the need to select appropriate loss weighting hyperparameters $(\lambda,\beta,\gamma)$, and learning/optimization overhead from maintaining multiple projections and dual maps. A plausible implication is that, in extremely sparse or dynamic networks, managing these structures may pose additional challenges.

SheafAlign thus generalizes the alignment problem from a monolithic global embedding space to a network of pairwise, sheaf-structured comparison spaces with decentralized learning, providing improved utility, robustness, and efficiency in distributed multimodal scenarios (Ghalkha et al., 23 Oct 2025).