Multi-Modal Hypergraph Information Completion

• The paper introduces a novel, end-to-end differentiable model employing spectral hypergraph convolution and gated temporal convolutions to complete missing entries in multi-modal event streams.
• It utilizes automated hypergraph construction to capture higher-order spatial correlations among sensor channels, enabling accurate imputation over complex spatiotemporal data.
• The framework’s modular design and masking-aware objectives facilitate robust reconstruction and interpretability, offering potential applications in industrial soft sensing and beyond.

Multi-modal hypergraph-based information completion refers to a class of end-to-end differentiable models and methodologies that leverage higher-order graph (hypergraph) structures to impute or reconstruct missing values in multi-channel event or sensor streams—especially those exhibiting complex spatiotemporal dependencies and partial observations. The paradigm is characterized by (1) explicit modeling of higher-order, multi-node relationships beyond simple pairwise interactions, (2) integration of both spatial and temporal feature learning via deep neural operators, and (3) masking-aware completion objectives, such that information is completed only at locations identified as missing throughout the pipeline. The ST-HCSS framework exemplifies this approach, combining automated hypergraph construction, spectral hypergraph convolution, and gated temporal convolutions to simultaneously achieve accurate event completion and interpretable modeling of sensor interactions (Tew et al., 2 Jan 2025).

1. Hypergraph Construction from Multi-Modal Event Streams

The initial step involves constructing a hypergraph to capture the latent spatial correlations among $M$ input modalities—often sensor channels or multi-view features—over a temporal horizon of $T$ steps. For each node $i$, a feature embedding $v_i \in \mathbb{R}^d$ is computed using either a multi-layer perceptron or direct normalization of the historical series $x_i \in \mathbb{R}^T$. Pairwise Euclidean distances $D(i, j) = \|v_i - v_j\|_2$ establish similarity relations.

For each centroid node $j$, the $k$ nearest neighbors $KNN(j)$ are selected, yielding a set of hyperedges where each hyperedge $e_j$ connects the centroid and its neighbors. The incidence matrix $H_{i,e_j}$ is defined such that $H_{i,e_j}=1$ if $i \in KNN(j)$ and $0$ otherwise; corresponding weights $W_{i,e_j} = \exp(-D(i,j)^2/\Delta)$ (with $\Delta$ as the mean pairwise distance) are assigned, encoding the strength of each membership. Vertex and hyperedge degree matrices $D_v$ and $D_e$ summarize per-node and per-hyperedge connectivities, and the normalized adjacency is computed as $N = D_v^{-1/2} H W D_e^{-1} H^T D_v^{-1/2}$. This automated procedure allows latent, data-driven hypergraph construction even in the absence of explicit domain structure (Tew et al., 2 Jan 2025).

2. Spectral Spatio-Temporal Hypergraph Convolution

Within the model, hypergraph convolution layers propagate and mix information across nodes as determined by the learned hypergraph. The operation follows the spectral approach:

$X^{(l+1)} = \mathrm{ReLU}(N X^{(l)} \Theta^{(l)})$

where $X^{(l)}$ is the input node-feature matrix at layer $l$, $\Theta^{(l)}$ is a learnable weight matrix, and $N$ is the normalized adjacency. The hypergraph Laplacian $L=I-N$ defines the spectral domain over which nonlocal interactions are aggregated, enabling the capture of higher-order dependencies not expressible in simple graph structures. Stacked hypergraph convolutional layers progressively refine node representations by integrating context from multi-node sets at each step (Tew et al., 2 Jan 2025).

3. Gated Temporal Convolutions for Sequential Dependency Modeling

Temporal dynamics are modeled by gated temporal convolution (GTC) modules. For each node, the time series is regarded as a one-dimensional input, to which a causal convolution is applied with kernel size $K$ (typically $K=7$) and dilation $d=1$:

$$h^{k+1}_i(t) = [w^k_f * x^k_i](t) \odot \sigma([w^k_g * x^k_i](t))$$

Here, $w^k_f$ and $w^k_g \in \mathbb{R}^K$ are the feature and gate kernels, $*$ indicates convolution, $\odot$ is elementwise multiplication, and $\sigma$ denotes the sigmoid nonlinearity. The gating mechanism allows selective temporal information flow, enhancing the model’s ability to capture long-range dependencies and varying temporal influences across nodes. GTC operations precede the hypergraph convolution at each block, providing temporally-processed inputs for spatial (multimodal) aggregation (Tew et al., 2 Jan 2025).

4. End-to-End Data Flow and Block Design

The model architecture ingests $X \in \mathbb{R}^{M \times W}$, where $W$ is the time-window, and optionally applies a multi-view mixer comprising two MLPs: one for time-mixing along $W$, the other for feature-mixing across $M$. The main body consists of $L$ (typically $L=3$) sequential blocks, each containing:

• Gated Temporal Convolution: transforms node features temporally ($M \times W \rightarrow M \times W'$),
• Hypergraph Convolution: mixes across nodes via the constructed hypergraph ($M \times W' \rightarrow M \times C_{l+1}$),
• Optional Residual: input summation if feature dimensions align.

The output feature $Z \in \mathbb{R}^{M \times C_L}$, after $L$ blocks, is projected to reconstruct $\hat{Y} \in \mathbb{R}^{M \times W'}$ via a final MLP. Dimensions flow as: $M \times 85$ (input) $\rightarrow$ $GTC$ $M \times 85$ $\rightarrow$ $HGC$ $M \times 64$ $\rightarrow$ $GTC$ $M \times 64$ $\rightarrow$ $HGC$ $M \times 32$ $\rightarrow$ ... $M \times W$ (output). This modular structure promotes both temporal and multi-modal feature fusion at multiple scales.

5. Masked Event Completion Mechanism

A binary mask $\mathrm{mask} \in \{0,1\}^{M \times W}$ indicates missing (0) versus observed (1) entries in $X$. Missing values are zero-filled prior to model ingestion, with the mask information propagated through the network or incorporated into attention or other modulation schemes. The final loss is computed solely on missing locations:

$$L_{\mathrm{rec}} = \sum_{i,t}(1-\mathrm{mask}_{i,t})\cdot (\hat{Y}_{i,t}-X^{\text{true}}_{i,t})^2$$

This masked mean square error targets only unobserved entries, enforcing selective imputation while preserving existing data points. The network thus focuses its representation power on reconstructing incomplete event streams where information is absent (Tew et al., 2 Jan 2025).

6. Optimization, Regularization, and Hyperparameter Selection

Training employs Adam optimization (learning rate $1\times10^{-3}$, weight decay $1\times10^{-5}$, batch size 64), with dropout ($0.2$) and a training regime of 200 epochs (split $60\%$ train/$20\%$ val/$20\%$ test). Additional regularization includes explicit weight decay and, optionally, sparsity or entropy penalties over the hyperedge weights $W$ to promote concise, interpretable hypergraph structures if $H$ or $W$ are set as learnable. Sliding window $W=85$, $k=5$ nearest neighbors per centroid, $L=3$ blocks, temporal kernel $K=7$, and hidden dimensions $C_1=64, C_2=32, C_3=16$ constitute the canonical configuration. Mixer blocks (2, if used) further augment representational richness (Tew et al., 2 Jan 2025).

7. Implications, Extensions, and Significance

The ST-HCSS framework substantiates several key properties of multi-modal hypergraph-based information completion: (1) automatic discovery of higher-order spatial relations via latent hypergraph construction, (2) robust modeling of long-range temporal correlations through gating, (3) end-to-end targeted imputation in incomplete multi-modal event streams, and (4) full differentiability enabling joint optimization of all components. This architectural pattern addresses the challenge of missing data in sensor networks and other multi-source streams, accommodating unknown relationships and variable modalities within a principled, trainable structure. A plausible implication is applicability to other domains—beyond industrial soft sensing—where multi-modal, partially observed data require both interpretability and accurate recovery (Tew et al., 2 Jan 2025).