CSMGAN: Cross & Self-Modal Graph Attention Network

• The paper introduces CSMGAN, a neural framework that uses cross- and self-modal graph attention to enable structured multimodal reasoning across diverse data types.
• It employs a hierarchical, multi-scale encoding strategy with alternating cross-modal and self-modal message passing to capture and align both global and local features.
• Ablation studies and benchmark results demonstrate that CSMGAN significantly improves performance metrics for video moment localization and 3D point cloud completion tasks.

A Cross- and Self-Modal Graph Attention Network (CSMGAN) is a neural framework for structured multimodal reasoning, integrating attention-based message passing across graph-structured representations within and between distinct modalities. The CSMGAN formalism enables the capture of both cross-modal interactions (e.g., between language and vision, or between point cloud geometry and image context) and self-modal relationships (e.g., intra-modal geometric consistency or temporal cues) at multiple levels of granularity, with high parameter sharing and explicit attention mechanisms. This architecture has been leveraged in diverse domains, exemplified by its application to video moment localization via language queries (Liu et al., 2020), and to cross-modal 3D point cloud completion (Zeng et al., 17 Sep 2025).

1. Architectural Foundations

A CSMGAN generally comprises two central graph components: the Cross-Modal Graph (CMG) and the Self-Modal Graph (SMG). In the CMG, nodes from distinct modalities (such as video frames and query words, or 3D points and image patches) are connected by directed, attention-weighted edges, supporting bi-directional exchange of information. In SMG, nodes within the same modality are linked by attention-based edges to model intra-modal relations such as temporal structure or geometric continuity.

Layer stacking—alternating or nesting CMG and SMG message-passing operations—enables the network to capture high-order dependencies and alignments across and within modalities. In practical instantiations, this approach is paired with hierarchical or multi-scale feature extraction pipelines, such as hierarchical point set encoders or multi-level sentence encoders.

2. Message Passing and Attention Mechanisms

Within CSMGAN, message passing between graph nodes is governed by parameterized attention mechanisms. For the cross-modal (CMG) component, attention weights are computed by projecting features of each modality via learnable matrices, resulting in an affinity matrix that determines the weight of each cross-modal edge. Formally, for encoded query features $H^{l-1}(Q) \in \mathbb{R}^{N_q \times d}$ and video/frame features $H^{l-1}(V) \in \mathbb{R}^{N_v \times d}$,

$E^l=(H^{l-1}(Q)\,W_q)\,(H^{l-1}(V)\,W_v)^\top$

is computed, and edge weights normalized per target node. The resulting messages are gated (e.g., by a sigmoid gate), aggregated, and the node representations are updated, frequently using recurrent schemes such as ConvGRU to preserve temporal or geometric continuity.

The self-modal (SMG) edges similarly aggregate intra-modal cues by attention-weighted summation, optionally incorporating positional encodings (e.g., sinusoidal) for structure sensitivity.

Multi-head attention, stacking of multiple message-passing layers (with $L=2$ empirically balancing complexity and information diffusion), and hierarchical abstraction are common features in these implementations.

3. Hierarchical and Multi-Scale Encoding

A distinguishing feature of CSMGANs is multilevel abstraction within each modality:

• In cross-modal 3D completion (Zeng et al., 17 Sep 2025), a Hierarchical Graph Attention (HGA) Encoder first extracts local (fine) and global (coarse) features from the point cloud, using a sequence of self-modal attention-based downsampling and neighborhood aggregation modules:
  • The Graph Descriptor (GD) Module computes edge embeddings $e_{ij} = \phi(p_i - p_j, f_i, f_j)$ for $K_1$ nearest neighbors, refined by convolutions, normalization, and pooling.
  • The Graph Attention Downsampling (GAD) Module scores nodes $w_i = \psi(f_i)$ and selects critical points, maintaining key geometric structure.
  • These modules are recursively applied, forming a hierarchy—preserving edge, corner, or high-curvature critical points at each level.
• In query-based moment localization (Liu et al., 2020), a hierarchical sentence encoder constructs contextual query representations at word, phrase, and sentence levels using convolutions and BiGRUs, enabling enhanced modeling of temporal operators and nuanced semantics.

This hierarchical construction ensures that both cross-modal and self-modal interactions operate on representations tailored to capture semantic or structural relationships at appropriate resolutions.

4. Cross-Modal Fusion and Alignment

The core of CSMGAN’s capability to fuse modalities lies in attention-based fusion modules. In 3D completion, a Multi-Scale Cross-Modal Fusion (MSCF) module projects global and local geometric features and image features to a common latent dimension, applies intra-modal and cross-modal self- and cross-attention: $$F'_g = \Psi_g(F_g),\, F'_l = \Psi_l(F_l),\, F'_I = \Psi_I(F_I)$$

$$\mathrm{Attention}(Q,K,V) = \mathrm{Softmax}\left(\frac{QK^\top}{\sqrt D}\right)V$$

This allows the model to align overall geometric context (object silhouette, global shape) and local details (semantic parts) from vision, with geometric features from point sets, yielding completion outputs that respect both modalities’ constraints.

In video localization, a joint representation of moment candidates is produced by fusing temporal and query features with cross-modal attention, whose outputs drive segment scoring and boundary regression.

Contrastive supervision (InfoNCE-style loss) may be used to explicitly minimize cross-modal feature distribution gaps, enhancing alignment and robustness to modality discrepancy.

5. Loss Functions and Training Strategies

CSMGAN training objectives typically combine application-specific reconstruction or localization losses with auxiliary constraints that promote cross-modal alignment:

• For point cloud completion, the loss is a weighted sum:

$$\mathcal{L}_\mathrm{total} = \lambda_{CD_2}\,\mathcal{L}_{CD_2} + \lambda_{\mathrm{contrast}}\,\mathcal{L}_{\mathrm{contrast}}$$

where $\mathcal{L}_{CD_2}$ is a bidirectional Chamfer Distance, and $\mathcal{L}_{\mathrm{contrast}}$ is an InfoNCE-based batch contrastive loss over paired global geometry and image features.

• For temporal moment localization, the composite loss incorporates an IoU-weighted alignment term and a boundary-regression loss over predicted segment boundaries.

Optimization is conducted on established datasets (e.g., ShapeNet-ViPC, YCB-Complete for 3D, Activity Caption and TACoS for videos), with standard Adam optimizer settings and minimal data augmentation.

6. Performance and Ablation Insights

Empirical results indicate substantiated improvements on relevant benchmarks. In 3D point completion,

• HGACNet (containing CSMGAN) achieves mean CD=1.002 and F-Score=0.887 on ShapeNet-ViPC, surpassing prior SOTA (e.g., EGIINet) by 17% in Chamfer Distance and 5.1 in F-Score,
• On YCB-Complete, known objects CD=0.073 ×10⁻³, F=0.995; unknown objects CD=7.996 ×10⁻³, F=0.405.

Ablation studies in both domains substantiate the necessity of each module: | Ablation | Completion CD (airplane/cabinet/car/watercraft, ×10⁻³) | Video Loc. R@1,IoU=0.3 (Δ) | |-----------------------------|-------------------------------------------------------|----------------------------| | − MSCF or CSG | 0.632/1.764/1.619/0.834 | ▼4.4% | | − local/self-modal features | 0.546/1.711/1.533/0.800 | ▼2.0% | | − C-Loss | 0.512/1.634/1.439/0.793 | | | − Hierarchical encoder | | ▼2.2% |

A plausible implication is that joint operation of cross- and self-modal attention is fundamental to the superior discriminative and generative performance observed.

7. Application Scope and Context

CSMGAN architectures are broadly applicable to tasks requiring intricate cross-modal reasoning with strong intra-modal dependencies. The approach generalizes across domains (vision-language, vision-geometry) and is extensible to multi-scale, hierarchical settings. Notably, the joint graph paradigm aids in precise region localization within untrimmed videos in response to language queries, and in reconstructing geometrically plausible 3D object models from incomplete observations using visual priors.

The demonstrated results on benchmarks establish CSMGAN as a robust scaffold for future research in both structured multimodal understanding and completion. Adoption in downstream applications such as robotic perception, manipulation planning, and open-world video understanding is supported by quantitative and qualitative evidence in the referenced works.