GCN-LSTM Interaction Modules

• GLI modules are neural network components that merge graph convolutional and LSTM architectures to capture both relational and temporal dependencies.
• They integrate sequential processing with attentional graph propagation through two variants— aGCN-out-LSTM and aGCN-in-LSTM—enhancing sentence summarization in video analysis.
• Empirical studies on benchmarks like ActivityNet and YouCook II demonstrate the model’s ability to generate context-aware, accurate event-level captions.

A GCN-LSTM Interaction (GLI) module is a neural network component that fuses the capabilities of Graph Convolutional Networks (GCN) and Long Short-Term Memory (LSTM) networks, specifically designed to capture both relational (graph-structured) and sequential (temporal) dependencies. In the context of dense video captioning, GLI modules address the challenge of effectively summarizing dynamic scene evolution within long, event-level video proposals—where both visual content and semantic progression are highly complex and non-uniform—by integrating graph-based and sequence-based modeling for sentence summarization.

1. Definition and Motivation of GLI Modules

GCN-LSTM Interaction modules, as proposed in the GPaS framework for dense video captioning, are designed to perform graph-based sentence summarization by explicitly modeling and refining the relationships among semantic words (or segments) via both sequential and graph-based propagation, with guidance from visual cues. The dense video captioning task divides long event proposals into finer-grained segments, generates a candidate sentence for each segment, and then employs GLI modules to summarize these segment-level sentences into a holistic event-level caption. The core problem addressed is the limitation of pure sequential or pure graph-based methods in capturing both local linguistic order and richer inter-word/segment dependencies, especially as scenes and objects evolve over long events.

2. Architectural Variants and Technical Integration

Two main schemes of GLI modules are introduced for dense video captioning:

• aGCN-out-LSTM: The GCN (“attentional GCN”, or aGCN) layer is applied outside of the LSTM cell. The LSTM processes word/segment sequences, and its hidden representations are then globally refined using GCN based on semantic relationships across segments or words. Specifically, an aGCN update of the hidden state for output word $i$ is:

$$\hat{\mathbf{h}}^d_i = \tanh\left(\mathbf{h}^d_i + \mathbf{W}^d \sum_{j=1}^{L_m \times L_k} \alpha^d_{ij} \hat{\mathbf{h}}^w_j\right)$$

where $\mathbf{h}^d_i$ is the decoder LSTM hidden state, and $\hat{\mathbf{h}}^w_j$ are encoder outputs from segment-level word nodes. The attention weights $\alpha^d_{ij}$ are learned by MLP-softmax mechanisms based on semantic similarity.

• aGCN-in-LSTM: The GCN is inserted inside the LSTM, operating directly on the cell states between LSTM steps, so that the sequential memory transitions are graph-refined:

$$\hat{\mathbf{c}}_i = \tanh\left(\mathbf{c}_i + \mathbf{W}\sum_{j\in \mathcal{P}_i} \alpha_{ij}\hat{\mathbf{c}}_j\right)$$

The refined cell state $\hat{\mathbf{c}}_i$ is then used, with the LSTM output gate, to compute the next hidden state and output. This arrangement allows the GLI module to propagate semantic and visual context simultaneously at both memory and representation levels.

Both modules rely on attentional propagation using aGCN, where the node update for semantic node $i$ is:

$$\hat{\mathbf{z}}_i = \tanh\left(\mathbf{z}_i + \mathbf{W} \sum_{j\in \mathcal{P}_i} \alpha_{ij}\mathbf{z}_j\right)$$

with attention scores $\alpha_{ij}$ computed via softmax over learned MLPs of node features.

3. Integration within the GPaS Summarization Framework

Within the GPaS (Graph-based Partition-and-Summarization) pipeline, the GLI module sits at the heart of the summarization stage:

1. The video event proposal is partitioned into segments.
2. Segment-level sentences are generated, with each word or entire sentence treated as a node in a graph.
3. The GLI module processes these nodes, leveraging both the sequential (LSTM) connections within segments and graph (GCN) connections across words/segments.
4. Visual cues (segment-level features) are attended to and fused into the node representations at every step, using mechanisms such as:

$$\mathbf{f}^{TVJ}_t = \textrm{MLP}(f_{att}(\mathbf{V}, \mathbf{h}_{t-1}), e(\mathbf{w}_t))$$

where $f_{att}$ computes an attention-weighted sum over the visual segment features $\mathbf{V}$.

The GLI modules support hierarchical graphs, allowing expanded models where segment-level nodes act as intermediaries between word and event-level nodes, enabling finer granularity in semantic integration.

4. Empirical Performance and Comparative Evaluation

Experiments demonstrate that GLI modules, particularly the aGCN-in-LSTM-expanded variant, yield substantial improvements over prior dense video captioning approaches. On the ActivityNet Captions and YouCook II datasets:

• GPaS with GLI modules achieves top Meteor and CIDEr-D scores, e.g., 11.04% Meteor and 28.20 CIDEr-D on ActivityNet ground truth, outperforming baselines such as DVC, DCE, and Bi-AFCG.
• Ablation studies show that adding aGCN-in-LSTM and expanded graph modeling both contribute incremental performance gains over plain LSTM or GCN or transformer attention (TA) summarization.
• Qualitative analysis shows the GLI-based system generates more semantically accurate and contextually linked event-level captions, especially in long, scene-evolving events.

5. Visual Cue Utilization and Multimodal Fusion

The GLI modules tightly integrate visual and semantic information by fusing visual segment features with word or node embeddings before sequential and graph propagation. Visual attention is computed for each time step or segment node, with attended features entering the LSTM pathway and informing GCN attention. This multimodal fusion grounds language generation in actual scene content, increasing robustness to scene/object changes, and improving alignment between video and description.

6. Theoretical Implications and Model Significance

By coupling sequential LSTM modeling with explicit graph-based semantic message passing, GLI modules overcome the limitations of conventional approaches that process sentences or words in linear order or ignore inter-segment dependencies. The in-LSTM and out-LSTM variants provide flexibility in how graph interactions are injected into the sequence modeling: in-LSTM for deeper memory-level refinement, out-LSTM for explicit post-sequence, graph-aggregated hidden updates. This suggests GLI modules serve as a general bridge for multi-source, multi-level representation fusion in multi-modal sequence summarization.

7. Summary Table: GLI Module Schemes in Graph-based Sentence Summarization

Variant	GCN Placement	LSTM Component Modified	Summary
aGCN-out-LSTM	Outside LSTM	Hidden state	Refines LSTM output with GCN message
aGCN-in-LSTM	Inside LSTM	Cell state	Injects GCN message before LSTM output

Conclusion

GCN-LSTM Interaction modules, as realized in the GPaS framework, provide a principled mechanism for blending sequential and graph-relational modeling in dense video captioning. Through attentional graph propagation and tight visual-semantic integration, they enable end-to-end learning of holistic, context-aware summaries from temporally and visually complex input, showing empirically superior results against existing summarization architectures.