Cell Attention Networks in Graphs & RNNs

• Cell Attention Networks are neural architectures that apply cell-level attention to overcome limitations in graph and sequence models by capturing complex local and global interactions.
• The graph-based CAN model leverages regular cell complexes and dual neighborhood attention to integrate vertex, edge, and polygon information for robust message passing.
• The recurrent CAN architecture augments standard RNN cells with attention over past inputs, mitigating vanishing saliency and enhancing interpretability in time series.

Cell Attention Networks (CAN) describe a family of neural architectures that introduce cell-level attention mechanisms to generalize and overcome fundamental expressive limitations of standard graph, sequential, and recurrent neural networks. Two distinct streams of work employ the "Cell Attention Network" terminology: (1) a hierarchical framework for graph-structured data that exploits cell complexes, higher-order relations, and dual neighborhood attention (Giusti et al., 2022); and (2) an attention-augmented recurrent cell architecture to address the vanishing saliency in time-series interpretability tasks (Ismail et al., 2019). Both frameworks independently broaden the ability of neural architectures to flexibly attend over multi-relational or temporally deep structures, producing interpretable, permutation-invariant, and robust representations.

1. Cell Attention Networks for Graph Complexes

The CAN architecture for graph-based learning (Giusti et al., 2022) operates on data defined over the vertices of a graph by leveraging the concept of a regular cell complex. In this context, a regular cell complex $\mathcal{C}$ is composed of 0-cells (vertices), 1-cells (edges), and higher order $k$-cells (such as 2-cells representing polygons or cycles) that capture higher-order topological structures beyond pairwise interactions.

The central formalism represents the data domain as a 2-complex $$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$, with $\mathcal{V}$ the set of vertices, $\mathcal{E}$ the set of edges, and $\mathcal{P}$ the set of 2-cells (polygons/chordless cycles). Edge neighborhoods are then defined in two ways:

• Lower neighborhood $N_{\downarrow}(e)$ includes all edges sharing an endpoint with $e$.
• Upper neighborhood $N_{\uparrow}(e)$ contains all edges that co-occur with $e$ within a shared polygon/cell.

This dual-neighborhood formalism enables CAN to capture patterns inaccessible to standard graph attention networks, which operate only on pairwise node relations.

2. Layerwise Architecture and Attention Mechanisms

The CAN workflow layers several key operations:

1. Attentional Lifting: Node features $k$0 on the endpoints of each edge $k$1 are "lifted" into an edge feature $k$2 by a multi-head masked self-attention. Each attention head computes

$k$3

with symmetric parameterization if desired, and concatenation across $k$4 heads.

2. Cell Attention Mechanisms: At each layer $k$5, edge features $k$6 are updated by two masked self-attentions: one over the lower neighborhood $k$7, one over the upper $k$8. For each neighbor (lower or upper), scores are computed and normalized via softmax:

$k$9

$$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$0

The normalized weights $$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$1 are then used in the feature update:

$$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$2

3. Hierarchical Edge Pooling: After attention, edges are pooled according to scores $$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$3, keeping a top-$$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$4 subset. Remaining edges are rescaled and passed to the next layer.

The CAN layer sequence—dual attention, aggregation, pooling—enables scalable hierarchical message passing that incorporates both vertex-, edge-, and polygon-level information.

3. Computational Complexity and Implementation Insights

CAN achieves low computational overhead relative to its expressivity. Key complexity characteristics include:

• Preprocessing for 2-complex construction (enumerating polygons up to ring size $$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$5) is $$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$6 for constant $$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$7.
• Attentional lifting is $$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$8, i.e., linear in the number of edges and attention heads.
• Each attention layer has per-edge message cost linear in the number of neighbors, typically small on real data ($$\mathcal{C} = (\mathcal{V}, \mathcal{E}, \mathcal{P})$$9 lower, $\mathcal{V}$0 upper).
• Edge pooling is $\mathcal{V}$1.

On parallel hardware, each layer's sequential depth is $\mathcal{V}$2. This efficiency enables practical application to graphs with thousands of edges and higher-order structure (Giusti et al., 2022).

4. Empirical Evaluation and Interpretability

CAN was evaluated on five molecular graph classification benchmarks (MUTAG, PTC, PROTEINS, NCI1, NCI109) from the TU-Dataset using two-class accuracy via 10-fold cross-validation. The architecture outperformed or matched state-of-the-art GNNs (GIN, PPGN, CIN, etc.) on four of five tasks, e.g., achieving 94.1% on MUTAG (vs. 90.6% for PPGN), 72.8% on PTC (vs. 68.2% for CIN), and 78.2% on PROTEINS (vs. 77.2% for PPGN) (Giusti et al., 2022).

These results demonstrate that explicitly encoding higher-order structures and leveraging dual attention yields measurable gains, particularly in domains where cycles, motifs, and multi-node interactions are salient (e.g., molecular chemistry).

5. Cell Attention Networks in Recurrent/Sequential Models

A separate but convergent line of work employs the term "Cell Attention Network" to denote RNN cell architectures augmented with input-cell attention (Ismail et al., 2019). In this framework, each time step $\mathcal{V}$3 receives not only $\mathcal{V}$4, but also attends over all previous inputs via a fixed-size embedding $\mathcal{V}$5, with $\mathcal{V}$6 attention hops. The attention mechanism computes

$\mathcal{V}$7

and

$\mathcal{V}$8

with $\mathcal{V}$9 formed either by flattening or averaging $\mathcal{E}$0.

The hidden state and gates (in RNN, LSTM, GRU variants) are then computed as explicit functions of this context embedding. For LSTM, for example:

$\mathcal{E}$1

6. Addressing Vanishing Saliency in Sequence Models

Traditional RNNs and LSTMs suffer from vanishing saliency—saliency maps become increasingly localized to the final time steps due to the vanishing gradient effect in backpropagation. In a standard model,

$\mathcal{E}$2

exhibits exponential decay with time lag $\mathcal{E}$3. The input-cell attention network architecture introduces a direct path from the final output to all past $\mathcal{E}$4 via $\mathcal{E}$5, yielding a persistent gradient term

$\mathcal{E}$6

thus maintaining saliency for early and late inputs (Ismail et al., 2019).

Experiments on synthetic box-embedding time series showed that CAN outperformed standard and self-attention RNNs in terms of normalized Euclidean and Jaccard similarity to ground truth saliency. In neuroscience fMRI applications (task/rest state classification across cortical parcellations), CAN saliency correctly localized to on-task windows and relevant brain ROIs, while standard RNNs misattributed importance to off-task periods (Ismail et al., 2019).

7. Extensions, Limitations, and Future Directions

For graph CAN, future developments include generalization to higher-dimensional complexes, multi-head attention at the cell level, and integration with message passing on non-Euclidean domains. Limitations include the computational cost of 2-complex construction and potential scaling issues for extremely large or dense graphs, though practical runtimes are competitive for most bioinformatics and cheminformatics benchmarks (Giusti et al., 2022).

For sequential CAN, challenges include quadratic attention overhead for long sequences and memory demands for non-averaged multi-hop embeddings. Natural extensions comprise learnable recency biases, multi-head attention, strict windowing, and Transformer-style cross-attention in encoder-decoder settings (Ismail et al., 2019).

Both streams represent a fundamental advance in capturing higher-order or temporally distributed relationships, with interpretability and performance benefits in domains where rich structural or spatiotemporal interactions are fundamental.