Concept Graph Convolutions: Message Passing in the Concept Space

Abstract: The trust in the predictions of Graph Neural Networks is limited by their opaque reasoning process. Prior methods have tried to explain graph networks via concept-based explanations extracted from the latent representations obtained after message passing. However, these explanations fall short of explaining the message passing process itself. To this aim, we propose the Concept Graph Convolution, the first graph convolution designed to operate on node-level concepts for improved interpretability. The proposed convolutional layer performs message passing on a combination of raw and concept representations using structural and attention-based edge weights. We also propose a pure variant of the convolution, only operating in the concept space. Our results show that the Concept Graph Convolution allows to obtain competitive task accuracy, while enabling an increased insight into the evolution of concepts across convolutional steps.

• The paper introduces Concept Graph Convolutions (CGC) as a novel operator that integrates raw latent embeddings with fuzzy, explicit concept encodings for transparent message passing in GNNs.
• It demonstrates that the CGC achieves competitive, and sometimes superior, accuracy compared to traditional GCNs while providing granular, layerwise interpretability of concept formation.
• Empirical evaluations reveal adaptive mixing of structural and attention-based weights via parameters γ and η, challenging the presumed accuracy–interpretability tradeoff in deep GNN architectures.

Concept Graph Convolutions: Explicit Message Passing in the Concept Space

Motivation and Context

Despite recent advances in graph neural networks (GNNs), the opacity of their reasoning processes presents significant obstacles for trustworthy deployment, especially in domains necessitating rigorous interpretability. Existing methods for concept-based explanations in GNNs primarily focus on extracting concepts post hoc by clustering end-of-network latent representations. However, these approaches fail to provide insight into the message passing process itself—the core computational mechanism of GNNs—where usually only aggregate node embeddings are available, eliding any explicit conceptual trajectory across layers.

The paper "Concept Graph Convolutions: Message Passing in the Concept Space" (2604.20082) addresses this deficiency by proposing the Concept Graph Convolution (CGC), a novel graph convolutional operator designed to support direct message passing over both raw and explicit concept representations. This enables the tracing of concept formation and interaction at each layer, conferring granular interpretability without sacrificing predictive power.

Methodology

The CGC layer is designed to combine the expressiveness of raw latent node embeddings with the interpretability afforded by concept encodings. Nodes are augmented with both (i) a standard latent feature vector and (ii) a fuzzy membership encoding in a learned concept space derived via a scaled softmax over the latent features.

On each layer, both latent and concept representations are projected via a shared transformation, then mixed using a learnable scalar $\eta$ (regularized toward concept reliance for interpretability). Edge weights for neighborhood aggregation are computed via a convex combination (weighted by learnable $\gamma$) of normalized adjacency (structural) and single-head concept-based attention mechanisms. After message passing, output embeddings are renormalized to form the new concept space. The pure CGC variant omits latent embeddings and operates solely in the concept domain, maximizing interpretability at the cost of some flexibility.

This mechanism is illustrated as follows:

Figure 1: The CGC processes raw and concept embeddings, computes structural and attention-based edge weights, and propagates messages to yield refined concept encodings. The pure CGC omits latent node embeddings.

Edge Weight Computation

• Structural weights: Normalized adjacency.
• Concept-based attention weights: Static single-headed attention over concept encodings.
• Edge weight convex combination: $$w_{ij} = (1-\gamma)\tilde{A}_{ij} + \gamma \alpha_{ij}$$, where $\gamma$ is trainable.

The hyperparameters $\eta$ and $\gamma$ are regularized to encourage increased reliance on the concept and attention components while preserving learnability.

Empirical Study and Results

Datasets and Baselines

Evaluation comprises both synthetic (BA-Shapes, BA-Community, BA-Grid, Tree-Cycle, Tree-Grid, STARS, House-Colour) and real-world (Mutagenicity, Reddit-Binary) node/graph classification benchmarks, including cases with explicit ground-truth graph motifs to allow fine-grained concept auditing.

CGC and pure CGC models are compared against standard GCN and GAT architectures, as well as relational concept bottleneck models (R-CBMs), though the latter operate at the full-architecture level rather than as graph convolutional layers.

Accuracy–Interpretability Tradeoff

Key results show that networks constructed with CGC layers attain accuracy competitive with standard GCNs, even outperforming them on several synthetic node and graph classification tasks. Mixing of latent and concept representations is generally beneficial for predictive performance, while the pure CGC achieves its highest accuracy on tasks where pure structural motifs are decisive.

A critical finding is that the inclusion of explicit concept processing does not degrade task accuracy, thereby challenging prevailing beliefs about an inherent accuracy–interpretability tradeoff for GNNs at the level of convolutional design.

Completeness and Evolution of Concepts

Concept completeness—the sufficiency of discovered concepts for explaining model predictions—monotonically increases across CGC layers, indicating that deeper message passing extracts progressively more discriminative (and interpretable) higher-order concepts, up to limits imposed by oversmoothing. This is quantitatively demonstrated across several tasks:

Figure 2: Concept completeness scores for CGC and pure CGC layers, showing increased discriminative capacity across layers aligned with receptive field growth.

A similar trend is observed for graph classification, although on tasks requiring multi-hop or global context, deeper layers lead to diminishing returns, highlighting oversmoothing-induced collapse in concept diversity.

Analysis of Control Parameters

• Mixing Parameter $\gamma$: Empirically, earlier layers bias toward structural aggregation ($\gamma$ low), while deeper layers allocate increasingly higher mass to attention over neighboring concept encodings, modulated by task structure and dataset properties.

Figure 3: Layerwise $\gamma$ values for CGC and pure CGC across benchmarks, illustrating the model's adaptive balancing of structure versus attention during propagation.

• Mixing Parameter $\eta$: Concept reliance, controlled by $\gamma$0, remains stable but seldom approaches full dominance except under strong regularization, suggesting that some unconstrained latent information remains beneficial even in regimes prioritizing interpretability.

Qualitative Visualization

The framework supports direct visualization and tracking of concept formation and aggregation at different positions and depths in the graph. For example, layerwise extraction surfaces interpretable motifs (e.g., 'bottom of a house' in BA-Shapes), with intermediate layers reflecting partial motifs or lower-order structures.

Figure 4: Example of concept identification at the second pure CGC layer; subgraphs show receptive field and attention allocation for the concept-represented node.

Such visualizations generalize to other datasets, including detection of desired chemical substructures (e.g., $\gamma$1 motifs in Mutagenicity), rare or fine-grained concepts (e.g., distinct color patterns in House-Colour), and compositional motifs in community and grid-based graphs.

Theoretical and Practical Implications

The introduction of a concept-based graph convolutional operator enables, for the first time, explicit tracking and auditing of conceptual state and influence at every step of GNN propagation. This is a marked improvement over post hoc concept explanation approaches, which typically fail to illuminate the dependencies established during message passing.

• Theoretical Implication: CGC directly operationalizes the principle of concept bottlenecks at the level of graph convolutions, not merely as architectural or post-analysis artifacts. This supports new formalizations of completeness, multi-level conceptual abstraction, and layerwise auditing relevant to XAI for GNNs.
• Practical Implication: The model design enables feature tracing, motif auditing, and anticipates regulatory requirements for explainability in high-stakes applications of GNNs (e.g., molecule property prediction, social network analysis).

For wider deployment, the approach can be integrated with hierarchical pooling or symbolic aggregation methods, supporting multi-scale interpretable reasoning. As concept completeness and accuracy are tightly coupled with network depth, regularization and adaptive depth control become necessary to avoid oversmoothing-induced interpretability loss in deep GNNs.

Limitations and Future Directions

A salient limitation is the repeated application of scaled softmax for promoting discrete concept formation, which may induce information loss and degrade gradients in deep architectures. Future refinement of the concept embedding mechanism—potentially with learnable, hierarchical, or information-theoretic regularization—could improve both stability and concept granularity. Additionally, broader adoption requires further work on (i) robust quantification of concept "quality" and (ii) extension to tasks such as sequence modeling or dynamic graphs.

Conclusion

Concept Graph Convolution establishes a principled framework for interpretable GNNs wherein every message passing step is rendered in explicit conceptual terms, reconciling the requirements of accuracy and transparency without resorting to post hoc analysis. The approach provides not only node-wise and subgraph-wise explanations but also a substrate for investigating the dynamics of concept formation in relational deep learning. It opens new avenues for architectures that are inherently interpretable, modular, and amenable to human inspection during both model development and deployment.