Parametric Grid Convolutional Attention Networks

• PGCANs are neural architectures that combine a trainable grid encoder, localized convolution, and attention-based decoder to solve PDEs without labeled data.
• They mitigate spectral bias by focusing on local interactions and high-frequency regions through dynamic balancing of loss terms.
• Variants of PGCANs extend to graph classification and neuroimaging, demonstrating their flexibility across diverse spatially structured data.

Searching arXiv for the specified papers to ground the article in the cited literature. Parametric Grid Convolutional Attention Networks (PGCANs) are neural architectures that combine grid-structured parameterization, localized convolution, and attention-based decoding or gating. In the formulation introduced for neural partial differential equation (PDE) solving, PGCANs are physics-informed networks designed to mitigate spectral bias by parameterizing the input domain with a trainable grid encoder whose vertex features are convolved, interpolated at query points, and mapped to the solution through an attention-based decoder, all trained without labeled data by minimizing physics residuals and boundary or initial condition losses (Shishehbor et al., 2024). Related work uses the same grid–convolution–attention pattern in graph classification and volumetric neuroimaging, where graph structure is normalized into a 2D grid or 3D feature maps are modulated by spatial attention, suggesting that PGCAN is best understood as an architectural family rather than a single fixed layer stack (Peng et al., 2018).

1. Architectural definition and parametric structure

In the PDE setting, PGCAN consists of three coupled components: a parametric grid encoder, a convolutional component acting on grid features, and an attention-based decoder. The encoder partitions the input space into cells on a single-resolution “mother” grid whose vertices carry trainable feature vectors. In two dimensions, the feature tensor is

$$F_0 \in \mathbb{R}^{n_{\mathrm{rep}} \times N_f \times N_v^x \times N_v^y},$$

where $n_{\mathrm{rep}}$ is the number of diagonally shifted grid repetitions, $N_f$ is the feature dimension, and $N_v^x \times N_v^y$ is the vertex lattice. The central design choice is locality: unlike fully connected networks whose parameters globally affect outputs, the grid parameterization restricts most interactions to nearby cells and vertices (Shishehbor et al., 2024).

The convolutional stage applies a feature-wise $3 \times 3$ kernel with $\tanh$ activation to each feature map. For repetition $p$ and feature index $q$,

$$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$

with $W_q \in \mathbb{R}^{3\times 3}$. This localized operator bidirectionally propagates information from boundaries into the interior, induces spatial coherence among neighboring vertices, and reduces parameter counts needed for fine resolutions by improving feature sharing through locality.

At a query point $n_{\mathrm{rep}}$0, the convolved vertex features are mapped to a continuous representation by cosine-warped bilinear interpolation. If $n_{\mathrm{rep}}$1 are local cell coordinates normalized to $n_{\mathrm{rep}}$2, the warp is

$n_{\mathrm{rep}}$3

The interpolated feature for repetition $n_{\mathrm{rep}}$4 is

$n_{\mathrm{rep}}$5

and the aggregate encoding is

$n_{\mathrm{rep}}$6

The feature vector is split into two equal parts, $n_{\mathrm{rep}}$7 and $n_{\mathrm{rep}}$8, which are passed to the decoder.

The decoder uses gated mixing and self-attention. With $n_{\mathrm{rep}}$9 and $N_f$0,

$N_f$1

$N_f$2

Self-attention is introduced through scaled dot-product attention,

$N_f$3

with learned projections of encoded features. In the PDE formulation, the stated role of self-attention is to focus training signal on cells or features where residuals are large, including high-frequency or high-gradient regions.

This parametric structure has different realizations in other domains. In motif-based graph classification, the parametric components are the convolution kernels, attention parameters, and classifier, while graph-to-grid normalization is fixed preprocessing (Peng et al., 2018). In 3D neuroimaging, the parametric components are a compact 3D CNN plus spatial attention gates computed by learnable $N_f$4 convolutions conditioned on global context (Lam et al., 2020). This suggests that “parametric” does not necessarily mean that the grid itself is learned; rather, it denotes that the convolutional and attentional operators acting on grid-structured representations are learned.

2. PDE formulation and training objective

The PDE formulation treated by PGCAN is stated generically as

$N_f$5

with boundary condition

$N_f$6

For time-dependent problems, the more general form is

$N_f$7

Training is label-free in the domain: PGCAN minimizes a physics residual loss and boundary or initial condition losses rather than fitting supervised solution pairs (Shishehbor et al., 2024).

For a collocation set $N_f$8, the residual loss is

$N_f$9

For a boundary collocation set $N_v^x \times N_v^y$0,

$N_v^x \times N_v^y$1

For time-dependent problems, the initial condition loss is

$N_v^x \times N_v^y$2

The total loss is

$N_v^x \times N_v^y$3

with dynamic weight balancing rather than an explicit spectral penalty.

The balancing rule is designed to align the gradient magnitudes of auxiliary losses with the PDE residual:

$N_v^x \times N_v^y$4

where $N_v^x \times N_v^y$5. The stated purpose is to prevent PDE residuals from dominating training dynamics and thereby underfitting boundary or initial conditions.

The implementation guidance given for PGCAN follows a specific sequence. One defines the PDE operator $N_v^x \times N_v^y$6, boundary operator $N_v^x \times N_v^y$7, domain $N_v^x \times N_v^y$8, and collocation sets; initializes grid features, convolution kernels, and decoder parameters; resamples interior and boundary points periodically; computes interpolated features and decoder outputs at collocation points; forms residual and BC or IC losses; updates the dynamic weights; and optimizes all learnable parameters with Adam. The recommended settings repeatedly used across benchmarks are a $N_v^x \times N_v^y$9 grid, $3 \times 3$0, $3 \times 3$1, a 3-layer decoder with 64 neurons per layer, and a total of approximately $3 \times 3$2k trainable parameters for scalar outputs.

3. Spectral bias, locality, and frequency-domain diagnosis

The motivating problem for PGCAN is spectral bias in neural PDE solvers. Spectral bias is described as the tendency of standard deep neural networks, especially fully connected feed-forward networks used in vanilla physics-informed neural networks (PINNs), to learn low-frequency components first while struggling with high-frequency features and localized sharp gradients. In PDE contexts, this leads to poor approximation of oscillatory solutions, stiff responses, and boundary-layer gradients, precisely where fidelity is often most consequential (Shishehbor et al., 2024).

The mechanism proposed in the PGCAN paper has three parts. First, local parameterization reduces the global coupling induced by shared parameters in fully connected networks. Second, convolution introduces spatial coherence, suppresses salt-and-pepper overfitting, and improves propagation of informative gradients from boundaries and initial conditions into interior regions. Third, attention selectively emphasizes features associated with large residuals or high curvature, steering optimization toward the hard components of the solution. No formal theorems are provided; the paper explicitly presents these as theoretical insights supported by empirical behavior and notes that the observations align with NTK-informed views of gradient pathologies in PINNs.

To quantify spectral bias, the paper introduces a directional power spectral density (PSD) analysis of the error field. Given an error “image” $3 \times 3$3, one computes the discrete Fourier transform and its power

$3 \times 3$4

The power is averaged along axes to obtain directional PSD curves, and the curves are normalized to start at a common initial value so that they reflect the distribution of error across frequencies rather than total power. The interpretive criterion is explicit: flatter curves indicate weaker spectral bias. In this frequency-domain view, PGCAN’s directional PSD curves are reported as consistently flatter than those of vPINN, PIXEL, and often M4, while also having lower overall error power.

A common misunderstanding is to equate spectral-bias mitigation with the addition of a frequency-domain regularizer. That is not the mechanism used here. The work states that there is no explicit spectral penalty; the mitigation is attributed to architectural locality, convolutional regularization, attention-based prioritization, and dynamic balancing of loss terms. Another potential misunderstanding is that performance gains come primarily from scale. The reported parameter counts are comparatively modest relative to PIXEL, and the paper explicitly argues that the gains stem from architecture rather than brute-force parameter growth.

4. Benchmarks, quantitative results, and operating regime

The PDE benchmarks cover one-dimensional, two-dimensional, and irregular three-dimensional domains. The tested systems are Burgers’ equation with periodic boundary conditions for $3 \times 3$5 and $3 \times 3$6; one-dimensional convection or advection with periodic boundary conditions for $3 \times 3$7 and $3 \times 3$8; the two-dimensional Helmholtz equation with Dirichlet boundary conditions for $3 \times 3$9 and $\tanh$0; the two-dimensional lid-driven cavity problem for incompressible Navier–Stokes with moving top lid amplitude $\tanh$1; and Poisson’s equation on a 3D torus defined by

$\tanh$2

with prescribed boundary values (Shishehbor et al., 2024).

The training protocol is also fixed across experiments: Adam with initial learning rate $\tanh$3, reduced by $\tanh$4 every $\tanh$5k epochs, for a total of $\tanh$6k epochs. The sampling scheme typically uses $\tanh$7k interior points resampled every $\tanh$8 epochs, except for the lid-driven cavity case, which uses $\tanh$9k interior points. Dynamic weights are updated every $p$0 epochs, except for Helmholtz, where they are updated every epoch.

Benchmark	PGCAN median relative $p$1 error	Comparator errors
Burgers, $p$2	$p$3	M4 $p$4; PIXEL $p$5; vPINN $p$6
Burgers, $p$7	$p$8	M4 $p$9; PIXEL $q$0; vPINN $q$1
Convection, $q$2	$q$3	M4 $q$4; PIXEL $q$5; vPINN $q$6
Convection, $q$7	$q$8	M4 $q$9; PIXEL $$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$0; vPINN $$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$1
Helmholtz, $$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$2	$$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$3	M4 $$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$4; PIXEL $$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$5; vPINN $$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$6
Helmholtz, $$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$7	$$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$8	M4 $$F_c^{p,q,i_x,i_y} = \tanh\!\left(\sum_{b=0}^{2}\sum_{a=0}^{2} F_0^{p,q,i_x+a-1,i_y+b-1}\, W_q(a,b)\right),$$9; PIXEL $W_q \in \mathbb{R}^{3\times 3}$0; vPINN $W_q \in \mathbb{R}^{3\times 3}$1
Lid-driven cavity, $W_q \in \mathbb{R}^{3\times 3}$2	$W_q \in \mathbb{R}^{3\times 3}$3	M4 $W_q \in \mathbb{R}^{3\times 3}$4; PIXEL $W_q \in \mathbb{R}^{3\times 3}$5; vPINN $W_q \in \mathbb{R}^{3\times 3}$6
Lid-driven cavity, $W_q \in \mathbb{R}^{3\times 3}$7	$W_q \in \mathbb{R}^{3\times 3}$8	M4 $W_q \in \mathbb{R}^{3\times 3}$9; PIXEL $n_{\mathrm{rep}}$00; vPINN $n_{\mathrm{rep}}$01

These results are reported as median relative $n_{\mathrm{rep}}$02 errors after $n_{\mathrm{rep}}$03k epochs. The stated pattern is that PGCAN maintains accuracy as frequency and gradient complexity increase, with especially large margins in high-frequency regimes such as advection with $n_{\mathrm{rep}}$04, Helmholtz with $n_{\mathrm{rep}}$05, and large-gradient fluid flow in the lid-driven cavity. The complexity and memory discussion reinforces the same point: PGCAN uses a small grid and small $n_{\mathrm{rep}}$06, convolution is lightweight because it is feature-wise with $n_{\mathrm{rep}}$07 kernels, and inference remains fast due to local interpolation and a shallow decoder.

The ablation and sensitivity observations delimit the operating regime. Removing convolution in parametric grids is said to increase overfitting and slow boundary information propagation. Increasing model size alone in M4 or vPINN does not recover PGCAN’s performance in the lid-driven cavity problem. As $n_{\mathrm{rep}}$08 decreases in Burgers’ equation and frequency grows in advection and Helmholtz, PGCAN’s errors increase more modestly than those of baselines, although very small $n_{\mathrm{rep}}$09 can slightly degrade global accuracy because the grid is uniform. Comparing with refined finite-element solutions, the paper further reports that model errors converge as the reference discretization stabilizes, which it interprets as grid-resolution invariance of the continuous decoder even though the encoder itself is grid-parameterized.

5. Related instantiations in graph learning and neuroimaging

A graph-classification architecture that the source material explicitly uses to contextualize PGCAN is motif-based attention graph convolutional neural networks (MA-GCNN). In that setting, an arbitrary graph is first normalized into a 2D node grid by selecting central nodes with closeness centrality, extracting BFS neighborhoods, enumerating two-hop path motifs, and arranging the matched motifs into a canonical matrix. The resulting tensor

$n_{\mathrm{rep}}$10

is then processed by CNN-style layers and subgraph-level self-attention (Peng et al., 2018).

The first graph convolution uses a $n_{\mathrm{rep}}$11 kernel with horizontal stride $n_{\mathrm{rep}}$12 and vertical stride $n_{\mathrm{rep}}$13 so that each motif-length band is convolved as a unit. The second also uses a $n_{\mathrm{rep}}$14 kernel, with horizontal stride $n_{\mathrm{rep}}$15 and vertical stride $n_{\mathrm{rep}}$16. Subgraph features are flattened and passed to additive or dot-product attention. In the additive formulation,

$n_{\mathrm{rep}}$17

and multi-head averaging yields a weighted representation of each subgraph. The source material is explicit that the graph-to-grid normalization is non-parametric, whereas the convolutions, attention weights, and classifier are learned. It therefore presents MA-GCNN as embodying the core design of a PGCAN in the graph domain.

A second related instantiation is the 3D Grid-Attention network for age prediction and Alzheimer’s disease classification from structural MRI. Here the grid is volumetric rather than graph-normalized. A shallow 3D CNN backbone with four convolutional blocks and channel progression $n_{\mathrm{rep}}$18 is augmented by two spatial attention modules placed after convolutions $n_{\mathrm{rep}}$19 and $n_{\mathrm{rep}}$20, denoted A1 and A2 (Lam et al., 2020).

For a local feature map $n_{\mathrm{rep}}$21 and global context feature map $n_{\mathrm{rep}}$22 from the final convolution, the attention gate is

$n_{\mathrm{rep}}$23

$n_{\mathrm{rep}}$24

The projections $n_{\mathrm{rep}}$25, $n_{\mathrm{rep}}$26, and $n_{\mathrm{rep}}$27 are all $n_{\mathrm{rep}}$28 convolutions, with $n_{\mathrm{rep}}$29 for A1 and $n_{\mathrm{rep}}$30 for A2. Predictions from the backbone, A1 head, and A2 head are averaged. The salience maps are the normalized attention masks themselves, making the attention native rather than post hoc.

The empirical results in that neuroimaging setting are concrete. On cognitively normal test subjects, the brain-age model reports $n_{\mathrm{rep}}$31 years, $n_{\mathrm{rep}}$32, and correlation $n_{\mathrm{rep}}$33. The AD classification model reports accuracy $n_{\mathrm{rep}}$34, balanced accuracy $n_{\mathrm{rep}}$35, precision $n_{\mathrm{rep}}$36, and recall $n_{\mathrm{rep}}$37. Average Jaccard overlap between age and AD salience is higher for the lower-level attention module A1 than for A2; for example, at the threshold $n_{\mathrm{rep}}$38, A1 is $n_{\mathrm{rep}}$39 for CN/AD and A2 is $n_{\mathrm{rep}}$40. The qualitative interpretation given is that age prediction relies on more widespread features, whereas AD classification emphasizes ventricular regions.

Taken together, the graph and MRI cases show that the grid–convolution–attention template is not confined to PDE solving. The grid may arise from motif-guided normalization of an arbitrary graph, from volumetric CNN feature maps, or from a trainable coordinate-domain parameterization. This suggests a unifying viewpoint in which PGCAN denotes a family of architectures that preserve spatial structure, exploit local convolutional inductive bias, and use parametric attention to weight spatial or subgraph-level evidence.

6. Limitations, misconceptions, and future directions

The PDE formulation has several stated limitations. Uniform grids may underresolve highly localized shocks or discontinuities, so learning can focus on small regions and slightly degrade global accuracy. The paper suggests that adaptive meshes or hierarchical encoders with multi-resolution grids could better allocate parameters spatially (Shishehbor et al., 2024). Excessively large $n_{\mathrm{rep}}$41 or high-resolution grids can overfit, as illustrated by the contrast with PIXEL, and PGCAN explicitly avoids that regime through small $n_{\mathrm{rep}}$42 and convolution.

The theoretical status of the method is also carefully delimited. The paper offers mechanistic explanations for why local parametric encoding, convolution, and attention should reduce spectral bias, but it does not provide formal theorems. Any stronger claim would therefore exceed the evidence presented. A plausible implication is that PGCAN presently occupies an empirically grounded middle ground: its behavior is motivated by architectural reasoning and supported by benchmark results, but not yet characterized by a complete theory.

The domain-general extensions carry their own constraints. In the graph setting, the method assumes that two-hop path motifs and closeness centrality are appropriate organizing primitives; if those assumptions fail, the source material suggests that learned positional encodings or multi-motif vocabularies may be needed (Peng et al., 2018). In the MRI setting, all scans are $n_{\mathrm{rep}}$43T and no explicit inter-scanner harmonization is applied, so external validation across scanners and populations remains future work; the same source proposes domain adaptation and harmonization as next steps and notes that the models are single-task rather than joint age–diagnosis models (Lam et al., 2020).

Several misconceptions can therefore be ruled out. PGCAN is not synonymous with a transformer-only model, because convolution and locality are central to the design. It is not merely a parametric grid without attention, because attention is explicitly used to prioritize salient features or spatial positions. It is not restricted to regular Euclidean image domains, since the PDE work includes an irregular 3D torus and the graph work uses deterministic graph-to-grid normalization. Nor is it a purely supervised method, because the PDE solver is trained without labeled solution data in the domain.

The forward-looking directions stated across the sources are consistent. For PDEs, adaptive grids, hierarchical encoders, explicit physics-informed attention, hybrid spectral–grid encoders, and extensions to very high-dimensional PDEs and complex geometries are proposed. For neuroimaging, multi-task learning, mild cognitive impairment, and cross-scanner generalization via harmonization and domain adaptation are proposed. The recurring theme is that grid-convolution-attention architectures are most compelling when local structure matters, but the grid itself may need to become more adaptive as geometry, dimensionality, or heterogeneity increase.