GAMA-Net: Grid-Aware Multiscale Adaptive Network

• GAMA-Net is a deep learning architecture that uses dual encoders to combine global and patch-level analysis for detailed depth estimation from monocular images.
• It integrates Patch-Based Hybrid Attention and Adaptive Multiscale Feature Fusion in a U-Net–style decoder to seamlessly merge local details with global context.
• Empirical evaluations demonstrate improved depth accuracy (e.g., 78.15% Acc1) and enhanced spine curve segmentation (IoU ≈ 0.81) compared to conventional approaches.

The Grid-Aware Multiscale Adaptive Network (GAMA-Net) is an adaptive deep learning architecture specifically tailored for structured grid-based, multiscale representation learning. Designed for precise surface depth and morphology estimation from monocular RGB imagery—demonstrated in the context of automated scoliosis assessment using unclothed back images—GAMA-Net fuses parallel global and patch-level feature extraction with dynamic attention and multi-path decoding for robust and spatially nuanced predictions (Wei et al., 30 Jul 2025).

1. Architectural Overview

GAMA-Net is composed of two primary stages. The first stage executes monocular depth estimation: extracting a per-pixel back surface geometry from a single RGB image. The second stage utilizes the predicted depth concatenated with the RGB input (resulting in a four-channel RGBD tensor) to generate a binary spine-curve mask via semantic segmentation.

The first stage is itself subdivided into three main components:

• Dual encoders: a global encoder processes the entire image; a patch encoder processes a 3×3 grid of tiles independently with shared weights.
• Patch-Based Hybrid Attention (PBHA) modules: deployed after the first three encoder layers, these fuse global and patch-level representations via cross-attention.
• U-Net–style decoder: featuring skip connections from each encoder layer and employing Adaptive Multiscale Feature Fusion (AMFF) blocks at each upsampling stage.

The second stage incorporates a lightweight U-Net segmentation network that takes the RGBD tensor as input, producing a binary mask that delineates the spine curve (Wei et al., 30 Jul 2025).

2. Dual Encoders and Grid-Aware Locality

The dual-encoder system is designed to decouple local and global image contexts:

• Global encoder: Ingests the full, high-resolution RGB image ($I \in \mathbb{R}^{H \times W \times 3}$, typically 480×240 px), resulting in layerwise feature maps of increasing channel dimension and receptive field.
• Patch encoder: Operates on each $P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$ for $i=0,...,8$, with weights shared across all 9 patches, ensuring consistent local feature extraction.

Both encoders have five layers, each composed of Conv2D → BatchNorm → ReLU:

Encoder Layer	Output Channels	Receptive Field (approx.)
1 (shallow)	64	3×3
2	128	7×7
3	256	15×15
4	512	31×31
5 (bottleneck)	1024	63×63

The global encoder’s deepest layer encompasses over a 60×60 spatial region, while each patch encoder sees its respective grid tile in full at each layer. This explicit “grid” structuring regularizes the extraction of minute, spatially localized cues (e.g., scapular surface relief) without global averaging artifacts (Wei et al., 30 Jul 2025).

3. Patch-Based Hybrid Attention (PBHA) Mechanism

PBHA modules embed cross-attentional fusion at encoder layers 1–3 to integrate local and global feature hierarchies. Operating on features at layer $l$:

1. Patch encoder produces $\{ F^p_i \}_{i=0}^{8}$, with each $F^p_i \in \mathbb{R}^{C \times h \times w}$.
2. Global encoder produces $F^g \in \mathbb{R}^{C \times H' \times W'}$, partitioned into 9 sub-blocks $F^g_i$ spatially corresponding to the tiles.

For each patch $i$:

$$Q_i = W_Q F^g_i, \quad K_i = W_K F^p_i, \quad V_i = W_V F^p_i$$

$P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$0

$P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$1

$P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$2

where $P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$3 are learnable projections, $P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$4. This cross-attention allows each patch feature to be contextually re-weighted by its global context, providing spatially adaptive blending of local detail and overall semantic cues (Wei et al., 30 Jul 2025).

4. Adaptive Multiscale Feature Fusion in Decoding

The decoding pathway employs Adaptive Multiscale Feature Fusion (AMFF) at each upsampling stage to merge multi-resolution representations. AMFF performs:

1. Upsampling and 1×1 convolution of higher-level decoder features $P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$5.
2. Channel-wise concatenation with corresponding skip-connected encoder feature $P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$6.
3. Computation of a sigmoid spatial gating mask $P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$7 via learned 1×1 convolution.
4. Channel-wise and pixel-wise fusion:

$P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$8

where $P_i \in \mathbb{R}^{(H/3) \times (W/3) \times 3}$9 are channel splits, and $i=0,...,8$0 denotes elementwise multiplication.

This dynamic mechanism enables per-pixel, per-channel weighting between deep semantic and shallow spatially precise cues, preventing redundancy while supporting both gross and fine prediction granularity (Wei et al., 30 Jul 2025).

5. Grid-Aware Multiscale Adaptation Principles

The core innovation is the explicit encoding of a spatial grid with multiscale adaptivity:

• Each input is decomposed into a 3×3 tiling, with both global and patch pathways learning in parallel.
• Parallel global/patch encoding ensures subtle depth cues (e.g., those near thoracic curvature or scapular edges) are maintained alongside holistic shape context.
• PBHA modules provide content-adaptive blending of local and global features, while AMFF modules in the decoder ensure seamless lateral flow of information from deep, semantically rich layers to shallow, high-resolution outputs.
• This structure enables the network to maintain spatial sensitivity and geometric fidelity, outperforming standard monocular depth estimation approaches particularly in cases where high-frequency anatomy must be delineated (Wei et al., 30 Jul 2025).

A related theoretical framework is the multigrid neural architectures paradigm, which generalizes convolutional networks to operate on a pyramid of spatial grids and supports explicit cross-scale message passing and gating (Ke et al., 2016). GAMA-Net instantiates these concepts by discretizing grids (3×3), using adaptive gating, and residualizing both within and across scales.

6. Optimization, Losses, and Training Regimen

Losses:

• Depth estimation ($i=0,...,8$1):

$i=0,...,8$2

• Berhu loss: smooth L1 variant tolerating small errors, weighted ($i=0,...,8$3), $i=0,...,8$4
• SSIM: $i=0,...,8$5 between predicted and true depth, weights structural similarity ($i=0,...,8$6)
• Gradient: mean absolute gradient difference ($i=0,...,8$7)
  • Spine curve segmentation ($i=0,...,8$8):

$i=0,...,8$9

Binary pixelwise cross-entropy, as standard for semantic segmentation.

Training Setup:

• Dataset: 2,213 triplets (RGB, depth, full-spine x-ray) collected via Azure Kinect DK, ages 10–18, gender split 1619F/594M, partitioned 8:1:1 train/val/test.
• Images aligned to 480×240.
• Data augmentation restricted to Stage 2 (segmentation): flips, Gaussian noise; not used in depth estimation due to patching constraints.
• PyTorch framework on RTX 3090; Adam optimizer (lr=$l$0, $l$1, $l$2); batch sizes 16 (depth) and 8 (curve).
• ~100 epochs with early stopping on validation loss for both stages (Wei et al., 30 Jul 2025).

7. Performance Metrics and Empirical Evaluation

Evaluation employs canonical depth estimation benchmarks:

• Accuracy thresholds:

$l$3

• $l$4, $l$5, $l$6
  • Error metrics:

$l$7

• Relative AbsRel is reduced by ~4% and Acc1 is improved by ~1.5% compared to leading monocular baselines (FDSI, FDE).
• Spine curve segmentation:
  • RGB only: IoU ≈ 0.8025, Dice ≈ 0.8887
  • Depth only: IoU ≈ 0.7914, Dice ≈ 0.8820
  • RGBD (GAMA-Net): IoU ≈ 0.8102, Dice ≈ 0.8939

Qualitatively, GAMA-Net depth predictions better preserve subtle contours around anatomical landmarks (e.g., scapula, paraspinal regions), yielding more continuous and anatomically faithful spine curve segmentations than either RGB or depth-only input modalities (Wei et al., 30 Jul 2025).

---

References:

• "A Dual-Feature Extractor Framework for Accurate Back Depth and Spine Morphology Estimation from Monocular RGB Images" (Wei et al., 30 Jul 2025)
• "Multigrid Neural Architectures" (Ke et al., 2016)