RAM-LPM: Recurrent Attention with Log-Polar Mapping

• RAM-LPM is a neural framework that combines recurrent attention and log-polar mapping to mimic the human foveated vision system.
• It achieves scale and rotation equivariance by transforming image coordinates via log-polar sampling, ensuring efficient computation and enhanced robustness.
• Its variants utilize deterministic, stochastic, and reinforcement learning strategies to optimize fixation selection and improve scene representation.

A Recurrent Attention Model with Log-Polar Mapping (RAM-LPM) is a neural framework that integrates a recurrent attention mechanism with log-polar spatial sampling, closely inspired by the human visual system’s foveated structure. This model processes images through a series of glimpses—localized, high-resolution log-polar patches centered on dynamic visual fixations ("saccades")—which are selected and sequenced using attention networks. Through these recurrent glimpses, RAM-LPM achieves scale and rotation equivariance, efficient computation, and, in several variants, significant robustness to adversarial attacks. Architectures and training regimens vary—with deterministic, differentiable sampling and localization in some approaches, and stochastic reinforcement learning-driven gaze policies in others—but all emphasize biologically motivated vision, efficient representation, and dynamic focus of computation (Kurbiel et al., 2020, Kiritani et al., 2020, Göransson et al., 2023).

1. Log-Polar Mapping: Mathematical Principles and Biological Motivation

Log-polar mapping transforms Cartesian image coordinates around a center $(x_c, y_c)$ into $(\rho, \varphi)$, where

$$r = \sqrt{(x - x_c)^2 + (y - y_c)^2}\,,\quad \rho = \log r,\quad \varphi = \mathrm{atan2}(y - y_c,\ x - x_c)$$

and the inverse: $$x = x_c + e^{\rho}\cos\varphi,\quad y = y_c + e^{\rho}\sin\varphi$$ This mapping mimics the primate retina: resolution is maximal at the center ("fovea") and decreases with eccentricity while global periphery is preserved at coarse scale. A log-polar patch of fixed size thus condenses both high-acuity local detail and full-scene context into a compact tensor. The angular axis is treated as periodic (wrap padding), while the radial axis uses reflect padding to maintain equivariance (Kurbiel et al., 2020, Kiritani et al., 2020).

In reinforcement learning variants, similar foveated sampling is approximated by composing multiple rectangular patches of varying resolutions centered at a gaze point $f = (f_x, f_y)$, achieving the same qualitative resolution profile but via integer-indexed, non-radial sampling (Göransson et al., 2023).

2. Attention Mechanism and Glimpse Generation

At each time step $t$, RAM-LPM extracts a log-polar glimpse $G_t$ centered at $(x_t, y_t)$, which is processed by convolutional/feature extraction backbones. The mechanism for selecting the next fixation varies:

• In deterministic, fully differentiable models, a convolutional localization network predicts a spatial softmax over the feature map, producing new log-polar coordinates $(\rho', \varphi')$ via expectation over fixed grids, which are mapped back to Cartesian next centers $(x_{t+1}, y_{t+1})$ (Kurbiel et al., 2020).
• In hard-attention models, a "where" pathway parameterizes a Gaussian policy, sampling the next fixation stochastically: $(\rho, \varphi)$0, where the mean is produced by an LSTM driven by current glimpse features and location (Kiritani et al., 2020).
• In reinforcement learning settings, a visual action head (policy) selects among discrete gaze shifts (up, down, left, right, stay), updating the focus via a spatially localized region-of-interest sampler (Göransson et al., 2023).

Through recurrence (typically unrolling $(\rho, \varphi)$1 steps), the model builds an integrated representation of the scene, each glimpse contributing new contextual or discriminative evidence.

3. Architectural Components and Variations

While all RAM-LPM variants leverage log-polar or log-polar–like patches, architectural pipelines diverge based on supervision regime and biological fidelity.

Canonical RAM-LPM (RetinotopicNet, deterministic):

• Glimpse sensor: differentiable log-polar transform implemented as a spatial transformer.
• CNN backbone: three 3×3 convolution layers (32→64→128 channels), tanh activations, with wrap and reflect padding, max pooling.
• Localization net: two 1×1 convolutions (64→64→1), tanh, spatial softmax to produce score maps for next saccade.
• RNN aggregator: vanilla recurrent, hidden size $(\rho, \varphi)$2; aggregates glimpse features $(\rho, \varphi)$3.
• Classifier: two fully connected (128→96→classes) layers.
• No reinforcement learning; all components trained end-to-end by standard backpropagation.

Hard-attention RAM-LPM (biologically motivated):

• Separate "what" (classification) and "where" (attention) pathways, each with a dedicated CNN and LSTM.
• “Where” pathway outputs Gaussian parameters for the next fixation; sampling introduces stochasticity.
• Training employs REINFORCE with a learned baseline to optimize fixation selection and classification.
• Optionally includes "reflex" CNNs for low-latency responses from initial glimpses (Kiritani et al., 2020).

Reinforcement-Learning RAM-LPM (A3C-LSTM for vision and gaze):

• Log-polar–like sampler (piecewise rectangular), shared convolutional torso (A3C-style), LSTM core.
• Outputs two policy/value heads: one for game actions, one for gaze control.
• Gaze actions and natural actions are optimized jointly, with shifted return assignment for visual actions.
• Peripheral low-res context is beneficial, accelerating learning and reducing required pixels (Göransson et al., 2023).

Selector Table: Attention Variants

Attention Selection	Mechanism	Training	Reference
Differentiable	Spatial softmax, deterministic	Backprop	(Kurbiel et al., 2020)
Stochastic (policy)	LSTM+Gaussian, hard-attention	REINFORCE	(Kiritani et al., 2020)
Discrete RL	Discrete shift policy, A3C	A3C+GAE RL	(Göransson et al., 2023)

4. Training Objectives and Optimization

Depending on model type, objectives comprise:

• Classification loss: Cross-entropy on final class probabilities $(\rho, \varphi)$4—standard for all variants.
• Attention loss: For stochastic models, REINFORCE objective with baseline:

$(\rho, \varphi)$5

• Joint optimization: Summed classification and attention loss, with optional weighting.
• Reinforcement learning (A3C+GAE): Value and policy losses for both natural (game) actions and visual (gaze) actions, with adjusted advantage estimation to credit gaze shifts for future rewards.
• Pretraining options: In some architectures, the localization network is trained independently (greedy pretraining), then the full unrolled model is fine-tuned end-to-end (Kurbiel et al., 2020).
• Optimizers: Adam is used across models, with learning rates tailored by subnetwork (e.g., $(\rho, \varphi)$6 for "what" path, $(\rho, \varphi)$7 for "where") and moderate batch sizes.

Standard data augmentation techniques (hue, saturation, brightness, contrast, geometric transforms) are used routinely (Kurbiel et al., 2020).

5. Computational, Invariance, and Robustness Properties

RAM-LPM designs confer several advantageous properties:

• Fixed computational cost: Each glimpse is a fixed-size tensor; total FLOPs per image scale with $(\rho, \varphi)$8, and are independent of the original image size, unlike conventional CNNs whose cost scales with input resolution (Kiritani et al., 2020).
• Rotation and scale equivariance: Global image transformations correspond to additive shifts on the log-polar axes,

$(\rho, \varphi)$9

making convolutional features approximately equivariant to these transformations, especially with angular wrap padding (Kurbiel et al., 2020, Kiritani et al., 2020).

• Robustness to adversarial attacks: The recurrent, glimpse-based attention, especially with stochastic (hard) sampling, frustrates gradient-based adversarial attacks. Only regions visible in the current glimpse are affected, whole-image gradient flow is masked, and consistent perturbation across time is required to fool the model. On ImageNet, RAM-LPM retains significantly higher accuracy than ResNet-50 under strong SPSA and PGD attacks at $$r = \sqrt{(x - x_c)^2 + (y - y_c)^2}\,,\quad \rho = \log r,\quad \varphi = \mathrm{atan2}(y - y_c,\ x - x_c)$$0 (Kiritani et al., 2020).
• Rapid inference (“reflex” pathway): A shallow CNN+linear head enables fast classifications from the first glimpse, analogous to neural reflex arcs (Kiritani et al., 2020).
• Sample efficiency: Log-polar–like sampling reduces the number of active pixels by factors of 5 or more, with little to no loss in task performance—even on challenging control environments (Göransson et al., 2023).
• Aggregation across saccades: Recurrence (RNN/LSTM) allows aggregation of diverse glimpses, improving classification, especially for cluttered scenes (multipoint aggregation yields $$r = \sqrt{(x - x_c)^2 + (y - y_c)^2}\,,\quad \rho = \log r,\quad \varphi = \mathrm{atan2}(y - y_c,\ x - x_c)$$110% higher accuracy than single-glimpse evaluation) (Kurbiel et al., 2020).

6. Empirical Results and Performance

Empirical validation demonstrates the competitiveness of RAM-LPM across a range of tasks:

• On image classification:
  • MNIST: $$r = \sqrt{(x - x_c)^2 + (y - y_c)^2}\,,\quad \rho = \log r,\quad \varphi = \mathrm{atan2}(y - y_c,\ x - x_c)$$2 accuracy
  • Fashion-MNIST: $$r = \sqrt{(x - x_c)^2 + (y - y_c)^2}\,,\quad \rho = \log r,\quad \varphi = \mathrm{atan2}(y - y_c,\ x - x_c)$$3
  • CIFAR-10: $$r = \sqrt{(x - x_c)^2 + (y - y_c)^2}\,,\quad \rho = \log r,\quad \varphi = \mathrm{atan2}(y - y_c,\ x - x_c)$$4 with $$r = \sqrt{(x - x_c)^2 + (y - y_c)^2}\,,\quad \rho = \log r,\quad \varphi = \mathrm{atan2}(y - y_c,\ x - x_c)$$5 glimpses and 32×32 log-polar patches (Kurbiel et al., 2020)
• Adversarial robustness:
  • Significantly higher resistance to PGD and SPSA attacks compared to ResNet-50, with much lower attack success rates, especially on full-size images (Kiritani et al., 2020).
• Atari games:
  • With log-polar–like sampling, a 5× reduction in input pixels is achieved without significant degradation in game performance, although training is somewhat slower due to increased state–action space (Göransson et al., 2023).

Ablation studies support the importance of angular wrap-padding for rotation equivariance (removal reduces rotation accuracy by $$r = \sqrt{(x - x_c)^2 + (y - y_c)^2}\,,\quad \rho = \log r,\quad \varphi = \mathrm{atan2}(y - y_c,\ x - x_c)$$6) and of peripheral context for rapid learning and robustness. Doubling the patch size yields diminishing returns, and saccade count above $$r = \sqrt{(x - x_c)^2 + (y - y_c)^2}\,,\quad \rho = \log r,\quad \varphi = \mathrm{atan2}(y - y_c,\ x - x_c)$$7 saturates gains (Kurbiel et al., 2020, Göransson et al., 2023).

7. Limitations, Practical Considerations, and Extensions

RAM-LPM implementations exhibit several trade-offs:

• Partial observability: Glimpse-based vision makes the system partially observable at each timestep; requiring the RNN to integrate across glimpses for global scene understanding.
• Credit assignment: For attention policies, especially in RL settings, assignment of delayed rewards to gaze shifts is nontrivial and necessitates careful advantage computation (Göransson et al., 2023).
• Variant “log-polar” designs: Some reinforcement learning models use rectangular, rather than true radial, sampling for computational simplicity, trading theoretical equivariance for engineering efficiency.
• Training efficiency: Training speed is generally reduced compared to full-frame approaches (especially in RL) due to increased effective state and action spaces and the need to learn gaze policies jointly (Göransson et al., 2023).
• Extensibility: Extensions to continuous gaze control, object detection, and robotic vision are plausible, leveraging the efficiency of log-polar sampling and its alignment with embodied scenes.

Overall, RAM-LPM unifies biologically grounded visual strategies with deep learning, providing state-of-the-art equivariance, efficient computation, and demonstrable robustness advantages in both supervised and reinforcement learning contexts (Kurbiel et al., 2020, Kiritani et al., 2020, Göransson et al., 2023).