---
title: 'Pixel R-CNN: Pixel-based Temporal Classification'
url: https://www.emergentmind.com/topics/pixel-r-cnn
type: topic
---

# Pixel R-CNN: Pixel-based Temporal Classification

Pixel R-CNN is a pixel-based deep learning model for land cover and crop classification (LC\&CC) from multi-temporal Sentinel-2 data. Introduced for a study area in Carpi, Emilia-Romagna (Italy), it combines Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN) to learn temporal correlations and temporal patterns directly from a single pixel’s multi-temporal, multi-spectral signature, reducing manual feature engineering and modeling crop phenological stages [2004.12880]. In the reported formulation, the model ingests $X^{(i)} \in \mathbb{R}^{T \times B}$ and outputs a $K$-way probability vector for 15 land cover and crop classes, achieving an Overall Accuracy of $96.5\%$ and Kappa of $0.914$ on the test set [2004.12880].

## 1. Concept and nomenclature

In the cited work, Pixel R-CNN denotes a recurrent-convolutional architecture for **pixel-based LC\&CC**, not an object-based detector. The paper explicitly contrasts existing solutions as “per-pixel based and object-based” and presents Pixel R-CNN as a “novel and optimal deep learning model for pixel-based LC\&CC” built from RNN and CNN components [2004.12880]. A plausible implication is that the name can be misread by readers familiar with region-based CNN families; in this context, however, the “R-CNN” label refers to a recurrent-convolutional combination operating on per-pixel temporal signatures rather than on spatial proposals.

The model is designed for multi-temporal classification tasks using Sentinel-2 imagery, particularly where diverse agricultural systems and multiple crop classes must be discriminated at scale. The study considered fifteen classes, including major agricultural crops, and emphasized automated extraction of spectral-temporal patterns from time series rather than hand-crafted phenological metrics [2004.12880].

## 2. Architectural organization

The architecture is organized into three main stages: **time-correlation**, **temporal-pattern extraction**, and **multiclass classification** [2004.12880]. For a single sample, the input is a pixel’s temporal-spectral signature $X^{(i)} \in \mathbb{R}^{T \times B}$. The time-correlation stage uses a peephole LSTM with 32 units, followed by Dropout with $p=0.2$ and a TimeDistributed Dense layer. The output is then reshaped to a 3D tensor with $(H=T, W=D, C=1)$ and passed to two Conv2D layers: the first with 16 filters and a $3 \times 3$ kernel, the second with 32 filters and a $7 \times 7$ kernel; both use ReLU activations [2004.12880]. The classification head consists of Flatten, Dense with $K=15$, and Softmax, yielding $\hat{y}^{(i)} \in [0,1]^K$ [2004.12880].

The stated purpose of this organization is division of labor across modules. The peephole LSTM learns time correlation across the sequence of images, while the convolutional stack refines temporal patterns hierarchically. The paper attributes the reported performance gains to this temporal feature learning, specifically noting that the RNN captures crop phenology over $T$ images, the convolution refines temporal patterns hierarchically, and end-to-end learning removes the need for hand-crafted phenological metrics [2004.12880].

## 3. Mathematical formulation

For each time step $t=1 \ldots T$ and sample $(i)$, the peephole LSTM is defined as follows [2004.12880]:

$$
i_t = \sigma(W_{ci} c_{t-1} + W_{hi} h_{t-1} + W_{xi} x_t + b_i)
$$

$$
f_t = \sigma(W_{cf} c_{t-1} + W_{hf} h_{t-1} + W_{xf} x_t + b_f)
$$

$$
g_t = \tanh(W_{hg} h_{t-1} + W_{xg} x_t + b_g)
$$

$$
c_t = f_t \odot c_{t-1} + i_t \odot g_t
$$

$$
o_t = \sigma(W_{co} c_t + W_{ho} h_{t-1} + W_{xo} x_t + b_o)
$$

$$
h_t = y_t = o_t \odot \tanh(c_t)
$$

After processing all $T$ steps, the model produces an output sequence $Y^{(i)}_{\text{lstm}} \in \mathbb{R}^{T \times H}$ [2004.12880]. The TimeDistributed Dense layer is then applied identically at each time step:

$$
Y^{(i)}_{\text{timeD}} = W_{\text{td}} \cdot Y^{(i)}_{\text{lstm}} + b_{\text{td}}
$$

This representation is reshaped to a 3D tensor $Z^{(i)} \in \mathbb{R}^{H \times D \times 1}$ and processed by two 2D convolutions [2004.12880]:

$$
Y_{\text{conv1}} = \text{ReLU}(W_1 * Z^{(i)} + b_1)
$$

$$
Y_{\text{conv2}} = \text{ReLU}(W_2 * Y_{\text{conv1}} + b_2)
$$

where $*$ denotes 2D convolution, $W_1$ has $(16,3,3)$ kernels, and $W_2$ has $(32,7,7)$ kernels [2004.12880]. The multiclass output is defined by flattening the final convolutional activation:

$$
a = \text{Flatten}(Y_{\text{conv2}}) \in \mathbb{R}^{M}
$$

$$
s_k = w_k^T a + b_k,\quad k=1 \ldots K
$$

$$
\hat{y}_k = \frac{\exp(s_k)}{\sum_{j=1}^{K} \exp(s_j)}
$$

Training minimizes categorical cross-entropy over the dataset $\{(X^{(i)}, y^{(i)})\}$ [2004.12880]:

$$
J(\Theta) = -\frac{1}{n} \sum_{i=1}^{n} \sum_{k=1}^{K} y_k^{(i)} \log \hat{y}_k^{(i)}
$$

Optimization uses AMSGrad updates without debiasing [2004.12880]:

$$
m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t
$$

$$
v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2
$$

$$
\hat{v}_t = \max(\hat{v}_{t-1}, v_t)
$$

$$
\Theta \leftarrow \Theta - \eta \cdot \frac{m_t}{\sqrt{\hat{v}_t}+\epsilon}
$$

## 4. Input representation, data, and preprocessing

The input data consisted of **ten Sentinel-2 L2A images (June 2015–July 2016)**, each with bands **B2/B3/B4/B8 at 10 m + NDVI** [2004.12880]. Each sample was represented as $X^{(i)} = [x_1, \ldots, x_T]$, with $x_t \in \mathbb{R}^5$ corresponding to blue, green, red, NIR, and NDVI [2004.12880]. The model therefore operates on a compact per-pixel temporal signature rather than on explicit spatial neighborhoods.

The dataset included **92 116 ground-truth pixels over 15 classes**. Preprocessing used a **stratified 60/40 train/test split**, producing $X_{\text{train}}$ with **55 270** samples and $X_{\text{test}}$ with **36 846** samples [2004.12880]. Standardization was performed per band using $x \leftarrow (x - \mu)/\sigma$, and **no further augmentation was applied** [2004.12880].

The class distribution reported in the study was as follows [2004.12880]:

| Class | Pixels | % |
|---|---:|---:|
| Tomatoes | 3 020 | 3.20 |
| Artificial | 9 343 | 10.14 |
| Trees | 7 384 | 8.01 |
| Rye | 4 382 | 4.75 |
| Wheat | 12 826 | 13.92 |
| Soya | 5 836 | 6.33 |
| Apple | 849 | 0.92 |
| Peer | 495 | 0.53 |
| Grassland | 1 744 | 1.89 |
| Water | 2 451 | 2.66 |
| Lucerne | 17 942 | 19.47 |
| Durum Wheat | 1 188 | 1.28 |
| Vineyard | 6 110 | 6.63 |
| Barley | 2 549 | 2.76 |
| Maize | 15 997 | 17.37 |

The study area was **Carpi, Emilia-Romagna (Italy), ~2 640 km², mixed croplands** [2004.12880]. This geographical and agronomic setting is significant because the paper frames the problem as classification in a diverse agricultural system dominated by economic crop types, with multiple crop classes present simultaneously.

## 5. Training protocol and empirical performance

Training used categorical cross-entropy, Dropout with $p=0.2$ between the peephole LSTM and TimeDistributed layer, and **AMSGrad with $\beta_1=0.86$, $\beta_2=0.98$, $\epsilon=1e{-9}$** [2004.12880]. The learning-rate schedule was **cosine annealing with cycle length = 150 epochs**, and the initial $\eta$ was found by **LR-range test** [2004.12880]. The reported training configuration used **batch size = 128** and **epochs = 150** [2004.12880].

Evaluation employed **Overall Accuracy (OA), Kappa (K), Producer’s Accuracy (PA), and User’s Accuracy (UA)** [2004.12880]. On the test set, Pixel R-CNN achieved **OA = 96.5 %** and **Kappa = 0.914** [2004.12880]. The confusion-matrix summary reported selected classwise accuracies: **Water** with **PA 100%** and **UA 99.1%**, **Trees** with **PA 98%** and **UA 99.3%**, **Maize** with **PA 99%** and **UA 98.7%**, **Lucerne** with **PA 98%** and **UA 96.9%**, and **Grassland** with **PA 65%** and **UA 63.0%** [2004.12880].

These results indicate uneven class difficulty within the same model. The paper directly identifies classes with few samples or high intra-class variability, such as grassland, as remaining challenging [2004.12880]. This suggests that the high global OA coexists with class-dependent failure modes that are relevant for operational LC\&CC workflows.

## 6. Comparative position, advantages, and limitations

The paper compared Pixel R-CNN with four “classic” classifiers tuned by randomized hyper-parameter search on the same feature set $\{X^{(i)}\}$ [2004.12880]:

| Model | Best settings | OA |
|---|---|---:|
| SVM (linear) | $C=1$ | 79.5% |
| Kernel SVM | RBF, $C=10$, $\gamma=0.3$ | 76.2% |
| Random Forest | $n\_estimators=200$, $max\_depth=15$ | 77.9% |
| XGBoost | $\eta=0.05$, $max\_depth=9$, $\gamma=0.1$ | 77.6% |
| Pixel R-CNN | — | 96.5% |

The temporal feature learning of Pixel R-CNN was reported to yield **more than 16 percentage points gain in OA** relative to these baselines [2004.12880]. The paper attributes this gain to three factors: RNN capture of crop phenology over the image sequence, hierarchical refinement of temporal patterns by convolution, and end-to-end learning without hand-crafted phenological metrics [2004.12880].

The stated advantages are: **automated extraction of spectral-temporal patterns, avoiding manual VI-derivation and thresholding**; **distributed representation**, in which LSTM and convolutional layers cooperate to disentangle intra-class variability; and a **lightweight** model with **less than 31 k parameters**, trainable in **approximately hours on a single GPU** [2004.12880]. The stated limitations are equally explicit: the model **operates per-pixel**, so **spatial context (neighborhood) is ignored**; it **requires clear-sky, co-registered time series**, so gaps due to clouds may degrade performance; and classes with few samples or high intra-class variability remain difficult [2004.12880].

Future enhancements proposed in the paper include **attention**, **data-fusion with SAR (Sentinel-1)**, **multiscale spatial features**, and **semi-supervised learning to handle label scarcity** [2004.12880]. A plausible implication is that Pixel R-CNN occupies a specific point in the design space of remote-sensing classifiers: it prioritizes compact temporal modeling at the pixel level, with the expectation that subsequent extensions would inject spatial context, multimodal sensing, or improved supervision.

Source: https://www.emergentmind.com/topics/pixel-r-cnn