Pixel R-CNN: Pixel-based Temporal Classification

• The paper introduces Pixel R-CNN, a novel integration of RNN and CNN for automated land cover and crop classification that achieves 96.5% overall accuracy.
• The model employs a peephole LSTM to capture temporal correlations in crop phenology while convolutional layers hierarchically refine spectral-temporal patterns.
• Its lightweight, end-to-end design removes the need for manual feature engineering but is limited by its per-pixel approach and lack of spatial context.

Pixel R-CNN is a pixel-based deep learning model for land cover and crop classification (LC&CC) from multi-temporal Sentinel-2 data. Introduced for a study area in Carpi, Emilia-Romagna (Italy), it combines Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN) to learn temporal correlations and temporal patterns directly from a single pixel’s multi-temporal, multi-spectral signature, reducing manual feature engineering and modeling crop phenological stages (Mazzia et al., 2020). In the reported formulation, the model ingests $X^{(i)} \in \mathbb{R}^{T \times B}$ and outputs a $K$-way probability vector for 15 land cover and crop classes, achieving an Overall Accuracy of $96.5\%$ and Kappa of $0.914$ on the test set (Mazzia et al., 2020).

1. Concept and nomenclature

In the cited work, Pixel R-CNN denotes a recurrent-convolutional architecture for pixel-based LC&CC, not an object-based detector. The paper explicitly contrasts existing solutions as “per-pixel based and object-based” and presents Pixel R-CNN as a “novel and optimal deep learning model for pixel-based LC&CC” built from RNN and CNN components (Mazzia et al., 2020). A plausible implication is that the name can be misread by readers familiar with region-based CNN families; in this context, however, the “R-CNN” label refers to a recurrent-convolutional combination operating on per-pixel temporal signatures rather than on spatial proposals.

The model is designed for multi-temporal classification tasks using Sentinel-2 imagery, particularly where diverse agricultural systems and multiple crop classes must be discriminated at scale. The study considered fifteen classes, including major agricultural crops, and emphasized automated extraction of spectral-temporal patterns from time series rather than hand-crafted phenological metrics (Mazzia et al., 2020).

2. Architectural organization

The architecture is organized into three main stages: time-correlation, temporal-pattern extraction, and multiclass classification (Mazzia et al., 2020). For a single sample, the input is a pixel’s temporal-spectral signature $X^{(i)} \in \mathbb{R}^{T \times B}$. The time-correlation stage uses a peephole LSTM with 32 units, followed by Dropout with $p=0.2$ and a TimeDistributed Dense layer. The output is then reshaped to a 3D tensor with $(H=T, W=D, C=1)$ and passed to two Conv2D layers: the first with 16 filters and a $3 \times 3$ kernel, the second with 32 filters and a $7 \times 7$ kernel; both use ReLU activations (Mazzia et al., 2020). The classification head consists of Flatten, Dense with $K=15$, and Softmax, yielding $K$0 (Mazzia et al., 2020).

The stated purpose of this organization is division of labor across modules. The peephole LSTM learns time correlation across the sequence of images, while the convolutional stack refines temporal patterns hierarchically. The paper attributes the reported performance gains to this temporal feature learning, specifically noting that the RNN captures crop phenology over $K$1 images, the convolution refines temporal patterns hierarchically, and end-to-end learning removes the need for hand-crafted phenological metrics (Mazzia et al., 2020).

3. Mathematical formulation

For each time step $K$2 and sample $K$3, the peephole LSTM is defined as follows (Mazzia et al., 2020):

$K$4

$K$5

$K$6

$K$7

$K$8

$K$9

After processing all $96.5\%$0 steps, the model produces an output sequence $96.5\%$1 (Mazzia et al., 2020). The TimeDistributed Dense layer is then applied identically at each time step:

$96.5\%$2

This representation is reshaped to a 3D tensor $96.5\%$3 and processed by two 2D convolutions (Mazzia et al., 2020):

$96.5\%$4

$96.5\%$5

where $96.5\%$6 denotes 2D convolution, $96.5\%$7 has $96.5\%$8 kernels, and $96.5\%$9 has $0.914$0 kernels (Mazzia et al., 2020). The multiclass output is defined by flattening the final convolutional activation:

$0.914$1

$0.914$2

$0.914$3

Training minimizes categorical cross-entropy over the dataset $0.914$4 (Mazzia et al., 2020):

$0.914$5

Optimization uses AMSGrad updates without debiasing (Mazzia et al., 2020):

$0.914$6

$0.914$7

$0.914$8

$0.914$9

4. Input representation, data, and preprocessing

The input data consisted of ten Sentinel-2 L2A images (June 2015–July 2016), each with bands B2/B3/B4/B8 at 10 m + NDVI (Mazzia et al., 2020). Each sample was represented as $X^{(i)} \in \mathbb{R}^{T \times B}$0, with $X^{(i)} \in \mathbb{R}^{T \times B}$1 corresponding to blue, green, red, NIR, and NDVI (Mazzia et al., 2020). The model therefore operates on a compact per-pixel temporal signature rather than on explicit spatial neighborhoods.

The dataset included 92 116 ground-truth pixels over 15 classes. Preprocessing used a stratified 60/40 train/test split, producing $X^{(i)} \in \mathbb{R}^{T \times B}$2 with 55 270 samples and $X^{(i)} \in \mathbb{R}^{T \times B}$3 with 36 846 samples (Mazzia et al., 2020). Standardization was performed per band using $X^{(i)} \in \mathbb{R}^{T \times B}$4, and no further augmentation was applied (Mazzia et al., 2020).

The class distribution reported in the study was as follows (Mazzia et al., 2020):

Class	Pixels	%
Tomatoes	3 020	3.20
Artificial	9 343	10.14
Trees	7 384	8.01
Rye	4 382	4.75
Wheat	12 826	13.92
Soya	5 836	6.33
Apple	849	0.92
Peer	495	0.53
Grassland	1 744	1.89
Water	2 451	2.66
Lucerne	17 942	19.47
Durum Wheat	1 188	1.28
Vineyard	6 110	6.63
Barley	2 549	2.76
Maize	15 997	17.37

The study area was Carpi, Emilia-Romagna (Italy), ~2 640 km², mixed croplands (Mazzia et al., 2020). This geographical and agronomic setting is significant because the paper frames the problem as classification in a diverse agricultural system dominated by economic crop types, with multiple crop classes present simultaneously.

5. Training protocol and empirical performance

Training used categorical cross-entropy, Dropout with $X^{(i)} \in \mathbb{R}^{T \times B}$5 between the peephole LSTM and TimeDistributed layer, and AMSGrad with $X^{(i)} \in \mathbb{R}^{T \times B}$6, $X^{(i)} \in \mathbb{R}^{T \times B}$7, $X^{(i)} \in \mathbb{R}^{T \times B}$8 (Mazzia et al., 2020). The learning-rate schedule was cosine annealing with cycle length = 150 epochs, and the initial $X^{(i)} \in \mathbb{R}^{T \times B}$9 was found by LR-range test (Mazzia et al., 2020). The reported training configuration used batch size = 128 and epochs = 150 (Mazzia et al., 2020).

Evaluation employed Overall Accuracy (OA), Kappa (K), Producer’s Accuracy (PA), and User’s Accuracy (UA) (Mazzia et al., 2020). On the test set, Pixel R-CNN achieved OA = 96.5 % and Kappa = 0.914 (Mazzia et al., 2020). The confusion-matrix summary reported selected classwise accuracies: Water with PA 100% and UA 99.1%, Trees with PA 98% and UA 99.3%, Maize with PA 99% and UA 98.7%, Lucerne with PA 98% and UA 96.9%, and Grassland with PA 65% and UA 63.0% (Mazzia et al., 2020).

These results indicate uneven class difficulty within the same model. The paper directly identifies classes with few samples or high intra-class variability, such as grassland, as remaining challenging (Mazzia et al., 2020). This suggests that the high global OA coexists with class-dependent failure modes that are relevant for operational LC&CC workflows.

6. Comparative position, advantages, and limitations

The paper compared Pixel R-CNN with four “classic” classifiers tuned by randomized hyper-parameter search on the same feature set $p=0.2$0 (Mazzia et al., 2020):

Model	Best settings	OA
SVM (linear)	$p=0.2$1	79.5%
Kernel SVM	RBF, $p=0.2$2, $p=0.2$3	76.2%
Random Forest	$p=0.2$4, $p=0.2$5	77.9%
XGBoost	$p=0.2$6, $p=0.2$7, $p=0.2$8	77.6%
Pixel R-CNN	—	96.5%

The temporal feature learning of Pixel R-CNN was reported to yield more than 16 percentage points gain in OA relative to these baselines (Mazzia et al., 2020). The paper attributes this gain to three factors: RNN capture of crop phenology over the image sequence, hierarchical refinement of temporal patterns by convolution, and end-to-end learning without hand-crafted phenological metrics (Mazzia et al., 2020).

The stated advantages are: automated extraction of spectral-temporal patterns, avoiding manual VI-derivation and thresholding; distributed representation, in which LSTM and convolutional layers cooperate to disentangle intra-class variability; and a lightweight model with less than 31 k parameters, trainable in approximately hours on a single GPU (Mazzia et al., 2020). The stated limitations are equally explicit: the model operates per-pixel, so spatial context (neighborhood) is ignored; it requires clear-sky, co-registered time series, so gaps due to clouds may degrade performance; and classes with few samples or high intra-class variability remain difficult (Mazzia et al., 2020).

Future enhancements proposed in the paper include attention, data-fusion with SAR (Sentinel-1), multiscale spatial features, and semi-supervised learning to handle label scarcity (Mazzia et al., 2020). A plausible implication is that Pixel R-CNN occupies a specific point in the design space of remote-sensing classifiers: it prioritizes compact temporal modeling at the pixel level, with the expectation that subsequent extensions would inject spatial context, multimodal sensing, or improved supervision.