Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries

Abstract: With advanced image journaling tools, one can easily alter the semantic meaning of an image by exploiting certain manipulation techniques such as copy-clone, object splicing, and removal, which mislead the viewers. In contrast, the identification of these manipulations becomes a very challenging task as manipulated regions are not visually apparent. This paper proposes a high-confidence manipulation localization architecture which utilizes resampling features, Long-Short Term Memory (LSTM) cells, and encoder-decoder network to segment out manipulated regions from non-manipulated ones. Resampling features are used to capture artifacts like JPEG quality loss, upsampling, downsampling, rotation, and shearing. The proposed network exploits larger receptive fields (spatial maps) and frequency domain correlation to analyze the discriminative characteristics between manipulated and non-manipulated regions by incorporating encoder and LSTM network. Finally, decoder network learns the mapping from low-resolution feature maps to pixel-wise predictions for image tamper localization. With predicted mask provided by final layer (softmax) of the proposed architecture, end-to-end training is performed to learn the network parameters through back-propagation using ground-truth masks. Furthermore, a large image splicing dataset is introduced to guide the training process. The proposed method is capable of localizing image manipulations at pixel level with high precision, which is demonstrated through rigorous experimentation on three diverse datasets.

• The paper presents a novel framework combining LSTM and encoder-decoder networks to achieve pixel-level detection of manipulated image regions.
• It leverages resampling features and Hilbert curve-based patch sequencing to capture subtle spatial and temporal manipulation artifacts.
• Quantitative results reveal up to 20.52% improvement over baseline methods on NIST'16, enhancing digital media forensic reliability.

Overview of Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries

This paper introduces a novel framework for detecting manipulated regions in digital images, specifically targeting content-changing forgeries such as splicing, object removal, and copy-move operations. The proposed method focuses on localizing manipulations at the pixel level, leveraging a combination of Long-Short Term Memory (LSTM) networks and encoder-decoder architectures to achieve fine-grained segmentation.

Leveraging resampling features, the framework effectively captures artifacts introduced by image editing techniques like JPEG quality loss, rotation, and scaling. The innovative use of LSTM allows the model to understand spatial correlations between manipulated and non-manipulated patches over the sequence defined by a Hilbert curve. This ordering considers spatial locality, enhancing the model's ability to detect subtle, non-obvious changes in the image that might be overlooked by conventional convolutional neural networks (CNNs).

The proposed system employs several key components: resampling feature extraction, a hybrid LSTM network, a convolutional encoder to capture spatial features, and a decoder network. The encoder processes the input image to produce spatial feature maps, while the LSTM network enhances manipulation detection by analyzing the sequence of resampling features from divided image patches. The decoder is responsible for transforming low-resolution feature maps into a pixel-wise prediction map, indicating image tamperings.

Strong Numerical Results and Extensive Dataset

The authors present a significant contribution to image forensics with the introduction of a large synthesized dataset, significantly surpassing the scale of existing datasets like CoMoFoD and COVERAGE both in volume and image resolution. The training of an end-to-end model on this dataset, referred to as the 'Base-Model,' serves as a solid foundation, facilitating fine-tuning on widely recognized datasets such as NIST'16 and IEEE Forensics Challenge. This strategic approach enhances the model's generalization and ensures robust evaluation across diverse scenarios.

Quantitative results showcase the strength of the approach, with pixel-wise accuracy improvements observed over baseline methods like Fully Convolutional Networks (FCN) and SegNet. Specifically, the finetuned model (LSTM-EnDec) outperforms the FCN and Encoder-Decoder networks significantly by 20.52% and 11.84% on the NIST'16 dataset, underscoring the efficacy of combining CNN spatial features with the temporal dependencies modeled by LSTM.

Implications and Future Directions in AI

This work has practical implications for enhancing the reliability of digital media by providing a robust solution for detecting forged content. This methodology not only improves the accuracy but also addresses critical challenges in pinpointing precise manipulation boundaries, crucial for applications in digital forensics and media integrity verification.

Theoretically, the inclusion of resampling features fills a gap overlooked by prior CNN-based methods, which often struggle with manipulations that lack distinct visual cues. This hybrid approach provides an exemplary model for future explorations in multimedia forensics, suggesting that subsequent research could explore further integrating frequency domain insights with advanced architectures like transformers for even more nuanced detection capabilities.

Avenues for future work could include refining the model to handle newer types of digital manipulations facilitated by generative adversarial networks (GANs) and exploring domain adaptation techniques to improve performance across diverse datasets without extensive retraining. Additionally, the proposed framework could evolve to support video forensics, extending its capability beyond static images, thus addressing a broader spectrum of multimedia manipulation challenges.