Deep filter banks for texture recognition, description, and segmentation (1507.02620v2)

Abstract: Visual textures have played a key role in image understanding because they convey important semantics of images, and because texture representations that pool local image descriptors in an orderless manner have had a tremendous impact in diverse applications. In this paper we make several contributions to texture understanding. First, instead of focusing on texture instance and material category recognition, we propose a human-interpretable vocabulary of texture attributes to describe common texture patterns, complemented by a new describable texture dataset for benchmarking. Second, we look at the problem of recognizing materials and texture attributes in realistic imaging conditions, including when textures appear in clutter, developing corresponding benchmarks on top of the recently proposed OpenSurfaces dataset. Third, we revisit classic texture representations, including bag-of-visual-words and the Fisher vectors, in the context of deep learning and show that these have excellent efficiency and generalization properties if the convolutional layers of a deep model are used as filter banks. We obtain in this manner state-of-the-art performance in numerous datasets well beyond textures, an efficient method to apply deep features to image regions, as well as benefit in transferring features from one domain to another.

• The paper establishes a benchmark for texture attributes by introducing 47 human-interpretable descriptors and the Describable Texture Dataset (DTD).
• The paper leverages convolutional networks as deep filter banks, achieving state-of-the-art texture and material recognition through advanced pooling methods like Fisher vectors.
• The paper demonstrates that deep CNN features offer robust texture description and domain transferability in cluttered, real-world scenes, outperforming traditional techniques.

Deep Filter Banks for Texture Recognition, Description, and Segmentation

The paper "Deep Filter Banks for Texture Recognition, Description, and Segmentation" presents notable contributions to the understanding and characterization of textures in the context of computer vision. It is structured around three main objectives and deploys advanced methodologies that intertwine classic approaches with modern deep learning techniques.

The first significant contribution lies in the introduction of a vocabulary of 47 human-interpretable texture attributes, accompanied by the construction of the Describable Texture Dataset (DTD), which serves as a benchmark for texture attribute recognition. Traditional approaches predominantly focused on texture instance recognition; however, this work shifts the emphasis toward describing generic texture patterns. This shift facilitates the integration of texture attributes into various applications, such as organizing large collections of textures and augmenting material recognition tasks. Experimental results robustly demonstrate the efficacy of these attributes in providing a compact and semantic-rich description of textures.

In tackling the second objective, the authors address the problem of material and texture attribute recognition within cluttered and realistic scenes, contrasting traditional texture datasets that usually assume controlled setups. For this purpose, new benchmarks were derived from the OpenSurfaces (OS) dataset. These benchmarks create a more challenging environment, as textures may appear amidst visual clutters, reflecting real-world complexities. Recognition results in these conditions underscore the viability of the proposed methods.

The third contribution is a technical innovation that revisits and enhances classical texture models by leveraging convolutional networks to act as deep filter banks. The work explores the generalization and efficiency properties of deep feature extraction, utilizing convolutional layers as filter banks to process image regions. This approach yields state-of-the-art performance, showcasing the strengths of deep CNN features compared to traditional encoders like Fisher Vectors and bag-of-words models when combined with modern pooling techniques. The approach also highlights the substantial benefits of employing CNNs, notably in the ease of domain transferability across visual tasks without fine-tuning the networks.

The experiments indicate that pooling methodologies such as Fisher Vectors (FV-CNN) offer improvements over fully-connected layers (FC-CNN) across texture categorization, material, object, and scene recognition. Notably, FV-CNN provides superior performance in texture recognition and facilitates the successful transfer of learned features to novel domains. Furthermore, the paper illustrates that convolutional features pooled via Fisher encoding outperform standard methods by effectively capturing the spatial information inherent in the data.

From a theoretical perspective, this work contributes to bridging the understanding of texture representation in convolutional architectures, supporting the hypothesis that convolutional filters, interpreted as deep filter banks, contribute positively to comprehensive texture analysis frameworks. Practically, the deployment of effective texture attributes and deep filter banks augments real-world texture recognition applications, shedding light on future directions in texture analysis and paving the way for more generalized approaches in various computer vision tasks.

Looking forward, this paper's methodologies underline the potential for further exploiting deep neural networks in texture recognition, including areas such as unsupervised learning, real-time applications, and enhanced texture synthesis. The fundamental insights into texture characteristics offered by this work form a pivotal basis for subsequent research and development in the broader field of visual understanding.