Kornia: an Open Source Differentiable Computer Vision Library for PyTorch (1910.02190v2)

Abstract: This work presents Kornia -- an open source computer vision library which consists of a set of differentiable routines and modules to solve generic computer vision problems. The package uses PyTorch as its main backend both for efficiency and to take advantage of the reverse-mode auto-differentiation to define and compute the gradient of complex functions. Inspired by OpenCV, Kornia is composed of a set of modules containing operators that can be inserted inside neural networks to train models to perform image transformations, camera calibration, epipolar geometry, and low level image processing techniques, such as filtering and edge detection that operate directly on high dimensional tensor representations. Examples of classical vision problems implemented using our framework are provided including a benchmark comparing to existing vision libraries.

• The paper introduces Kornia as a GPU-optimized library that embeds differentiable vision functions into neural network pipelines using PyTorch's auto-differentiation.
• It details a modular API supporting classical tasks like image transformations, camera calibration, and feature matching with seamless hardware acceleration.
• Extensive use cases, including batch processing, image registration, and depth estimation, demonstrate its efficiency and potential for advanced vision applications.

Overview of Kornia: A Differentiable Computer Vision Library for PyTorch

The paper presents "Kornia," an open-source library designed to address challenges in modern deep learning applications related to computer vision. Built around the PyTorch ecosystem, Kornia provides differentiable routines that integrate seamlessly into neural network workflows, offering functionalities for classical computer vision tasks such as image transformations, camera calibration, and epipolar geometry.

Key Design Principles and Features

Kornia builds upon earlier CPU-focused libraries by shifting emphasis to GPU-optimized operations, thereby taking advantage of PyTorch's reverse-mode auto-differentiation for gradient calculation. The library incorporates several critical design features:

• Differentiability: Operators in Kornia can be embedded as layers within neural networks, enabling backpropagation through complex vision functions. This is facilitated by PyTorch’s automatic differentiation framework.
• Transparent API: Kornia's API is designed to be agnostic to the input data source, whether it is CPU or GPU-based, allowing for seamless hardware acceleration with minimal user intervention.
• Parallel and Distributed Processing: The framework supports data parallelism through batch processing and distributed multiprocess parallelism across computation nodes, suitable for large-scale applications.
• Production-Ready: By leveraging PyTorch's just-in-time (JIT) compiler, models utilizing Kornia can be serialized and optimized for deployment in production environments.

Library Structure

Kornia is modular, with submodules tailored to specific vision tasks:

• Color and Filters: Operators for color space conversion and image filtering are available, covering transformations like RGB to Grayscale and edge detection via Sobel.
• Features and Geometry: The library includes local feature detection and description, as well as 2D and 3D geometric transformations and conversions.
• Losses and Contrib: Specialized loss functions for vision tasks, alongside experimental operators, provide the flexibility needed for developing custom models or enhancing existing ones.

Experimental Use Cases

The utility of Kornia is demonstrated through several use cases:

1. Batch Image Processing: Kornia outpaces traditional libraries in GPU scenarios for operations like Sobel edge detection, highlighting its processing efficiency with native PyTorch optimizations.
2. Image Registration: Utilizing a multi-scale Lucas-Kanade approach, Kornia optimizes homography parameters to minimize photometric error between images, illustrating its efficacy in tasks demanding 2D planar geometry handling.
3. Depth Estimation: A pipeline employing the Kornia DepthWarper operator allows for the estimation of depth maps from multiple calibrated camera images, integrating seamlessly into end-to-end differentiable depth reconstruction tasks.
4. Adversarial Feature-Matching Attack: Demonstrating differentiable wide baseline stereo matching, Kornia’s feature operators are capable of adversarial attacks, compelling matches in input pairs designed to evade standard matching criteria.

Implications and Future Directions

Kornia serves as a bridge between traditional computer vision methodologies and contemporary end-to-end deep learning architectures. Its GPU-accelerated, differentiable nature enables the embedding of classical vision algorithms directly within neural network training processes, reducing the gap between pre/post-processing and core model computation. The library’s adoption could lead to richer data augmentation strategies and novel, integrated network architectures in practical computer vision applications.

Future research and development using Kornia may explore expanding its operator set, optimizing existing functions further for specific hardware architectures, and exploring novel integrations with cutting-edge neural network designs to enhance performance and capabilities in real-time vision systems. The community is encouraged to contribute, potentially accelerating the development of standardized, high-performance vision solutions within the deep learning framework.