UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory (1609.02132v1)

Abstract: In this work we introduce a convolutional neural network (CNN) that jointly handles low-, mid-, and high-level vision tasks in a unified architecture that is trained end-to-end. Such a universal network can act like a `swiss knife' for vision tasks; we call this architecture an UberNet to indicate its overarching nature. We address two main technical challenges that emerge when broadening up the range of tasks handled by a single CNN: (i) training a deep architecture while relying on diverse training sets and (ii) training many (potentially unlimited) tasks with a limited memory budget. Properly addressing these two problems allows us to train accurate predictors for a host of tasks, without compromising accuracy. Through these advances we train in an end-to-end manner a CNN that simultaneously addresses (a) boundary detection (b) normal estimation (c) saliency estimation (d) semantic segmentation (e) human part segmentation (f) semantic boundary detection, (g) region proposal generation and object detection. We obtain competitive performance while jointly addressing all of these tasks in 0.7 seconds per frame on a single GPU. A demonstration of this system can be found at http://cvn.ecp.fr/ubernet/.

• The paper introduces a single CNN that simultaneously handles diverse tasks like segmentation, detection, and boundary estimation.
• It employs a task-aware loss function to effectively integrate multiple datasets despite incomplete annotations.
• The architecture achieves competitive performance and sub-linear memory growth, processing frames in just 0.7 seconds on a single GPU.

Overview of UberNet: A Universal Convolutional Neural Network Architecture

The paper introduces UberNet, a CNN architecture designed to tackle various low-, mid-, and high-level vision tasks, operating under a unified framework. This research addresses significant technical challenges associated with training a single deep network across diverse datasets and tasks while managing limited computational memory.

Core Contributions

The UberNet architecture simultaneously processes an array of vision tasks: boundary detection, normal estimation, saliency estimation, semantic segmentation, human part segmentation, semantic boundary detection, and object detection. It achieves this in 0.7 seconds per frame on a single GPU, showcasing competitive performance across all tasks.

Technical Innovations

1. Diverse Dataset Integration: A crucial advancement in this work is the capacity to train on diverse datasets, where no single dataset contains all the requisite annotations for each task. A task-aware loss function is implemented, adapting to the available ground truth per sample. This enables training without the need for imputation of missing data, effectively integrating multiple datasets.
2. Memory Efficiency: UberNet is built to handle many tasks without escalating memory costs. Utilizing recent breakthroughs in memory-efficient backpropagation, the architecture maintains sub-linear memory complexity relative to the number of tasks. This allows practical implementation on current hardware without task-induced memory bloat.

Performance and Implications

The UberNet architecture exhibits robust performance across vision tasks, but some decline is observed when increasing the task count. For instance, while object detection shows a slight drop when transitioning from two tasks to seven, results remain close to dedicated models using separate networks. In tasks like semantic segmentation and boundary detection, skip layers and multi-resolution processing enhance performance.

This work hints at broader implications for multitask learning in CNNs, suggesting pathways to streamline computer vision applications via a unified network. The generative and discriminative capabilities of UberNet may influence future designs in AI, focusing on holistic solutions versus specialized, single-task networks.

Theoretical and Practical Impact

From a theoretical standpoint, UberNet represents a move toward architectural minimalism in complex task environments—it amalgamates tasks traditionally handled by discrete models into a cohesive system. Practically, such integration can reduce the computational footprint and increase the versatility of AI systems used in dynamic environments like autonomous driving or augmented reality.

Future Directions

The research opens avenues for more diverse task integration, application of deeper networks like ResNets, and the incorporation of structured prediction techniques to enhance task performance further. These directions will likely involve balancing task weightings to optimize the shared representation without sacrificing individual task precision.

In summary, UberNet typifies a significant stride in robust, multipurpose architectures, enabling efficient processing of complex visual input through a singular, streamlined model.