Markerless tracking of user-defined features with deep learning (1804.03142v1)

Abstract: Quantifying behavior is crucial for many applications in neuroscience. Videography provides easy methods for the observation and recording of animal behavior in diverse settings, yet extracting particular aspects of a behavior for further analysis can be highly time consuming. In motor control studies, humans or other animals are often marked with reflective markers to assist with computer-based tracking, yet markers are intrusive (especially for smaller animals), and the number and location of the markers must be determined a priori. Here, we present a highly efficient method for markerless tracking based on transfer learning with deep neural networks that achieves excellent results with minimal training data. We demonstrate the versatility of this framework by tracking various body parts in a broad collection of experimental settings: mice odor trail-tracking, egg-laying behavior in drosophila, and mouse hand articulation in a skilled forelimb task. For example, during the skilled reaching behavior, individual joints can be automatically tracked (and a confidence score is reported). Remarkably, even when a small number of frames are labeled ($\approx 200$), the algorithm achieves excellent tracking performance on test frames that is comparable to human accuracy.

• The paper introduces a markerless deep learning method adapting the DeeperCut framework to achieve human-level pose estimation on animal behavior tasks.
• By leveraging transfer learning with minimal annotated frames, the approach accurately tracks diverse body parts across challenging experimental contexts.
• This framework paves the way for ethical, scalable, and real-time behavioral quantification, revolutionizing experimental neuroscience protocols.

Markerless Tracking of User-Defined Features with Deep Learning

The quantification of behavior is fundamental in neuroscience, serving as a cornerstone for exploring brain function and animal behavior. However, traditional methods of behavioral quantification often require marking subjects with reflective markers, which can be intrusive and pre-defines the body parts to be tracked. The paper "Markerless Tracking of User-Defined Features with Deep Learning" by Mathis et al. introduces an efficient, markerless method using deep learning for tracking various body parts in diverse experimental settings.

Methodology

The core of this paper is the adaptation and application of the DeeperCut framework for lab-specific pose estimation problems. DeeperCut is a cutting-edge deep learning model designed for human pose estimation that extends the capabilities of Deep Residual Networks (ResNets). By leveraging transfer learning, the model significantly reduces the volume of annotated training data required to achieve high accuracy.

The process involves several steps:

1. Data Collection and Pre-Processing: Distinct frames representing important postures are extracted from videos.
2. Manual Annotation: Various body parts of interest are manually labeled in these frames.
3. Training: A deep neuro network, tailored to predict the labeled body-part locations, is trained using these annotations. Transfer learning is employed by initializing the network with weights pre-trained on large image datasets like ImageNet.
4. Prediction and Evaluation: Once trained, the network is used to predict the body part locations in new videos, followed by a thorough evaluation of its performance metrics (e.g., Root Mean Square Error - RMSE).

Throughout the experiments, the network's predictions are benchmarked against human-level accuracy, ensuring the model's reliability.

Experimental Validation

The proposed framework was validated across multiple experimental contexts:

1. Odor Trail-Tracking in Mice: This task involves mice following odor trails on an 'endless' spool of paper. The video data posed notable challenges including inhomogeneous illumination, shadows, and distortions due to a wide-angle lens. Despite these challenges, the trained network achieved human-level accuracy for key body parts, such as the snout and tail base, with minimal training data (around 200 frames).
2. Tracking in Drosophila: The versatility of the framework was further demonstrated by tracking various body parts of freely behaving fruit flies in a 3D chamber. Despite significant postural and background variations, the network consistently located parts with a mean error of about 4 pixels—close to that of human annotation accuracy.
3. Digit Tracking During Reach Tasks in Mice: Here, the task was to track individual mouse digits which have intricate and highly variable articulations. The model required only a few training frames to achieve a test error that was well within acceptable limits for practical utility.

In addition to these specific tasks, the paper explored the potential for the network's generalization and transfer learning capabilities, notably demonstrating that a network trained on single-mouse frames could recognize multiple mice in new, unseen scenarios—an invaluable property for studying social behaviors.

Implications

The implications of this research are vast for both practical applications and theoretical advancements in AI and neuroscience:

1. Efficiency and Ease of Use: This method significantly reduces the need for extensive manual annotation and the use of intrusive markers, leading to more ethical and less stressful conditions for animal subjects.
2. Scalability: The ability to train accurate models with minimal data paves the way for broader applications across various species and experimental setups.
3. Future Applications: This framework holds promise for real-time tracking and automated behavioral analysis, potentially revolutionizing experimental protocols in neuroscience and beyond.

Future Directions

Future developments in AI and deep learning could further enhance the capabilities and applications of this framework. Areas of particular interest include incorporating real-time feedback mechanisms, extending the model to additional species and behaviors, and improving the model's robustness to occlusions and complex interactions, as often seen in social behaviors.

Conclusion

In summary, Mathis et al.'s paper on markerless tracking using deep learning presents a powerful, adaptable, and efficient framework for behavioral quantification in neuroscience. By leveraging advanced deep learning models and transfer learning, this approach achieves high accuracy with minimal training data, opening new avenues for experimental research and analysis without the typical constraints of traditional methods. The implications of this work underscore both immediate practical benefits and long-term potential for advancing our understanding of behavior through improved computational tools.