Multi-camera Realtime 3D Tracking of Multiple Flying Animals (1001.4297v1)

Abstract: Automated tracking of animal movement allows analyses that would not otherwise be possible by providing great quantities of data. The additional capability of tracking in realtime - with minimal latency - opens up the experimental possibility of manipulating sensory feedback, thus allowing detailed explorations of the neural basis for control of behavior. Here we describe a new system capable of tracking the position and body orientation of animals such as flies and birds. The system operates with less than 40 msec latency and can track multiple animals simultaneously. To achieve these results, a multi target tracking algorithm was developed based on the Extended Kalman Filter and the Nearest Neighbor Standard Filter data association algorithm. In one implementation, an eleven camera system is capable of tracking three flies simultaneously at 60 frames per second using a gigabit network of nine standard Intel Pentium 4 and Core 2 Duo computers. This manuscript presents the rationale and details of the algorithms employed and shows three implementations of the system. An experiment was performed using the tracking system to measure the effect of visual contrast on the flight speed of Drosophila melanogaster. At low contrasts, speed is more variable and faster on average than at high contrasts. Thus, the system is already a useful tool to study the neurobiology and behavior of freely flying animals. If combined with other techniques, such as `virtual reality'-type computer graphics or genetic manipulation, the tracking system would offer a powerful new way to investigate the biology of flying animals.

• The paper introduces Flydra, a novel system that achieves real-time, markerless 3D tracking of multiple flying animals with less than 40ms latency.
• It employs an Extended Kalman Filter combined with a Nearest Neighbor Standard Filter to triangulate accurate 3D positions from multiple 2D camera views.
• The study demonstrates that Drosophila increase flight speed at lower visual contrasts, highlighting its potential for advanced behavioral and neurobiological research.

Overview of Multi-camera Realtime 3D Tracking of Multiple Flying Animals

The paper details a sophisticated system for the automated tracking of flying animals using multi-camera setups, focusing on real-time 3D tracking of multiple targets such as flies and birds. The authors developed a system capable of operating with less than 40 milliseconds of latency while effectively handling multiple animals. This system's cornerstone lies in a combination of an Extended Kalman Filter and a Nearest Neighbor Standard Filter for data association.

Methodological Foundations

This real-time tracking system leverages a multi-camera setup that allows for high spatial and temporal resolutions over large volumes, addressing challenges such as occlusion and the necessity for high accuracy in experimental setups. The system, termed "flydra," integrates established algorithms and off-the-shelf hardware to provide an efficient platform for the paper of animal behavior and neurobiology.

The system's primary innovation is its ability to use an arbitrary number of inexpensive cameras to achieve markerless, real-time multi-target tracking. Bayesian frameworks underpin the multi-target tracking approach, wherein the posterior probability distribution of each target's state is recursively updated using the Extended Kalman Filter. This allows the system to maintain and update accurate estimations of target positions, even as multiple targets move through the tracking volume.

Components and Algorithms

The system performs 2D feature extraction using a background subtraction algorithm, which is crucial for processing real-time digital images into actionable data points. Subsequent steps involve computing the optimal 3D position by triangulating from multiple 2D camera views, a process managed by a low-latency data association routine that employs the Nearest Neighbor Standard Filter.

Flydra's real-time tracking capabilities are supported by an architecture utilizing high-speed communication buses and multi-threaded computational processes. The model incorporates mechanisms for dealing with the targets entering and leaving the tracking volume, demonstrating the robustness required for long-duration and large-scale experiments in naturalistic settings.

Results and Implications

An experiment detailed within the paper investigated the effect of visual contrast on the flight speed of Drosophila melanogaster. The results illustrated that flies tend to fly faster and with more variability at lower contrasts. Thus, the real-time monitoring capability of flydra facilitates intricate behavioral studies and stands to be a powerful tool for exploring the interplay between sensory stimuli and animal behavior.

Future Directions

The authors suggest future integration with other experimental methods, such as virtual reality systems and genetic manipulations, to create a more comprehensive toolkit for biological research. This might include the enhancement of tracking capabilities to a larger number of targets or increased precision in more complex environments.

Given the current transitions in AI and machine learning, particularly in computer vision, the system could incorporate advanced methods in tracking and data interpretation, potentially using deep learning frameworks to enhance data association and target recognition.

Conclusion

This research presents a substantial advancement in the biological paper of flight through its multi-camera, real-time tracking system, flydra. It not only provides valuable insights into animal behavior research but also lays a foundation for more refined experimental methodologies combining real-time tracking with responsive environmental manipulations. This platform holds promise for significant contributions to the field of neurobiology, particularly in understanding the neural control of complex behaviors.