A multi-camera system for tracking freely-flying animals in realtime

Andrew D. Straw and Michael H. Dickinson

summary

Flydra is a realtime tracking system for performing marker-less tracking of moving image features, such as flying insects or birds. This multi-camera system runs simultaneously on multiple computers connected to multiple cameras to enable high spatial and temporal resolution in a large tracking volume with standard commercial components.

more details

While two cameras are sufficient to reconstruct 3D information and have previously been used in the Dickinson lab to track the position of fruit flies in realtime as they fly freely through large (e.g. 1m diameter) arenas, several issues prompted the development of a new realtime camera tracking system. Foremost among these were two desires: 1) an increase in spatial resolving power sufficient to determine body axis direction and 2) an increase in the spatial extent of the tracking volume. We recently developed a multiple camera system to meet these needs that extracts fly position and orientation in soft realtime. Currently, this system functions with 5 cameras at 100 frames per second with about 30 msec latency.

The imaging front end of the system consists of five monochrome digital cameras. Each camera is connected to a dedicated image processing computer using open source software we developed, which performs a glorified background-subtraction algorithm to extract local 2D points and, for each point where the "eccentricity" (oriented luminance) exceeds some criterion, an orientation is also extracted. These data are sent over gigabit ethernet to a central "mainbrain" computer where data from all cameras are combined into 3D position and orientation. The algorithm used (Hartley & Zisserman, 2000) constructs an observation matrix from each 2D image point and the camera calibration matrices, and the linear least squares fit for 3D position calculated using a singular value decomposition of this observation matrix. A similar operation is performed to extract 3D orientation. Currently, the cameras are calibrated using a simultaneous multi-camera calibration method (Svoboda, et al., 2005).