Comparison of two 3D tracking paradigms for freely flying insects

Abstract

In this paper, we discuss and compare state-of-the-art 3D tracking paradigms for flying insects such as Drosophila melanogaster. If two cameras are employed to estimate the trajectories of these identical appearing objects, calculating stereo and temporal correspondences leads to an $\mathcal{N}\mathcal{P}$-hard assignment problem. Currently, there are two different types of approaches discussed in the literature: probabilistic approaches and global correspondence selection approaches. Both have advantages and limitations in terms of accuracy and complexity. Here, we present algorithms for both paradigms. The probabilistic approach utilizes the Kalman filter for temporal tracking. The correspondence selection approach calculates the trajectories based on an overall cost function. Limitations of both approaches are addressed by integrating a third camera to verify consistency of the stereo pairings and to reduce the complexity of the global selection. Furthermore, a novel greedy optimization scheme is introduced for the correspondence selection approach. We compare both paradigms based on synthetic data with ground truth availability. Results show that the global selection is more accurate, while the previously proposed tracking-by-matching (probabilistic) approach is causal and feasible for longer tracking periods and very high target densities. We further demonstrate that our extended global selection scheme outperforms current correspondence selection approaches in tracking accuracy and tracking time.

1 Introduction

The investigation of complex movement patterns of various organisms has become an integral subject of biological research. From a biological point of view, motion is the visual response to any kind of perceivable stimulation. The nervous system is responsible for the perception, the integration of the information, and the execution of the final response. One of the most popular model organisms to study how the nervous system controls locomotion is Drosophila melanogaster (i.e., fruit fly). Sophisticated genetic tools as well as advanced imaging techniques allow the functional dissection of neural circuits [1–4].

Drosophila is a holomethabolous insect. In the larval stage, locomotion is confined to two dimensions, whereas the adult fly moves in two and three dimensions. Approaches dealing with crawling larvae are common praxis; thus, two-dimensional (2D) tracking is well established in behavioral experiments [3, 5–8]. In addition, flies confined to 2D motion are often used in behavioral experiments [2, 9–11]. Basically, there are two ways to prevent the flies from takeoff: cutting the wings [12] or using an arena with a flat ceiling [13]. Both manipulations could lead to unnatural behavior [14]. Thus, three-dimensional (3D) tracking approaches are needed to address all kinds of behavioral phenotypes.

1.1 Related work

Work on freely flying fruit flies is still in its infancy, because it requires dynamic 3D correspondence analysis [15]. This analysis involves two challenging tasks: stereo matching (i.e., correspondence between camera views) and temporal tracking (i.e., correspondence over time). Together they form the so-called general multi-index assignment problem [16]. This problem is non-deterministically polynomial-time hard ( $\mathcal{N}\mathcal{P}$-hard) [16]. If all correspondences are known, triangulation is used to determine the 3D positions.

To avoid expensive multi-camera multi-target 3D tracking, existing approaches typically either track in two dimensions (no stereo matching) [2, 9–11] or track only a single target (no ambiguities over time) [14, 17]. If multi-camera multi-target 3D tracking is required, stereo matching and temporal tracking can be solved separately by accepting a decrease of tracking accuracy [18–20].

Among others, there are two fundamentally different paradigms used to capture 3D trajectories of multiple adult Drosophila. The first paradigm uses the extended Kalman filter and avoids complexity by separating stereo and temporal correspondence associations [21, 22]. Due to this separation, optimal results cannot be guaranteed, and fragmented tracks prevent the preservation of the fly identities over time. The second paradigm performs a global selection by combining both tasks to calculate the overall best assignment [23]. As a result, identity preservation can be achieved for many flies and frames. However, the amount of possible combinations increases exponentially with the number of animals and time steps; thus, current solutions are only able to track for a short period.

Another probabilistic approach addresses the trade-off between identity preservation and long-term experiments [24]. The authors use the Hungarian algorithm and Kalman filtering for stereo matching and temporal correspondence association. Focusing on applicability for biologists, up to seven flies were evaluated in several experiments.

All the above-mentioned approaches focus on either tracking a few hundreds of targets for a short period of time or tracking less targets for more frames. High-density tracking is used in different research areas like particle tracking velocimetry [18, 25] and tracking bats [26, 27], bees [28] or fruit flies [20, 23]. A quantitative comparison of several three-dimensional Lagrangian particle tracking approaches for high-density situations is given in [29].

Examples of long-term tracking approaches for fruit flies are given in [21, 22, 24, 30, 31]. In a recent publication, problems like noise and low frame rates are addressed to calculate trajectories of wild mosquitoes [32]. The authors used a probabilistic multi-target tracking for swarms of 6 to 25 mosquitoes. If hundreds of flies are tracked for a comparatively long period, trajectories are fragmented and the identity is not preserved. Furthermore, tracking several hundreds of flies simultaneously is not practical for most biological applications [24, 30]. Only if swarming behavior needs to be analyzed, ambiguous animals are neglected leading to a strongly varying number of targets over time [33].

In a recent publication, multi-path branching was used to handle occlusions by employing global optimization when calculating the trajectories [34]. The algorithm was exhaustively tested for both high-density and long-term situations. Again, the tracking accuracy decreases if the number of targets and the number of frames increases simultaneously.

1.2 Proposed algorithms and comparison scheme

In this paper, we compare identity preserving 3D tracking approaches for long-term experiments considering biological usability. First, we present algorithms for both the above-mentioned paradigms (see Figure 1):

• The previously proposed tracking-by-matching (TbM) solution [35] integrates a third camera to conduct projection consistency check into the probabilistic approach.

• In addition, we introduce a global correspondence selection (GCS) algorithm (extension of [23]), calculating the global search space and minimizing a cost function afterwards.

Figure 1

General comparison scheme of this paper. The new approach is highlighted in yellow.

Limitations of the TbM and the GCS approach are addressed by utilizing a third camera to verify the consistency of stereo pairings. The third camera is integrated by the so-called projection consistency [35]. As a result, the amount of ambiguous temporal associations is reduced in the TbM approach. The GCS approach benefits from the projection consistency by means of a reduced overall complexity. Besides utilizing Gibbs sampling for optimization, as suggested by [23], we introduce an alternative greedy selection scheme (see Figure 1). It should be pointed out that we use GCS in terms of optimizing a global search space, not determining the global optimum for our optimization task.

We compare both paradigms, the TbM and the GCS approach, based on synthetic data; thus, the ground truth is available. Global correspondence selection was done via Gibbs sampling [23] and greedy optimization utilizing projection consistency. This leads to the comparison scheme illustrated in Figure 1.

This paper is organized as follows: In Section 2, we provide notes about notations and central equations. In particular, the projection consistency is described in detail. Algorithms are presented in Section 2.2. Section 2.2.3 describes the extensions of the GCS approach. The synthetic data and measures used for comparison are described in Section 3. All results are listed in Section 4: We compare GCS approaches with the TbM approach in Sections 4.2, 4.3, and 4.4. In addition, we compare both GCS approaches in more detail in Section 4.1. A concluding discussion of both paradigms is given in Section 5.

2 Methods

Both algorithms expect time-synchronized image streams from up to three cameras. Let ${\mathcal{I}}_t^i$ represent these images of cameras i = 1, 2, 3, and time t = 1, …, T. All cameras need to be calibrated; thus, the camera matrices Kⁱ, rotation matrices (from camera i to camera j) R^ij, and translation vectors t^ij are given. Then, the fundamental matrices can be calculated by

$${\mathbf{\text{F}}}^{\mathit{\text{ij}}}=\left({\mathbf{\text{K}}}^j{\mathbf{\text{t}}}^{\mathit{\text{ij}}}\right)\times \left({\mathbf{\text{K}}}^j{\mathbf{\text{R}}}^{\mathit{\text{ij}}}{({\mathbf{\text{K}}}^i)}^{-1}\right)$$

(1)

(for more details, see [36]). Consider a swarm of flying targets of similar appearance and small size. The centers of detected targets (i.e., blobs) in a single image ${\mathcal{I}}_t^i$ are denoted by $M_t^i=\{m_{n^i,t}^i\}=\{(x,y)\}$ for $n^i=1,\dots ,N_t^i$ targets at time t, where (x, y) is the image coordinate of the objects’ centroid. The value $N_t^i$ may differ due to occlusions or noise.

To calculate the 3D positions of the flies, stereo correspondences between detected blobs need to be established. Since we use three cameras, triplets of image points $(m_{n^1,t}^1,m_{n^2,t}^2,m_{n^3,t}^3)$ correspond to one target. In general, two 2D image coordinates are sufficient to calculate a single 3D position; thus, we define all possible pairs given by $H_t^{\mathit{\text{ij}}}=M_t^i\times M_t^j$ between camera i and j. A pairing $s_{k,t}^{\mathit{\text{ij}}}\in H_t^{\mathit{\text{ij}}}$ could represent either a true or a false correspondence for target k.

2.1 Stereo matching and projection consistency

Both paradigms perform stereo matching based on epipolar geometry and verify matches using the so-called projection consistency constraint [35].

2.1.1 Stereo matching

Stereo matching is used to identify possible pairings between two respective views and thus result in possible correspondences. For matching a point $m_{n^i,t}^i$ in ${\mathcal{I}}_t^i$ with a point $m_{n^j,t}^j$ in ${\mathcal{I}}_t^j$, both need to be located on the same epipolar line. Given $m_{n^i,t}^i$, the corresponding epipolar line ${\mathbf{\text{l}}}_{n^i}^j$ in ${\mathcal{I}}_t^j$ can be calculated by

$${\mathbf{\text{l}}}_{n^i}^j={\mathbf{\text{F}}}^{\mathit{\text{ij}}}{m}_{n^i,t}^i.$$

(2)

Detected points from $M_t^j$ lying on this epipolar line are matched to $m_{n^i,t}^i$ and indicate possible pairings $\{s_{k,t}^{\mathit{\text{ij}}}\}\subset H_t^{\mathit{\text{ij}}}$.

2.1.2 Projection consistency

Since we use three calibrated cameras, triplets of 2D points $(m_{n^1,t}^1,m_{n^2,t}^2,m_{n^3,t}^3)$ located in images ${\mathcal{I}}_t^1,{\mathcal{I}}_t^2$, and ${\mathcal{I}}_t^3$ correspond to the same target in the 3D space. Projection consistency is applied to those triplets to verify the overall match. This constraint is satisfied if the respective projections from two 2D points $(m_{n^i,t}^i,m_{n^j,t}^j)$ into the third view ${\mathcal{I}}_t^h$ are sufficiently close to $m_{n^h,t}^h$. To project $(m_{n^i,t}^i,m_{n^j,t}^j)$ into ${\mathcal{I}}_t^h$, we calculate the 3D coordinate $p_t^h$ by triangulating $(m_{n^i,t}^i,m_{n^j,t}^j)$ and use the camera matrices from ${\mathcal{I}}^h$ to obtain the hypothetical 2D position ${\tilde{m}}_t^h$ in ${\mathcal{I}}_t^h$. Afterwards, we search for the closest point $m_{\ast ,t}^h$ in $M_t^h$ by

$$m_{\ast ,t}^h=\underset{n^h\in \{1,\dots ,N_t^h\}}{\text{min}}\left(\text{dist}({\tilde{m}}_t^h,m_{n^h,t}^h)+{({\mathbf{\text{l}}}_{n^i}^h)}^T{m}_{n^h,t}^h\right),$$

(3)

where the first summand is the Euclidean distance between the hypothetic position ${\tilde{m}}_t^h$ and the measured positions in $M_t^h$ and the second summand is the distance between a measured point and the epipolar line ${\mathbf{\text{l}}}_{n^i}^h$ in view ${\mathcal{I}}^h$ corresponding to view ${\mathcal{I}}^i$. If

$$\text{dist}({\tilde{m}}_t^h,m_{\ast ,t}^h)<\tau ,$$

(4)

then blob ${\tilde{m}}_t^h$ (and thus the underlying pairing $(m_{n^i,t}^i,m_{n^j,t}^j)$) describes a correct stereo correspondence. τ is the threshold for the projection consistency and depends on calibration accuracy. The triplet $(m_{n^i,t}^i,m_{n^j,t}^j,m_{n^h,t}^h)$ satisfies the projection consistency if $m_{n^h,t}^h=m_{\ast ,t}^h$, for all possible combinations i, j, h ∈ {1, 2, 3}, (i ≠ j ≠ h).

2.2 Presented algorithms

To compare current state-of-the-art tracking paradigms, we introduce the TbM algorithm and a GCS algorithm. We try to overcome the limitations of probabilistic tracking, namely the separation of stereo matching and temporal tracking, by integrating projection consistency into the temporal tracking routine. Exponential complexity, arising in global correspondence selection algorithms, is avoided by reducing the global search space based on the projection consistency. Correspondence selection can be done by Gibbs sampling [37] or in a greedy manner. Since we introduce a novel selection scheme for the GCS approach (yellow box in Figure 1), we describe it in more detail. The TbM is explicitly described in [35].

2.2.1 Tracking-by-matching algorithm

As in all probabilistic tracking approaches, our tracking algorithm models the position and motion information of the targets independently from the stereo correspondences between the views. We use the unscented Kalman filter (UKF) as a Bayesian framework for 2D tracking [8, 38]. Using the notation introduced above, every target $m_{n^i,t}^i$ in every view (i ∈ {1, 2, 3}) is represented by its own tracker $T_k^i$ (i.e., a single UKF). Temporal tracking is achieved by referring one of the measured nⁱ-th targets to a specific tracker k over time t (yellow box 'temporal tracking’ in Figure 2). Thus, for each new frame triplet, every UKF predicts the next possible 2D position for its target. Then, detections close to the predictions are verified with projection consistency (green box 'Verify new triplet via proj. consist.’; for details, refer to [35]). In this way, the projection consistency constraint is used to integrate stereo matching into temporal tracking. After updating all trackers $T_k^i$, this procedure is repeated as long as there are further frames available (see Figure 2).

Figure 2

Flow charts of the two 3D tracking paradigms. Temporal tracking is marked in yellow, stereo matching is marked in green, and projection consistency is highlighted by dots.

2.2.2 Global correspondence selection

Before going into formal details, the following section introduces the GCS algorithm in a top-down manner.

General workflow of the GCS approach. In distinction from the TbM approach, temporal tracking and stereo matching is not done within the main loop (compare to Figure 2). In fact, a global search space named is constructed over all the accessible frames before the actual tracking (compare to box 'Construction of ’). Afterwards, the best possible assignments are calculated by minimizing an overall cost function operating on .

To reduce the size of this search space, epipolar and temporal assumptions are made before considering actual correspondences. As illustrated in Figure 2, possible stereo correspondences between cameras 1 and 2 are calculated for every time step. Only blob pairings close to their respective epipolar lines are considered as possible stereo matches (box 'Calculate stereo correspondences’). If projection consistency is used (dashed box in Figure 2), invalid matches are removed or replaced via Equation 4. Note that the third camera is only used to replace incorrect matches from cameras 1 and 2. In other words, projection consistency is used to further reduce the set of possible pairings arising from ${\mathcal{I}}^1$ and ${\mathcal{I}}^2$. The resulting set of matches can be interpreted as a set of possible 3D positions. Given two sets of 3D positions for consecutive time steps, possible temporal assignments can be calculated (compared to box 'Calculate temporal correspondences’). If a target has no successor within a 3D neighborhood (given by the maximal flight speed), it is removed from .

This reduction is done for all available frames and time steps T. After constructing the search space, several assignments are unique. Ambiguous pairings and ambiguous temporal correspondences form natural clusters in . Thus, only samples inside these clusters need to be optimized (see box 'Get ambiguous clusters from ’ in Figure 2). The subsequent optimization is done by a cost function introduced below which incorporates stereo and temporal matching (Equation 6). We implemented two optimization strategies to find possible samples in the respective clusters (see box 'Greedy / Gibbs cluster optimization’).

Formal definition of the search space. Let $\mathbb{P}(H_t^{\mathit{\text{ij}}})$ be the power set of all pairings $s_{k,t}^{\mathit{\text{ij}}}\in H_t^{\mathit{\text{ij}}}$. To avoid additional complexity, arising from pairwise pairings between three views, we use cameras 1 and 2 for stereo matching. Thus, the initial search space is constructed for $H_t^{12}$. Blobs from ${\mathcal{I}}_t^3$ are only considered if projection consistency is used (see Section 2.2.3).

The subset containing N _t pairings from the power set $\mathbb{P}(H_t^{12})$ is given by the set of N _t permutations ${\mathcal{P}}_{N_t}(H_t^{12})\subset \mathbb{P}(H_t^{12})$ (i.e., ${\mathcal{P}}_{N_t}(H_t^{12})$ contains all possible combinations of pairings with N _t elements). A single set $S_t^{12}=(s_{1,t}^{12},s_{2,t}^{12},\dots ,s_{N_t,t}^{12})\in {\mathcal{P}}_{N_t}(H_t^{12})$ is called a configuration and contains N _t stereo correspondences for time step t. If camera indices are not necessary, we use $(s_{1,t},s_{2,t},\dots ,s_{N_t,t})=S_t=S_t^{\mathit{\text{ij}}}$ and $H_t=H_t^{\mathit{\text{ij}}}$ for i, j ∈ {1, 2, 3}, i ≠ j.

Let $\mathfrak{S}=(\mathbb{P}(H_1),\mathbb{P}(H_2),\dots ,\mathbb{P}(H_T))$ be the set of all configurations over all time steps, or ${\mathfrak{S}}_{\mathcal{P}}=({\mathcal{P}}_{N_1}(H_1),{\mathcal{P}}_{N_2}(H_2),\dots ,{\mathcal{P}}_{N_T}(H_T))$ if the number of targets per time step is known. A sequence of configurations between two time steps t - 1 and t is denoted by S _t-1:t and contains temporal correspondences between consecutive frames. Thus, an overall solution, containing all tracks for all flies and T time steps, is given by a sequence $S_{1:T}\in \mathfrak{S}$. The entire 3D trajectory of target k is then given by s _k,1:T  = (s _k,1, s _k,2, … ,s _k,T ).

Cost function. Stereo matching and temporal tracking is incorporated into a single optimization task, solving the optimization problem

$$S_{1:T}^{\ast }=\text{arg}\underset{S_{1:T}\in \mathfrak{S}}{\text{min}}f(S_{1:T}).$$

(5)

The cost function f(•) incorporates an epipolar constraint f _E (•) for stereo matching, kinetic coherence f _K (•) for temporal tracking, and a so-called conservation-observation match f _C (•) to punish multiple assignments. Thus, f(•) can be written as a sum of all the abovementioned constraints

$$\begin{array}{ll}f(S_{1:T})& =\alpha \sum _{t=1}^T{f}_E(S_t)+\beta \sum _{t=1}^T{f}_C(S_t,H_t)\\ +\gamma \sum _{t=1}^T{f}_K(S_{t-1},S_t)\end{array}$$

(6)

with weights α, β, and γ (compare to [23]).

Cost function summands. Epipolar costs are defined as

$$f_E(S_t)=\sum _{k=1}^{N_t}{\rho }_e(s_{k,t}),$$

where ρ _e (s _k,t ) sums the distances between the blobs $m_{k,t}^i,m_{k,t}^j$ from s _k,t to its epipolar lines (compare Section 2.1.1). To avoid improbable stereo matchings, values f _E (•) larger than a threshold ε _E are set to ∞:

$$f_E(S_t)=\left\{\begin{array}{ll}\sum _{k=1}^{N_t}{\rho }_e(s_{k,t})& \forall k:{\rho }_e(s_{k,t})<{\epsilon }_E\\ \infty & \text{otherwise}.\end{array}\right.$$

(7)

The kinetic coherence

$$f_K(S_{t-1},S_t)=\sum _{k=1}^{N_t}{\rho }_k(s_{k,t-1},s_{k,t})=\sum _{k=1}^{N_t}\text{dist}(p_{k,t-1},p_{k,t})$$

calculates the Euclidean distances dist(•) between 3D positions p _k,t-1 and p _k,t (defined by s _k,t-1 and s _k,t ). ρ _k (•) expects two consecutive pairings for 3D coordinate calculation. Improbable temporal connections are set to ∞:

$$f_K(S_{t-1},S_t)=\left\{\begin{array}{ll}{\sum }_{k=1}^{N_t}{\rho }_k(s_{k,t-1},s_{k,t})& \forall k:{\rho }_k(s_{k,t-1},s_{k,t})<{\epsilon }_K\\ \infty & \text{otherwise.}\end{array}\right.$$

(8)

Finally, the conservation observation match is defined as

$$\begin{array}{ll}{f}_C(S_t,H_t)& =\frac{1}{N_t^1}\sum _{k=1}^{N_t^1}|n_c(m_{k,t}^1,S_t)-1|\\ +\frac{1}{N_t^2}\sum _{k=1}^{N_t^2}|n_c(m_{k,t}^2,S_t)-1|,\end{array}$$

where $n_c(m_{k,t}^i,S_t)$ adds up the contributions of a blob $m_{k,t}^i$ in configuration S _t . If the number of correspondences exceeds a threshold ε _C , configuration costs are set to ∞:

$$f_C(S_t,H_t)=\left\{\begin{array}{ll}{\rho }_c(S_t,M_t^1)+{\rho }_c(S_t,M_t^2)& \forall k:n_c(m_{k,t},S_t)<{\epsilon }_C\\ \infty & \text{otherwise}.\end{array}\right.$$

(9)

where ${\rho }_c(S_t,M_t^i)=\frac{1}{N_t^i}{\sum }_{k=1}^{N_t^i}|n_c(m_{k,t}^i,S_t)-1|$.

Recursive decomposition. Equation 6 can be rewritten in a recursive manner as follows:

$$f(S_{1:T})=f(S_{1:t-1})+\mathrm{\Delta }f(S_t)$$

(10)

with

$$\mathrm{\Delta }f(S_t)=\alpha f_E(S_t)+\beta f_C(S_t,H_t)+\gamma f_K(S_{t-1},S_t).$$

(11)

Thus, the whole optimization can be done by dynamic programming (for more details, see [23]).

Reduction of $\mathfrak{S}.$ In [23], Gibbs sampling [37] is suggested to find the best possible sequence of configurations $S_{1:T}^{\ast }\in \mathfrak{S}$. Since is a set of T power sets $\mathbb{P}(H_t)$, several steps are suggested to reduce the search space. First of all, sampling for solutions with N _t targets for time t leads to a reduced set ${\mathcal{P}}_{N_t}(H_t)$. Thus, we redefine the overall search space for $S_{1:T}^{\ast }$ by ${\mathfrak{S}}_{\mathcal{P}}=({\mathcal{P}}_{N_1}(H_1),{\mathcal{P}}_{N_2}(H_2),\dots ,{\mathcal{P}}_{N_T}(H_T))$.

The set H _t is reduced by rejecting pairings which do not satisfy Equation 7. In the remaining subset ${\mathfrak{H}}_t\subset H_t$, only blob pairings s _k,t close to the respective epipolar lines are considered. Due to the recursive decomposition given in Equation 10, the successor to S _t-1 can be selected from the N _t permutation ${\mathcal{P}}_{N_t}({\mathfrak{H}}_t)$. Since kinetic costs are limited (see Equation 8), improbable temporal correspondences can be rejected from ${\mathcal{P}}_{N_t}({\mathfrak{H}}_t)$. Figure 3 illustrates the reduction of the cardinality for an N ₁ permutation.

Figure 3

Example for the cardinality reduction of . For a given time t, only one target needs to be found; thus, $\mathbb{P}(H_t)$ reduces to ${\mathcal{P}}_{N_1}(H_t)$. The resultant search space is illustrated in column ${\mathcal{P}}_{N_1}(H_t)$, containing only combinations of two blobs. The red blob detections do not satisfy the epipolar constraint (illustrated via red epipolar lines). In addition, the kinetic costs exceed the limit (indicated by the search sphere around the red blob). Thus, the cardinality of ${\mathcal{P}}_{N_1}(H_t)$ further decreases to ${\mathcal{P}}_{N_1}({\mathfrak{H}}_t)$, containing a single unique pairing.

After rejecting both impossible stereo matchings and temporal correspondences, some sequences of configurations $S_{t-1:t}\in ({\mathcal{P}}_{N_{t-1}}({\mathfrak{H}}_{t-1}),{\mathcal{P}}_{N_t}({\mathfrak{H}}_t))$ are unique. The remaining ambiguities form natural clusters ${\mathcal{C}}_{(t-\delta :t),\nu }\subset {\mathfrak{S}}_{\mathcal{P}}$ for δ + 1 frames and ν flies. Zou et al. [23] extend ambiguous clusters by adjacent pairings. However, these pairings can again be involved in an ambiguous cluster. Since we tried to keep the identity over time, we merged the clusters in these situations as long as there are no ambiguous situations before and after each cluster anymore. In this way, the resultant clusters include overall ambiguous situations, and the domain of Equation 5 is global.

Since the cluster size increases exponentially with the number of targets N and time steps T, Gibbs sampling also requires thousands of sampling steps to guarantee good results. Indeed, the authors of [23] were only able to track for less than 1 s of recording.

2.2.3 Introduced improvements for the GCS approach

Here, we introduce two extensions to improve the performance of the GCS approach:

• Utilizing projection consistency to reject ambiguous pairings s _k,t and thus reducing the sizes of the clusters

• Performing optimization in a greedy manner by selecting the best successor directly based on Equation 11

GCS with projection consistency. Similar to the above introduced probabilistic tracking approach, ambiguities and wrong stereo matches increase the size of the search space ${\mathfrak{H}}_t^{12}$. Thus, all pairings $s_{k,t}^{12}\in {\mathfrak{H}}_t^{12}$ are projected into the view of the third camera ${\mathcal{I}}^3$. Only pairings satisfying Equation 4 remain in ${\mathfrak{H}}_t^{12}$; inconsistent pairings are rejected.

An ambiguity is found if two pairings $s_{i,t}^{12},s_{j,t}^{12}$ in the remaining ${\mathfrak{H}}_t^{12}$ share a single blob m _k,t . The blob $m_{\ast ,t}^3$ satisfying Equation 4 is then used to generate a new pairing $s_{i,t}^{\ast }$, containing $m_{\ast ,t}^3$ from ${\mathcal{I}}_t^3$ and the unambiguous blob from ${\mathcal{I}}_t^1$ or ${\mathcal{I}}_t^2$. Afterwards, ambiguous pairings in ${\mathfrak{H}}_t^{12}$ are replaced by unique pairings $s_{i,t}^{\ast }$.

Let ${\mathfrak{H}}_t^{\ast }$ be the reduced subset, containing the new unique pairing $s_{i,t}^{\ast }$ (compare ' ${\mathfrak{H}}^{\ast }$ with PC’ in Figure 4). The overall search space for Equation 5 is then given by ${\mathfrak{S}}_{\mathcal{P}}^{\ast }=({\mathcal{P}}_{N_1}({\mathfrak{H}}_1^{\ast }),{\mathcal{P}}_{N_2}({\mathfrak{H}}_2^{\ast }),\dots ,{\mathcal{P}}_{N_T}({\mathfrak{H}}_T^{\ast }))$.

Figure 4

Example for pairings with and without projection consistency (PC). One target is occluded in ${\mathcal{I}}^2$, and all possible combinations are generated between ${\mathcal{I}}^1$ and ${\mathcal{I}}^2$. Pairings that do not satisfy the PC constraint are removed, and ambiguous pairings are corrected using projection consistency.

The optimization of clusters based on Equation 5 via Gibbs sampling is described in [23]. The greedy optimization strategy is described below.

Greedy optimization. Given a cluster with ambiguities ${\mathcal{C}}_{(t-\delta :t),\nu }\subset {\mathfrak{S}}_{\mathcal{P}}^{\ast }$, a sequence of configurations $S_{t-\delta :t}\in {\mathcal{C}}_{(t-\delta :t),\nu }$ must be determined. Let the sequence of configurations be S _t-δ:t  = (S ₀, S ₁, …, S _δ ), which must be optimized for ν flies. For a given S _i  = (s _{1, i} , s _{2, i} , …, s _ν,i ), a successor to every pairing s _k,i is selected based on Equation 11, by choosing the pairing s _∗,i+1 with minimal costs (starting with i = 0 and k = 1). If, for example, s _1,0 is already assigned to s _1,1, a successor to s _2,0 is selected by $s_{2,1}^{\ast }=\text{arg}\underset{s_{2,1}\in S_1}{\text{min}}\alpha {\sum }_{i=1}^{N_1}{f}_E(S_1)+\beta {\sum }_{i=1}^{N_1}{f}_C(S_1,H_1)+\gamma {\sum }_{i=1}^{N_1}{f}_K(S_0,S_1)$. This is successively done for all pairings and all configurations until every pairing in every configuration has a successor.

2.2.4 Complexity of the algorithms

The complexity of the GCS search space and thus the memory storage is $\mathcal{O}(k^{\mathit{\text{NT}}})$ in theory (N is the number of targets, T is the number of time steps, and k ≤ N denotes ambiguities after cardinality reduction), since there are k^N possible configurations between two views and each of these configurations at t can be combined with all configurations at (t + 1). Optimization is only necessary for ambiguous clusters ${\mathcal{C}}_{(t-\delta :t),\nu }$, therefore N = ν specifies the number of flies in this cluster and T = δ + 1 specifies the length of the cluster. Thus, the global optimum must be calculated based on k^NT possible cluster configurations. Given the recursive decomposition of Equation 10, the complexity of the cost calculations decreases to $\mathcal{O}(T{k}^N)$. The theoretical computational complexity is $\mathcal{O}(\mathit{\text{NTL}})$ for Gibbs sampling (where L is the number of sampling steps) and $\mathcal{O}(\mathit{\text{NT}})$ for greedy optimization.

For TbM algorithm, no extra space for tracking is needed. Thus, the memory storage complexity is $\mathcal{O}(\mathit{\text{NT}})$. The time complexity is $\mathcal{O}({(\mathit{\text{NT}})}^2)$. For the first three frames, exhausted search for triplets requires matching all detections in two views.

3 Experiments

3.1 Synthetic data

Both tracking paradigms are evaluated using synthetic data, generated by the swarm simulator introduced in [35]. The simulator generates all necessary data for tracking (i.e., rendered images and camera matrices) and evaluation (i.e., ground truth of the 2D and 3D trajectories). For our tests, three synchronized and calibrated cameras are placed around a 20 × 20 × 20 cm³ chamber. All movies are recorded with 800×800 pixel resolution and 150 fps. Since the beam width of the field of view is 45°, all cameras are placed 80 cm away from the cube’s center. Rotations around the y-axes, for cameras 1, 2, and 3, are 0°, -120°, and 120°, respectively.

According to [39], the maximum flight speed is set to 0.8 m/s. The crawling speed is reduced by the factor 0.1, and we use a Gaussian random walk for flight movement calculation [35]. To achieve more realistic conditions and to increase the probability of occlusions and nearby targets, we integrated negative geotaxis within our random walk model. Negative geotaxis describes the tendency of Drosophila to orient themselves against the earth’s gravity [40]. We integrated negative geotaxis by manipulating the randomly generated velocity in the y direction v _t  = Θv _t-1+n _t (with Gaussian noise $n_t\in \mathcal{N}(0,{\sigma }^2)$ and smoothness Θ ∈ [0, 1]). With a probability of 0.002% the y entry of n _t is forced to be zero or positive over time.

In this way, we generated several test movies with an increasing number of targets. For most real-world locomotion experiments, 50 flies per run are sufficient; thus, we generated movies with 10 to 50 targets and 1,000 frames (approximately 6 s; Sections 4.2 and 4.1). In addition, we made a long-term movie with 50 flies over 3,000 frames (Section 4.3) and high-density movies with a few hundreds of flies and time steps (Section 4.4).

To guarantee identical raw data for both algorithms, the 2D positions of all views are established by a separate blob detection routine. Resultant measurements contain time steps with several occluded flies in all views (leading to changes in $N_t^i$; compare to Table 1). We also added Gaussian noise (σ² = 0.001 in the intensity domain [0,1]) to the ground truth videos to simulate blob detections under realistic conditions. Figure 5 shows an example triplet of noisy images of 200 flies.

Table 1 Overview of the results

Figure 5

Example of a noisy image triplet generated by the swarm simulator.

3.2 Evaluation and comparison measure

Both paradigms are compared in terms of tracking accuracy using the correspondence and association errors ( E _ca) [35]. The E _ca is defined as follows:

$$E_{\text{ca}}=\frac{N_c+N_a}{T},$$

(12)

where N _c is the number of incorrect stereo matches, N _a is the number of false temporal associations, and T is the number of frames. To calculate N _a and N _c , all computed 3D trajectories are assigned to their respective ground truth paths. This assignment is used to calculate Euclidean distances between calculated positions and ground truth positions. If the distance is not within a tolerance, N _c is incremented for each frame and time step. The temporal association value is incremented if the ID of the calculated 3D paths changes between consecutive frames.

4 Results

We tested all combinations illustrated in Figure 1 as follows:

• Tracking by matching method (named TbM)

• GCS optimized via Gibbs sampling analogous to [23] (named Gibbs)

• GCS with projection consistency (PC) optimized via Greedy (named Greedy PC)

General tracking results for 50 flies and over 1,000 time steps are given in Figure 6.

Figure 6

Tracking results of 50 flies over 1,000 time steps and ground truth. The GCS image was made with projection consistency and greedy selection.

Table 1 summarizes results for all approaches. The resultant E _ca value is additionally plotted in Figure 7a.

Figure 7

Resultant E _ca and N _c and N _a values. (a) Resultant E _ca value plotted for all cases from Table 1. (b) Resultant N _c and N _a values plotted for the TbM and the Greedy PC tests.

4.1 Gibbs sampling vs. greedy optimization

The first observation is related to the number of occlusions and maximal cluster sizes. In general, the complexity of the global search space increases with the number of targets and frames [16]. If there are only a few ambiguities (e.g., occlusions, nearby 3D paths), most of the correspondences are unique and latter optimization is only necessary for a few small clusters (compare to $max|\mathcal{C}|$ in Table 1).

This can be observed in all movies besides the movie with 20 flies: the maximum cluster contains 19,024 pairings and 2,140 pairings without and with projection consistency, respectively (Table 1). Thus, the tracking time increases for both GCS approaches. However, the greedy selection is still able to calculate sufficient tracks, whereas Gibbs sampling results in less reliable results. The reason is that Gibbs needs to sample one sequences of configurations in a cluster containing almost 20,000 pairings covering 18 targets for 843 frames. Given one wrong correspondence selection prevents Gibbs from converging in the global optimum. Since we sampled for 10,000 iterations, this was not possible in reasonable time.

This coherence is also observable in the long-term and high-density experiments. In contrast to Zou et al. [23], we merge overlapping clusters for both joint time steps and joint targets (see Section 2.2.3) to guarantee a global search space. Thus, given very dense situations with hundreds of flies, the natural segmentation of the clusters is no longer available. In all measurements, the cluster size of the Gibbs approach was equivalent to the overall search space so that $|{\mathcal{C}}_{(t-\delta :t),\nu }|=|{\mathcal{C}}_{T,N}|=|{\mathfrak{S}}_{\mathcal{P}}|$. The latter optimization must therefore sample one sequence of configurations out of k^ν(δ+1) (k ≤ ν and ν → N, δ → T) possible sequences (compare to Section 2.2.4) statistically. This is why Gibbs sampling requires millions of sampling steps to calculate stable results [37] which was neither shown in [23] nor possible in our data for thousands of frames in reasonable time. Since algorithms requiring more than 4 h for only a few seconds movie length are not suitable for biological applications, we neglect these tracking results in Table 1 (indicated by n/a). Thus, Gibbs sampling for high-density or long-term situations is more interesting from a theoretical point of view [24].

4.2 TbM vs. GCS

Obviously, the Greedy PC approach has the best overall performance int the general experiments. The TbM approach is between the Greedy and Gibbs solution. Optimization of GCS without PC and via Gibbs leads to the worst results with irregular E _ca values.

If the number of flies increases, the E _ca increases for both TbM and Greedy tracking (compare to Figure 7a). Since both measurements increase proportional to the number of wrong correspondences between views N _c and wrong associations over time N _a , these values are examined in Figure 7b.

As apparent, an increasing N _c value leads to high error measurements for both the TbM and the Greedy PC approach. The main reason for wrong or missing stereo correspondences is caused by occlusions. The more flies are located in the chamber, the more occlusions arise during blob detection (compare to Table 1). Especially in latter frames, occlusions arise very frequently because of the negative geotaxis (compare to Section 3.1). During the movie with 50 flies, up to 4 flies are occluded for several frames in camera 1, for example. Thus, even in situations with up to 50 flies, the target density is comparatively high. Since TbM and Greedy PC try to overcome this events using the projection consistency, both have much lower N _c than the two camera tracking solutions. Gibbs has up to 1,677 wrong stereo correspondences (data not shown). Therefore, it is not able to calculate the global optimum even after 10,000 sampling steps.

In other words, the overall search space ${\mathfrak{S}}_{\mathcal{P}}^{\ast }$, containing more than 1,000 occlusions, cannot be sampled sufficiently because of the growth of the clusters (Table 1). However, Greedy PC benefits from the previously calculated overall search space ${\mathfrak{S}}_{\mathcal{P}}^{\ast }$: since all possible pairings and sequences of configurations are used for coast calculations, ambiguities caused by occlusions can be corrected more frequently.

4.3 Long-term tracking

In the long-term experiment, 50 flies were tracked for 3,000 frames. Gibbs failed in this experiments because the size of the clusters increases drastically for 3,000 frames. Thus, only TbM and Greedy PC were able to track during this experiment.

In long-term movies, the TbM approach can achieve better results than the Greedy PC algorithm (Table 1). The reason for this inversion compared to the 1,000 frame results is that the size of the clusters $|\mathcal{C}|$ increases to much for 3,000 frames. Thus, the probability of getting a local optimum via greedy selection increases accordingly. However, tracking accuracy is still convenient in the Greedy PC approach. On the other hand, TbM, as a causal method, is not affected by the length of tracking sequences.

In contrast to the TbM approach, GCS can miss targets during optimization (see Table 1). However, projection consistency reduces the amount of missed targets. Furthermore, GCS optimization leads to less fragmented trajectories than the TbM approach. Whereas TbM results in 137 trajectories (N _a  = 99) for 50 flies (over 3,000 frames), the Greedy PC approach calculates 49 complete tracks of 50 tracks in total (N _a  = 7). If complete trajectories are required (i.e., identity of the flies must remain over time), Greedy PC is recommended but with the possibility of loosing flies.

4.4 High-density tracking

To evaluate the behavior of both tracking approaches, we tracked up to 200 flies. Similar to the long-term experiment in Section 4.3, we limit our comparisons to Greedy PC and TbM.

Table 1 highlights the measurements for 100, 150, and 200 targets. We decreased the number of frames for the movies with 150 and 200 flies to limit the size of ${\mathfrak{S}}_{\mathcal{P}}^{\ast }$. For up to 100 targets and 200 frames, Greedy correspondence selection is more accurate than TbM. However, given more than 100 targets, resulting in very high fly densities, the TbM outperforms the GCS approach. Most importantly, there are no missing targets in the probabilistic approach, whereas Greedy was not able to find trajectories for all flies. Furthermore, TbM can achieve better overall accuracy in high-density situations in less tracking time. The only drawback of the probabilistic tracking is again the fragmentation of the trajectories: TbM calculates more tracks than Greedy PC resulting in many identity changes.

5 Conclusion

In this paper, we discussed two tracking paradigms for identical appearing objects such as Drosophila melanogaster in 3D. One paradigm is based on a probabilistic approach conducting tracking and matching alternatively [35]. The other paradigm constructs a global search space over all targets and time steps, which is optimized in a second step [23].

Due to the high complexity of the second GCS paradigm, we introduced two improvements, namely projection consistency and greedy optimization. Especially, the projection consistency is able to reduce the overall complexity and thus improve the tracking results without yielding into local optima. Since Gibbs sampling, used for GCS optimization in [23], needs thousands of iterations to guarantee good results, our greedy selection scheme outperforms Gibbs sampling. However, a global result cannot be guaranteed via greedy optimization.

We demonstrated several advantages and disadvantages of both the TbM and GCS approach. Thus, the decision which approach to use must be done carefully. If the identity of the flies is not important, TbM can be used to track for several thousands of frames (compare to Section 4.3). All flies were detected, but the trajectories were fragmented due to occlusions.

The GCS approach was not able to track all flies in all experiments: only 49 of 50 flies were detected. On the other hand, the trajectories of the detected flies were less fragmented (compare to Section 4.2). In addition, the GCS was able to solve collisions and occlusions more frequently because of the global search space. This leads to the higher tracking accuracy illustrated in Figure 7a. If dozens of flies must be tracked for a comparatively short period, GCS outperforms TbM tracking. For very long sequences, it is the other way around.

If high fly densities are needed for a comparatively long period, the size of the global search space prevents Gibbs optimization, because it requires too many sampling iterations. In addition, greedy tracking quality decreases drastically compared to TbM (see Section 4.4). Thus, without further reductions of the global search space, probabilistic tracking is the preferable paradigm in high-density experiments.

TbM could be optimized in terms of tracking accuracy, whereas GCS could be optimized for longer tracking durations and higher target densities. Possible improvements for the TbM approach are discussed in [35]. Here, we want to focus on improvements of the GCS approach.

Currently, we use the third camera only to correct mismatches between cameras 1 and 2. The optimization scheme is still executed on pairings. Since pairwise comparison in a triplet would further reduce the search space, all optimization steps could be done on three image points.

The kinetic model given by Equation 8 is also a current drawback of the GCS approach. Only motion form (t - 1) is considered for time step t. Thus, a more appropriate motion model would further improve the accuracy of GCS tracking.

Currently, we are developing a three-camera real-world setup to capture movies of adult Drosophila flies. Thus, we are going to test both algorithms on real video sequences, comparable to the synthetic data introduced above.