Quantitative comparison of motion history image variants for video-based depression assessment

2.1 Audio/visual emotion challenge

Although decision support systems based on visual cues have not been incorporated into standard clinical practice, several approaches for developing such systems can be found in the literature. The majority of published approaches materialized within the AVEC “Depression Recognition Sub-challenge” (DSC) [9–11].

2.2 Motion history image

MHI is a robust, yet relatively straightforward, algorithm developed to represent the motion that occurs in the course of a complete video recording with a single image [18]. The algorithm produces a grayscale image, in which the white pixels correspond to the most recent movements and the darkest gray correspond to the earliest motion elements. Black pixels indicate absence of movement. It is a popular algorithm for motion analysis [18] and has been extensively used in the field of human action recognition [8].

An early approach of MHI to facial image analysis was that of Valstar et al. [19], who employed MHI in facial action recognition from videos. Meng et al. [20] published a continuous depression assessment approach in their participation to the DSC of AVEC’13; they proposed an extension of MHI, the Motion History Histograms (MHH), which considers patterns of movement. In the DSC of AVEC’14, Pérez Espinoza et al. [21] employed MHI, and for the same challenge, Jan et al. [22] proposed the 1-D MHH, an extension of MHI, which is computed on the feature vector sequence instead of the intensity image. As part of their DSC-AVEC’16 participation, Pampouchidou et al. [23] introduced Landmark Motion History Images (LMHI), which instead of considering intensities from image sequences, considers sequences of facial landmarks.

2.3 Gabor filtering and inhibition

Gabor filters have been frequently used in both facial expression analysis and emotion recognition [24, 25]. In relevant approaches, the feature vector is extracted from the convolution of the original image with a 2D Gabor wavelet function at different orientations and wavelengths. This describes the spatial frequency structure around each pixel. In [26], the Gabor energy was used for facial emotion recognition, which gives a smoother response to an edge or a line of appropriate width with a local maximum exactly at the edge or in the center of the line. The authors also applied background texture suppression on the response of the filter, by removing an image filtered by the difference of Gaussians (DoG) from the original response for each orientation. This approach, also known as anisotropic inhibition [27], removes noise and provides a sharper representation of facial features.

2.4 Deep learning

Deep learning, which has become increasingly popular during recent years, is a self-learning tool designed to identify patterns in several sets of data samples, extracted from multiple processing layers. Each layer is composed of representation learning methods and is processed in a higher and more abstract level [28]. Convolutional Neural Network (CNN) is a particular deep feedforward network with higher generalization efficiency than other fully connected networks. There are typically two types of layers: the convolutional (conv.) layer and the pooling layer. In the conv. layer, all units are arranged in a feature map and connected to the weights (also known as filter banks), while the weighted sum is inserted to the Rectified Linear Unit (ReLU). The CNN architecture used in the present work was employed in the participation that won the 2012 ImageNet competition (ILSVRC) [29, 30].

Deep learning approaches have been widely tested in the field of facial expression recognition [31]. In the field of depression assessment from visual cues, however, we could find only two published reports. Dibeklioğlu et al. [32] employed Stacked Denoising Autoencoders in a multimodal context to perform video classification according to three levels of depressive symptomatology on the Pittsburgh dataset. Moreover, Zhu et al. [33] employed Deep Convolutional Neural Networks to achieve the highest performance among the unimodal (visual) approaches addressing the aim of AVEC’13 and AVEC’14 competitions.

3.1 Preprocessing

Meaningful processing of video frames requires, first, extraction of the region of interest (face). Detection of 2D facial landmarks (c.f. Fig. 1) and extraction of aligned facial images, of size 112 ×112 pixels, were accomplished using OpenFace, an open source application [13]. A binary “success” score is provided for each frame, with “0” and “1” indicating unsuccessful and successful detection, respectively. In the present work, only successfully detected frames were retained for further processing.

3.2 Motion representation

It is well supported in clinical literature that most of the non-verbal signs of depression are dynamic by nature [6, 7]. Therefore, the use of video-based methods (dynamic), as opposed to frame-based (static), is preferable. In the proposed work, three different motion history images were implemented: (a) the Motion History Image (MHI) as derived from the basic algorithm, (b) the Landmark Motion History Image (LMHI) which relies on facial landmarks, and (c) the Gabor Motion History Image (GMHI). More details regarding the specific motion representation algorithms are presented below with implementation examples illustrated in Fig. 4.

3.2.3 Gabor Motion History Image

GMHI is another variant of MHI, where Gabor-inhibited images substitute original image intensities. The motivation for implementing this variant is that it focuses on the important details of the facial features and thus extracts the most relevant information. The motion representation algorithm is identical to the one described in Section 3.2.1, but the input image I is the result of the Gabor inhibition. The process of obtaining the Gabor-inhibited image is explained in detail below.

The Gabor wavelet at position (x,y) is given by:

$$ {} \Psi_{\lambda,\theta,\phi,\sigma,\gamma}(x,y)= exp\left(-\frac{x^{'2}+\gamma^{2}y^{'2}}{2\sigma^{2}}\right)\cos\left(2\pi\frac{x^{'}}{\lambda}+\phi\right) $$

(5)

with

$$ \begin{aligned} x^{'} = x\; \cos\;\theta + y\; \sin\;\theta \\ y^{'} = -x\; \sin\;\theta + y\; \cos\;\theta \end{aligned} $$

(6)

where λ stands for the wavelength, θ for the orientation, ϕ for the phase offset, σ for the standard deviation of the Gaussian, and γ for the spatial aspect ratio [34].

The input image is usually filtered with many wavelets for multiple orientations and wavelengths. The energy filter response is obtained by combining the convolutions obtained from two different phase offsets (ϕ ₀=0 and ϕ ₁=π/2) using the L2-norm. Background texture suppression is applied on the filter response, by removing a DoG-filtered image from the original response for each orientation [27]. Finally, the mean response of Gabor filtering is used to combine the responses across the different orientations, resulting in the pseudo-image used to compute the GMHI. An example of applying the common Gabor and the Gabor-inhibited algorithms to an aligned face image is illustrated in Fig. 3, where the Gabor-inhibited image appears to be sharper and with less texture in uniform regions than the original Gabor response.

3.3 Feature extraction

Feature extraction was implemented in the present work using two alternative approaches. The first employs appearance-based descriptors, popular in facial image analysis, while the second presents a preliminary attempt to address the problem based on deep learning methods. In both cases, the features were extracted from motion images, instead of the original video recordings.

3.3.2 Visual Graphic Geometry

Visual Graphic Geometry (VGG) is a CNN variant proposed by Simonyan and Zisserman [38]. Using VGG, they achieved 92.7% top-5 test accuracy on the ImageNet Dataset, which comprises of over 14 million images in 1000 classes. The microarchitecture of VGG16 can be seen in Fig. 5.

The RGB image, with pixel values ranging between 0 and 255, is normalized by subtracting the mean pixel value. The input to VGG (a fixed-size 224 ×224 RGB image) passes through a stack of conv. layers, where the very small filters are of receptive field size 3 ×3 to capture the notion of left/right, up/down, and center. The convolution stride is fixed to 1 pixel; the spatial padding of a conv. layer input is such that the spatial resolution is preserved after convolution, i.e., the padding is 1 pixel for 3 ×3 conv. layers. Spatial pooling is carried out by five max-pooling layers, which follow some of the conv. layers (not all the conv. layers are followed by max-pooling). Max-pooling is performed over a 2 ×2 pixel window, with stride 2. A stack of conv. layers (which has a different depth in different architectures) is followed by three fully connected (FC) layers: the first two have 4096 channels each and the third performs 1000-way ILSVRC classification and thus contains 1000 channels (one for each class). The final layer is the soft-max layer. The configuration of the fully connected layers is the same in all networks. All hidden layers are characterized by non-linearity afforded by ReLU [30].

In the present work, a pre-trained VGG16 network was employed at different VGG layers, for each motion history image separately as well as combined in the form of an RGB image. The proposed method involves transfer learning, by employing the pre-trained VGG. It is applied on the motion history images in order to extract features before the 1st fully connected layer, as well as after the 1st, 2nd, and 3rd fully connected layers. Specifically, the D version of the network was chosen, as it has shown excellent results in related medical applications. The extracted features are subsequently used for classification purposes in the exact same manner as the appearance-based descriptors. The different implementations are explained in what follows.

Before fully connected layer (BFCL) provides the features to the fully connected layer of the VGG16, as shown in Fig. 5. Filter size is 14 ×14 with 512 kernels. Mean and max values are calculated from each filter, each resulting in a matrix of size 1 ×512 for each image.

After fully connected layer (AFCL) is the second approach. There are three fully connected layers in VGG16. Layers 1 and 2 operate on a feature matrix of size 1 ×4096, and layer 3 on a feature matrix of size 1 ×1000.

3.4 Dimensionality reduction and classification

In the present work, principal component analysis (PCA) was employed to achieve dimensionality reduction. PCA is one of the most popular methods for this purpose and is based on the linear transformation of the original feature vector, into a set of uncorrelated principal components. For a dataset of size N×M (i.e., N samples and M features), PCA identifies a M×M coefficient matrix (component loadings) that maps each data vector from the original space to a new space of M principal components. However, by properly selecting a smaller set of K<M components, the dimensionality of the data can be reduced while still retaining much of the information (i.e., variance) in the original dataset.

In the present work, classification was based on a supervised learning model using Support Vector Machines (SVMs). SVM is a non-probabilistic binary linear classifier which, based on the training samples, attempts to identify an optimal hyperplane that maximizes the distance between the two classes.

4.1 Configuration of parameters

The analysis pipeline proposed here was tested on the benchmark dataset provided by AVEC’14. As mentioned in Section 2, AVEC’14 included data from two tasks: FreeForm and NorthWind and the dataset was partitioned into three subsets: training, development, and test. The BDI scores for each video-clip were provided only for the training and development sets, comprising 200 labeled recordings in total. In order to address the categorical depression assessment, a cutoff of 13/14 points on the BDI-II was set as in [14]. After applying the cutoff, the dataset was fairly balanced consisting of 96 individuals who scored high and 104 persons who scored low on BDI-II, henceforth labeled as “depressed” and “non-depressed” groups.

Regarding specific parameters of the motion representation algorithms, the value of ξ was set to 25 for MHI and to 8 for GMHI. These thresholds were chosen empirically, so that the static background did not present movement in the motion image. This effect was noted at lower ξ values, where differences in pixel intensity were attributed to illumination variations. Setting the appropriate threshold ensures that the movement represented by the motion images is meaningful and can be attributed solely to movements. LBP was tested with two sets of [radius, neighborhood], namely [1,8] and [2,16]. The Gaussian kernel was chosen for the SVM classifier, with the expected proportion of outliers in the training data set to 10%. The log(x/(1−x)) transform function was applied. The number of principal components retained was selected empirically: k=100 for the appearance-based descriptors and k=60 for VGG. A variant of Leave-One-Out (LOO), the Leave-One-Subject-Out (LOSO), was used for cross-validation. LOSO was selected instead of LOO as a non-biased and person-independent method, given that the dataset contained more than one recordings of the same subject (ranging from 2 to 6).

4.2 Performance evaluation

Additional performance metrics for the best performing model are reported in Table 4, whereas Table 5 compares the present findings to previously published results using similar datasets. It should be noted that the results presented by Senoussaoui et al. [14] and Alghowinem et al. [15] were obtained using different organizations of the AVEC datasets; thus, a direct comparison with the present results is not possible. In the cross-corpus approach of Alghowinem et al. [15] the used set was carefully selected from the original dataset (AVEC’13) in terms of the total number and duration of recordings per participant, in order to match the other two datasets. On the other hand, in [14], the algorithm was applied to the training dataset provided by the challenge organizers (AVEC’14) and tested on the development dataset. Although the AVEC dataset has been widely employed in approaches for continuous depression assessment, the aforementioned approaches, to the best of the authors’ knowledge, are the only ones attempting categorical depression assessment on the specific dataset.