Thermal spatio-temporal data for stress recognition

Abstract

Stress is a serious concern facing our world today, motivating the development of a better objective understanding through the use of non-intrusive means for stress recognition by reducing restrictions to natural human behavior. As an initial step in computer vision-based stress detection, this paper proposes a temporal thermal spectrum (TS) and visible spectrum (VS) video database ANUStressDB - a major contribution to stress research. The database contains videos of 35 subjects watching stressed and not-stressed film clips validated by the subjects. We present the experiment and the process conducted to acquire videos of subjects' faces while they watched the films for the ANUStressDB. Further, a baseline model based on computing local binary patterns on three orthogonal planes (LBP-TOP) descriptor on VS and TS videos for stress detection is presented. A LBP-TOP-inspired descriptor was used to capture dynamic thermal patterns in histograms (HDTP) which exploited spatio-temporal characteristics in TS videos. Support vector machines were used for our stress detection model. A genetic algorithm was used to select salient facial block divisions for stress classification and to determine whether certain regions of the face of subjects showed better stress patterns. Results showed that a fusion of facial patterns from VS and TS videos produced statistically significantly better stress recognition rates than patterns from VS or TS videos used in isolation. Moreover, the genetic algorithm selection method led to statistically significantly better stress detection rates than classifiers that used all the facial block divisions. In addition, the best stress recognition rate was obtained from HDTP features fused with LBP-TOP features for TS and VS videos using a hybrid of a genetic algorithm and a support vector machine stress detection model. The model produced an accuracy of 86%.

1 Introduction

Stress is a part of everyday life, and it has been widely accepted that stress, which leads to less favorable states (such as anxiety, fear, or anger), is a growing concern to a person's health and well-being, functioning, social interaction, and financial aspects. The term stress was coined by Hans Selye, which he defined as ‘the non-specific response of the body to any demand for change’ [1]. Stress is a natural alarm, resistance, and exhaustion system [2] for the body to prepare for a fight or flight response to either defend or make the body adjust to threats and changes. The body shows stress through symptoms such as frustration, anger, agitation, preoccupation, fear, anxiety, and tenseness [3]. When chronic and left untreated, stress can lead to incurable illnesses (e.g., cardiovascular diseases [4], diabetes [5], and cancer [6]), relationship deterioration [7, 8], and high economic costs, especially in developed countries [9, 10]. It is important to recognize stress early to diminish the risks. Stress research is beneficial to our society with a range of benefits, motivating interest and posing technical challenges in computer science in general and affective computing in particular.

Various computational techniques have been used to objectively recognize stress using models based on techniques such as Bayesian networks [11], decision trees [12], support vector machines [13], and artificial neural networks [14]. These techniques have used a range of physiological (e.g., heart activity [15, 16], brain activity [17, 18], galvanic skin response [19], and skin temperature [12, 20]) and physical (e.g., eye gaze [11], facial information [21]) measures for stress as inputs. Physiological signal acquisition requires sensors to be in contact with a person, and this can be obtrusive [3]. In addition, the physiological sensors are usually required to be placed on specific locations of the body, and sensor calibration time is usually required as well, e.g., approximately 5 min is needed for the isotonic gel to settle before galvanic skin response readings can be taken satisfactorily using the BIOPAC System [22]. The trend in this area of research is leading towards obtaining symptom of stress measures through less or non-intrusive methods. This paper proposes a stress recognition method using facial imaging and does not require body contact with sensors unlike the usual physiological sensors.

A relatively new area of research is recognition of stress using facial data in the thermal (TS) and visible (VS) spectrums. Blood flow through superficial blood vessels, which are situated under the skin and above the bone and muscle layer of the human body, allows TS images to be captured. It has been reported in the literature that stress can be successfully detected from thermal imaging [23] due to changes in skin temperature under stress. In addition, facial expressions have been analyzed [24] and classified [25–27] using TS imaging. Commonly, VS imaging has been used for modeling facial expressions, and associated robust facial recognition techniques have been developed [28–30]. However, from our understanding, the literature has not developed computational models for stress recognition using both TS and VS imaging together as yet. This paper addresses the gap and presents a robust method to use information from temporal and texture characteristics of facial regions for stress recognition.

Automatic facial expression analysis is a long researched problem. Techniques have been developed for analyzing the temporal dynamics of facial muscle movements. A detailed survey of facial expression recognition methods can be found in [31]. Further, vision-based facial dynamics have been used for affective computing tasks such as pain monitoring [32] and depression analysis [30]. This motivated us to explore vision-based stress analysis where inspiration can be taken from the vast field of facial expression analysis. Descriptors such as the local binary pattern (LBP) have been developed for texture analysis and have been successfully applied to facial expression analysis [25, 33, 34], depression analysis [30], and face recognition [35]. A particular LBP extension for analysis of temporal data - local binary patterns on three orthogonal planes (LBP-TOP) - has gained attention and is suitable for the work in this study. LBP-TOP provides features that incorporate appearance and motion, and is robust to illumination variations and image transformations [25]. This paper presents an application of LBP-TOP to TS and VS videos.

Various facial dynamics databases have been proposed in the literature. For facial expression analysis, one of the most popular databases is the Cohn-Kanade + [32], which contains facial action coding system (FACS) and generic expression labels. Subjects were asked to pose and display various expressions. There are other databases in the literature which are spontaneous or close to spontaneous, such as RU-FACS [36], Belfast [37], VAM [38], and AFEW [39]. However, these are limited to emotion-related labels which do not serve the problem in the paper, i.e., stress classification. Lucey et al. [32] proposed the UNBC McMasters database comprising video clips where patients were asked to move the arm up and their reaction was recorded. For creating ANUStressDB, subjects were shown stressful and non-stressful video clips. This database is similar to that in [32].

There are various forms of stressors, i.e., demands or stimuli that cause stress [23, 40–42] validated by self-reports (e.g., self-assessment [43, 44]) and observer reports (e.g., human behavior coder [42]). Some examples of stressors are playing video (action) games [45, 46], solving difficult mathematical/logical problems [47], and listening to energetic music [45]. Among these stressors are films, which were used to stimulate stress in this work. In this work, we develop a computed stress measure[3] using facial imaging in VS and TS. Our work analyzes dynamic facial expressions that are as natural as possible elicited by a typical stressful, tense, or fearful environment from film clips. Unlike the previous work in the literature that uses posed facial expressions for classification [48], the work presented in this paper provides an investigation of spontaneous facial expressions as responses or reactions to environments portrayed by the films.

This paper describes a method for collecting and computationally analyzing data for stress recognition from TS and VS videos. A stress database (ANUStressDB) of videos of faces is presented. An experiment was conducted to collect the data where experiment participants watched stressful and non-stressful film clips. ANUStressDB contains videos of 35 subjects watching film clips that created stressed and not-stressed environments validated by the person. Facial expressions in the videos were stimulated by the film clips. Spatio-temporal features were extracted from the TS and VS videos, and these features were provided as inputs to a support vector machine (SVM) classifier to recognize stress patterns. A hybrid of a genetic algorithm (GA) and SVM was used to select salient divisions of facial block regions and determine whether using the block regions improved the stress recognition rate. The paper compares the quality of the stress classifications produced from using LBP-TOP and HDTP (our thermal spatio-temporal descriptor) features from TS and VS data with and without using facial block selection.

The organization of the paper is as follows: Section 2 presents the experiment for TS, VS, and self-reported data collection. Section 3 describes the facial imaging processing steps for the TS and VS data. The new thermal spatio-temporal descriptor, HDTP, is proposed in Section 4. Stress classification models are described in Section 5. Section 6 presents the results, an analysis of the results, and suggestions for future work.

2 Data collection from the film experiment

After receiving approval from the Australian National University Human Research Ethics Committee, an experiment was conducted to collect TS and VS videos of faces of individuals while they watched films. Thirty-five graduate students consisting of 22 males and 13 females between the ages of 23 and 39 years old volunteered to be experiment participants. Each participant had to understand the experiment requirements from written experiment instructions with the guidance of an experiment instructor before they filled in the consent form. The participant was provided information about the experiment and its purpose from a script to ensure that there was consistency in the experiment information provided across all participants. After providing consent, the participant was seated in front of a LCD display (placed between two speakers). The distance between the screen and subject was in the range between 70 and 90 cm. The instructor started the films, which triggered a blank screen with a countdown of the numbers 3, 2, and 1 transitioning in and out slowly with one before the other. The reason for the countdown display and the blank screen was for participants to move away from their thoughts at the time and get ready to pay attention to the films that were about to start. This approach was like that used in experiments for similar work in [49]. Subsequent to the countdown display, a blank screen was shown for 15 s, which was followed by a sequence of film clips with 5-s blank screens in between. After watching the films, the participant was asked to do a survey, which related to the films they watched and provided validation for the film labels. The experiment took approximately 45 min for each participant. An outline of the process of the experiment for an experiment participant is shown in Figure 1.

Figure 1

An outline of the process followed by each experiment participant in the film experiment.

Participants watched two types of films either labeled as stressed or not-stressed. Stressed films had stressful content (e.g., suspense with jumpy music), whereas not-stressed films created illusions of meditative environments (e.g., swans and ducks paddling in a lake) and had content that was not stressful or at least was relatively less stressful compared with films labeled as stressed. There were six film clips for each type of film. The survey done by experiment participants validated the film labels. The survey asked participants to rate the films they watched in terms of levels of stress portrayed by the film and the degree of tension and relaxation they felt. Participants found the films that were labeled stressed as stressful and films labeled not-stressed as not stressful with a statistical significance of p < 0.001 according to the Wilcoxon test.

While the participants watched the film clips, TS and VS videos of their faces were recorded. A schematic diagram of the experiment setup is shown in Figure 2. TS videos were captured using a FLIR infrared camera (model number SC620, FLIR Systems, Inc. Notting Hill, Australia), and VS videos were recorded using a Microsoft webcam (Microsoft Corporation, Redmond, WA, USA). Both the videos were recorded with a sampling rate of 30 Hz, and the frame width and height were 640 and 480 pixels, respectively. Each participant had a TS and VS video for each film they watched. As a consequence, a participant had 12 video clips made up of six stressed videos and six not-stressed videos. We name the database that has the collected labeled video data and its protocols as the ANU Stress database (ANUStressDB).

Figure 2

Setup for the film experiment to obtain facial video data in thermal and visible spectrums.

Note the usage of the terms film and video in this paper. We use the term film to refer to a video portraying entertaining content, colloquially called a ‘film’ or ‘movie’, which a participant watched during the experiment. We use the term video to refer to a visual recording of a participant's face and its movement during the time period while they watched a film. Thus in this paper, a film is something which is watched, while a video is something recorded about the watcher.

3 Face pre-processing pipeline

Facial regions in VS videos were detected using the Viola-Jones face detector. However, facial regions could not be recognized satisfactorily using the Viola-Jones algorithm in thermal spectrum (TS) videos, so a face detection method based on eye coordinates [50, 51] and a template matching algorithm was used. A template of a facial region was developed from the first frame of a TS video. The facial region was extracted using the Pretty Helpful Development Functions toolbox for Face Recognition [50–52], which calculated the intraocular displacement to detect a facial region in an image. This facial region formed a template for facial regions in each video frame of the TS videos, which were extracted using MATLAB's Template Matcher system [53]. The Template Matcher was set to search the minimum difference pixel by pixel to find the area of the frame that best matched the template. Examples of facial regions that were detected in the VS and TS videos for a participant are presented in Figure 3.

Figure 3

Examples of facial regions extracted from the ANUStressDB database. The facial regions are of an experiment participant watching the different types of film clips. (a) The participant was watching a not-stressed film clip. (b) The participant was watching a stressed film clip. (i) A frame in the visual spectrum. (ii) The corresponding frame in the thermal spectrum. The crosshairs in the thermal frame were added by the recording software and represents the camera auto-focus.

Facial regions were extracted from each frame of a VS video and its corresponding TS video. Grouped and arranged in order of time of appearance in a video, the facial regions formed volumes of the facial region frames. Examples of facial blocks in TS and VS are shown in Figure 4.

Figure 4

Examples of facial volumes extracted from the ANUStressDB database. The facial volumes are of an experiment participant watching the different types of film clips. (a) The participant was watching a not-stressed film clip. (b) The participant was watching a stressed film clip. (i) A facial volume in the visual spectrum. (ii) The corresponding facial volume in the thermal spectrum.

4 Spatio-temporal features

There are claims in the literature that features from segmented image blocks of a facial image region can provide more information than features directly extracted from an image of a full facial region in VS [25]. Examples of full facial regions are shown in Figure 4, and blocks of a full facial region are presented in Figure 5. To illustrate the claim, features from each of the blocks used in conjunction with features from the other blocks in Figure 5 (i) can offer more information than features obtained from Figure 4a (i). The claim aligns with the results from classifying stress based on facial thermal characteristics [23]. As a consequence, the facial regions in this work were segmented into a grid of 3 × 3 blocks for each video segment, or facial volume, forming 3 × 3 blocks. A block has X, Y, and T components where X, Y, and T represent the width, height, and time components of an image sequence, respectively. Each block represented a division of a full facial block region or facial volume. LBP-TOP features were calculated for each block.

Figure 5

The facial region in Figure4a segmented into 3 × 3 blocks. (i) Blocks of the frame in the visual spectrum. (ii) Blocks of the corresponding frame in the thermal spectrum.

LBP-TOP is the temporal variant of local binary patterns (LBP). In LBP-TOP, LBP is applied to three planes - XY, XT, and YT - to describe the appearance of an image, horizontal motion, and vertical motion, respectively. For a center pixel O _p of an orthogonal plane O and its neighboring pixels N _i , a decimal value is assigned to it:

$$\mathit{d}={\displaystyle \sum _{\mathit{O}}^{\mathit{XY},\mathit{XT},\mathit{YT}}{\displaystyle \sum _{\mathit{p}}{\displaystyle \sum _{\mathit{i}=1}^{\mathit{k}}{2}^{\mathit{i}-1}\mathit{I}\left({\mathit{O}}_{\mathit{p}},{\mathit{N}}_{\mathit{i}}\right)}}}$$

(1)

According to a study that investigated facial expression recognition using LBP-TOP features, VS and near-infrared images produced similar facial expression recognition rates, provided that VS images had strong illumination [33]. Due to the fact that TS videos are defined by colors and different color variations, LBP-TOP features may not be able to fully exploit thermal information provided in TS videos and in particular capture thermal patterns for stress. In addition, LBP-TOP features have been mainly extracted from image sequences of people told to show some facial expression, which is not like the image sequences obtained from our film experiment. In our film experiment, participants watched films and involuntary facial expressions were captured. The recordings may have more subtle facial expressions of the kind of facial expressions analyzed in the literature using LBP-TOP. With the subtleness in facial movement, it is possible that LBP-TOP may not be able to offer as much information for stress analysis. These points motivate the development of a new set of features that exploits thermal patterns in TS videos for stress recognition. We propose a new type of feature for TS videos that captures dynamic thermal patterns in histograms (HDTP). This feature makes use of thermal data in each frame of a TS video of a face over the course of the video.

4.1 Histogram of dynamic thermal patterns

HDTP captures normalized dynamic thermal patterns, which enables individual-independent stress analysis. Some people may be more tolerant to some stressors than others [54, 55]. This could mean that some people may show higher degree responses to stress than others. Additionally in general, the baseline for human response can vary from person to person. To consider these characteristics in features used for individual-independent stress analysis, ways have been developed to normalize data for each participant for their type of data [42]. HDTP is defined in terms of a participant's overall thermal state to minimize individual bias in stress analysis.

A HDTP feature is calculated for each facial block region. Firstly, a statistic (consider the standard deviation) is calculated for each facial region frame for a participant for a particular block (e.g., facial block region situated at the top right corner of the facial region in the XY plane) for all the videos. The statistic values from all these frames are partitioned to define empty bins. A bin has a continuous value range with a location defined from the statistic values. The bins are used to partition statistic values for each facial block region where the value for each bin is the frequency of statistic values in the block that falls within the bounds of the bin range. Consequently, a histogram for each block can be formed from the frequencies. An algorithm presenting the approach for developing histograms of dynamic thermal patterns in thermal videos for a participant who has a set of facial videos is provided in Figure 6.

Figure 6

The HDTP algorithm captures dynamic thermal patterns in histograms from thermal image sequences.

As an illustration, consider that the statistic used is the standard deviation and the facial block region for which we want to develop a histogram is situated at the top right corner of the facial region in the XY plane (FBR₁) for video V₁ when a participant P _i was watching film F₁. In order to create a histogram, the bin locations and sizes need to be calculated. To do this, the standard deviation needs to be calculated for all frames in FBR₁ in all videos (V_1-12) for P _i . This will give standard deviation values from which the global minimum and maximum can be obtained and used to calculate the bin location and sizes. Then, the histogram for FBR₁, for V₁, and for P _i is calculated by filling the bins with the standard deviation values for each frame in FBR₁. This method then provides normalized features that also take into account the image and motion, and can be used as inputs to a classifier.

5 Stress classification system using a hybrid of a support vector machine and a genetic algorithm

SVMs have been widely used in the literature to model classification problems including facial expression recognition [27, 33, 34]. Provided a set of training samples, a SVM transforms the data samples using a nonlinear mapping to a higher dimension with the aim to determine a hyperplane that partitions data by class or labels. A hyperplane is chosen based on support vectors, which are training data samples that define maximum margins from the support vectors to the hyperplane to form the best decision boundary.

It has been reported in the literature that thermal patterns for certain regions of a face provide more information for stress than other regions [23]. The performance of the stress classifier can degrade if irrelevant features are provided as inputs. As a consequence and due to its benefits noted in literature, the classification system was extended to include a feature selection component, which used a GA to select facial block regions appropriate for the stress classification. GAs are inspired by biological evolution and the concept of survival of the fittest. A GA is a global search technique and has been shown to be useful for optimization problems and problems concerning optimal feature selection for classification [56].

The GA evolves a population of candidate solutions, represented by chromosomes, using crossover, mutation, and selection operations in search for a better quality population based on some fitness measure. Crossover and mutation operations are applied to chromosomes to achieve diversity in the population and reduce the risk of the search being stuck with a local optimal population. After each generation during the search, the GA selects chromosomes, probabilistically mostly made up of better quality chromosomes, for the population in the next generation to direct the search to more favorable chromosomes.

Given a population of subsets of facial block regions with corresponding features, a GA was defined to evolve sets of blocks by applying crossover and mutation operations, and selecting block sets during each iteration of the search to determine sets of blocks that produce better quality SVM classifications. Each block set was represented by a binary fixed-length chromosome where an index or locus symbolized a facial block region; its value or allele depicted whether or not the block was used in the classification and the length of the chromosome matched the number of blocks for a video. The search space had 3 × 3 blocks (as shown in Figure 5) with an addition of blocks that overlapped each other by 50%. The architecture for the GA-SVM classification system is shown in Figure 7. The characteristics of the GA implemented for facial block region selection is provided in Table 1.

Figure 7

The architecture of the GA-SVM hybrid stress classification system.

Table 1 GA implementation settings for facial block region selection

In summary, various stress classification systems using a SVM were developed which differed in terms of the following input characteristics:

• VS_LBP-TOP: LBP-TOP features for VS videos

• TS_LBP-TOP: LBP-TOP features for TS videos

• TS_HDTP: HDTP features (as described in Section 4.1) for TS videos

• VS_LBP-TOP + TS_LBP-TOP: VS_LBP-TOP and TS_LBP-TOP

• VS_LBP-TOP + TS_HDTP: VS_LBP-TOP and TS_HDTP

• TS_LBP-TOP + TS_HDTP: TS_LBP-TOP and TS_HDTP

• VS_LBP-TOP + TS_LBP-TOP + TS_HDTP

These inputs were also provided as inputs to the GA-SVM classification systems to determine whether the system produced better stress recognition rates.

6 Results and discussion

Each of the different features is derived from VS and TS facial videos using LBP-TOP and HDTP facial descriptors on standardized data and provided as inputs to a SVM for stress classification. Facial videos of participants watching stressed films were assigned to the stressed class, and videos associated with not-stressed films were assigned to the not-stressed class. Furthermore, their corresponding features were assigned to corresponding classes. Recognition rates and F-scores for the classifications were obtained using 10-fold cross-validation for each type of input. The results are shown in Figure 8.

Figure 8

Performance measures for SVM and GA-SVM stress recognition systems. The measures were obtained for various input features based on 10-fold cross-validation. The labels on the horizontal axes are shortened to improve readability. L and H stand for LBP-TOP and HDTP, respectively. (a) Recognition rate measure for the stress recognition systems. (b) F-score measure for the stress recognition systems.

Results show that when HDTP features for TS videos (TS_HDTP) were provided as input to the SVM classifier, there were improvements in the stress recognition measures. The best recognition measures for the SVM were obtained when VS_LBP-TOP + TS_HDTP was provided as input. It produced a recognition rate that was at least 0.10 greater than the recognition rate for inputs without TS_HDTP where the range for recognition rates was 0.13. This provides evidence that TS_HDTP had a significant contribution towards the better classification performance and suggests that TS_HDTP captured more patterns associated with stress than VS_LBP-TOP and TS_LBP-TOP. The performance for the classification was the lowest when TS_LBP-TOP was provided as input.

The features were also provided as inputs to a GA which selected facial block regions with a goal to disregard irrelevant facial block regions for stress recognition and to improve the SVM-based recognition measures. Performances of the classifications using 10-fold cross-validation on the different inputs are provided in Figure 8. For all types of inputs, GA-SVM produced significantly better stress recognition measures. According to the Wilcoxon non-parametric statistical test, the statistical significance was p < 0.01. Similar to the trend observed for stress recognition measures produced by the SVM, TS_HDTP also contributed to the improved results in GA-SVM. The best recognition measures were obtained when VS_LBP-TOP + TS_LBP-TOP + TS_HDTP was provided as input to the GA-SVM classifier. The performance of the classifier was highly similar when it received VS_LBP-TOP + TS_HDTP as inputs with a difference of 0.01 in the recognition rate. Results show that when a combination of at least two of VS_LBP-TOP, TS_LBP-TOP, and TS_HDTP was provided as input, then it performed better than when only one of VS_LBP-TOP, TS_LBP-TOP, or TS_HDTP was used.

Further, stress recognition systems provided with TS_HDTP as input produced significantly better stress recognition measures than inputs with TS_HDTP replaced by TS_LBP-TOP (p < 0.01). This suggests that stress patterns were better captured by TS_HDTP features than TS_LBP-TOP features.

In addition, blocks selected by the GA in the GA-SVM classifier for the different inputs were recorded. When VS_LBP-TOP was given as inputs to a GA, the blocks that produced better recognition results were the blocks that corresponded to the cheeks and mouth regions on the XY plane. For VS_LBP-TOP, fewer blocks were selected and they were situated around the nose. On the other hand for TS_HDTP, more blocks were used in the classification - nose, mouth, and cheek regions and regions on the forehead were selected by the GA. Future work could extend the investigation by more complex block definitions to find and use more precise regions showing symptoms of stress for classification.

Future work could also investigate other block selection methods different from the GA used in this work. The GA search took approximately 5 min to reach convergence, but it could take longer if the chromosome is extended to encode more general information for a block, e.g., coordinate values and the size for the block. The literature has claimed that a GA usually takes longer execution times than other types of feature selection techniques, such as correlation analysis [57]. Therefore in future, other block selection methods could be investigated that do not require execution times as long as a GA and still produce stress recognition measures comparable to the GA hybrid.

7 Conclusions

The ANU Stress database (ANUStressDB) was presented which has videos of faces in temporal thermal (TS) and visible (VS) spectrums for stress recognition. A computational classification model of stress using spatial and temporal characteristics of facial regions in the ANUStressDB was successfully developed. In the process, a new method for capturing patterns in thermal videos was defined - HDTP. The approach was defined so that it reduced individual bias in the computational models and enhanced participant-independent recognition of symptoms of stress. For computing the baseline for stress classification, a SVM was used. Facial block regions selected informed by a genetic algorithm improved the rates of the classifications regardless of the type of video - videos in TS or VS. The best recognition rates, however, were obtained when features from TS and VS videos were provided as inputs to the GA-SVM classifier. In addition, stress recognition rates were significantly better for classifiers provided with HDTP features instead of LBP-TOP features for TS. Future work could extend the investigation by developing features for facial block regions to capture more complex patterns and examining different forms of facial block regions for stress recognition.